[chef] Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Deep merge in 0.10.10 and 10.12.0

[Michael Glenney < >]

We have a control system that fires off the deploy (chef-client run) and provides the nodes to use.  So I'll have to get creative.  Can't use node search data.  need to be able to pass a block of data and have it acted upon.

I think I need to go with something along the lines of what you said about interacting with the node data from the Recipe or Node DSL's.  Is the data provided in json via the cli stored in an object I can access from my recipes?  If I had access to that data then I could act on it knowing it's not mixed with existing node data.  If it is available and you have tips/links/etc about accessing it I'd appreciate any head-start I can get.

MG

On Tue, Sep 4, 2012 at 5:02 PM, AJ Christensen < " target="_blank"> > wrote:

Hello,

On 5 September 2012 11:36, Michael Glenney < "> > wrote:
> Thanks for all the replies.  I'll have a read through all that but it sounds
> like I'm going to have to reengineer how we supply node data for our haproxy
> configs.  Or stay on 0.10.8 :)   Unfortunately, we don't use roles in our
> organization because they can't be versioned so that's not an option.

I'd suggest using Search for this.

I developed a cookbook some time ago to assist with using Search
directly from recipes (to reduce dire repetition):
- http://ckbk.it/discovery
- https://github.com/hw-cookbooks/discovery/tree/develop

>
> It does seem like a weird thing to get rid off though.  Is there another way
> to clear an attribute and put in new data without that?  Or once a node has
> data that data must live on forever unless you use roles?

You can interact with the node data from the Recipe DSL or Node
Attribute DSL, or well; mostly anywhere before the final node.save is
called during convergence -- I guess technically you could modify the
node in the report handler, and then save again there.. but that's
gross!

You could use Node#delete(key) to delete a key from any of the
attribute classes for the (local) node: normal, default and override,
e.g.:

old_nodes = search(..) # could use discovery_search here
old_nodes.each do |remote_node,params|
  node['haproxy']['some_hash_of_server_data'].delete(remote_node) if
node['haproxy']['some_hash_of_server_data'].key? remote_node
end
node.save

> I suppose the problem ultimately is that we don't use roles.  I just need to
> find a way around this.  Putting this data in data_bags isn't an option
> since nodes can't edit data_bags unless they have admin rights so we only
> use data bags for data that doesn't get updated often or at runtime.

Using this combination of technologies, you may be able to build a
plan-and-commit HAProxy back-end management system with Chef Node data
-- In an Infrastructure I built long ago, we made use of Knife Exec to
perform Search queries to extract the information from the
Infrastructure; this in turn generated a data-bag in JSON form which
was uploaded by a Human control-gate; allowing congruence at the
load-balancer layer dependent upon multiple systems convergence while
tolerating a fluid number of back-ends.

I'm increasingly less fond of Human-gates for this kind of stuff
(updating the source of truth) but sometimes we are useful!

Cheers,

--AJ

>
> MG
>
>
> On Tue, Sep 4, 2012 at 4:18 PM, Daniel DeLeo < "> > wrote:
>>
>> Note that this behavior is intentional:
>>
>> http://tickets.opscode.com/browse/CHEF-2737
>> http://tickets.opscode.com/browse/CHEF-2818
>>
>> Also, knockouts will most likely be removed from Chef 11.0. See:
>>
>> http://lists.opscode.com/sympa/arc/chef/2011-12/msg00079.html
>>
>> --
>> Daniel DeLeo
>>
>> On Tuesday, September 4, 2012 at 4:14 PM, Juanje Ojeda Croissier wrote:
>>
>> Yes, I'm running 10.12.0, but I did it with a role. Maybe that's it.
>>
>> On Wed, Sep 5, 2012 at 12:02 AM, Michael Glenney < "> >
>> wrote:
>>
>> Hmm. You're running 10.12.0? I just tested again. Started with the
>> following:
>>
>> Node data (with 'knife node show <nodename> -Fj'):
>>
>> "haproxy": {
>> "config": "classroom.cfg.erb",
>> "nodes": {
>> "portal_api": [
>> "2.2.2.2",
>> "3.3.3.3"
>> ]
>> }
>>
>> haproxy config:
>>
>> server portal_api-2.2.2.2 2.2.2.2:443 check port 443 inter 10s rise 4 fall
>> 4
>> server portal_api-3.3.3.3 3.3.3.3:443 check port 443 inter 10s rise 4 fall
>> 4
>>
>> I ran 'chef-client -j /etc/chef/routing_rules.json' with the following in
>> "routing_rules.json":
>>
>> { "haproxy": { "nodes": { "portal_api": [ "!merge:", "10.10.10.10",
>> "11.11.11.11" ] } } }
>>
>>
>> This is what I ended up with:
>>
>> Node data:
>>
>> "haproxy": {
>> "config": "classroom.cfg.erb",
>> "nodes": {
>> "portal_api": [
>> "2.2.2.2",
>> "3.3.3.3",
>> "!merge:",
>> "10.10.10.10",
>> "11.11.11.11"
>> ]
>> }
>>
>> haproxy config:
>>
>> server portal_api-2.2.2.2 2.2.2.2:443 check port 443 inter 10s rise 4 fall
>> 4
>> server portal_api-3.3.3.3 3.3.3.3:443 check port 443 inter 10s rise 4 fall
>> 4
>>
>> server portal_api-!merge: !merge::443 check port 443 inter 10s rise 4 fall
>> 4
>> server portal_api-10.10.10.10 10.10.10.10:443 check port 443 inter 10s
>> rise 4 fall 4
>> server portal_api-11.11.11.11 11.11.11.11:443 check port 443 inter 10s
>> rise 4 fall 4
>>
>> The docs say you can run without the colon now but I'm not seeing a
>> difference. Doesn't work either way.
>>
>> MG
>>
>>
>> On Tue, Sep 4, 2012 at 3:41 PM, Juanje Ojeda Croissier
>> < "> > wrote:
>>
>>
>> I believe the issue here is the " (without the colon) ". I had some
>> troubles here as well, but it's now working with the colon
>> ("!merge:").
>>
>> Could you test it with colon, please?
>>
>> On Tue, Sep 4, 2012 at 8:27 PM, Michael Glenney < "> >
>> wrote:
>>
>> I have now verified that putting !merge does not work in 10.12.0 I'll
>> go
>> back an read over the link AJ sent again but it appears as if deep merge
>> is
>> gone or broken.
>>
>> Here's the json I passed to the chef-client run:
>>
>> { "haproxy": { "nodes": { "portal_api": [ "!merge", "4.4.4.4", "5.5.5.5"
>> ] }
>> } }
>>
>> Here's the resulting haproxy config:
>>
>> server portal_api-!merge !merge:443 check port 443 inter 10s rise 4
>> fall 4
>> server portal_api-4.4.4.4 4.4.4.4:443 check port 443 inter 10s rise 4
>> fall
>> 4
>> server portal_api-5.5.5.5 5.5.5.5:443 check port 443 inter 10s rise 4
>> fall
>> 4
>>
>> Not what I was going for obviously.
>>
>> MG
>>
>>
>> On Sun, Sep 2, 2012 at 6:46 PM, Michael Glenney < "> >
>> wrote:
>>
>>
>> Yeah. Looking at your link it looks like using "!merge" (without the
>> colon) will probably work. just need to test it. That goes along with
>> what
>> it says in the wiki except for the fact that the wiki says it "can" be
>> used
>> instead of saying it must.
>>
>> MG
>>
>> On Sun, Sep 2, 2012 at 3:11 PM, AJ Christensen < "> >
>> wrote:
>>
>>
>> From DeepMerge mixin class:
>>
>> # Inherited roles use the knockout_prefix array subtraction
>> functionality
>> # This is likely to go away in Chef >= 0.11
>> def role_merge(first, second)
>> first = Mash.new(first) unless first.kind_of?(Mash)
>> second = Mash.new(second) unless second.kind_of?(Mash)
>>
>> DeepMerge.deep_merge(second, first, {:knockout_prefix =>
>> "!merge", :preserve_unmergeables => false})
>> end
>>
>>
>>
>>
>> https://github.com/opscode/chef/blob/master/chef/lib/chef/mixin/deep_merge.rb#L44
>>
>> Maybe there's something useful for you here.
>>
>> I've only once used the attribute knockout functionality -- what is
>> your use case? There may be another simple solution.
>>
>> --AJ
>>
>> On 3 September 2012 09:08, Michael Glenney < "> >
>> wrote:
>>
>> Weird. For some reason it's not sending me my own. Oh well. Have
>> there
>> been any responses to my question? Because I dont see any of those
>> either
>>
>> Michael Glenney
>> Sent from my iPhone
>>
>> On Sep 2, 2012, at 11:43 AM, Adam Jacob < "> > wrote:
>>
>> Nope - I saw it two days ago.
>>
>> Adam
>>
>> On Sun, Sep 2, 2012 at 10:04 AM, Michael Glenney
>> < "> > wrote:
>>
>> I sent this on friday. Still hasn't made it to my inbox. Am I
>> getting
>> moderated?
>>
>> Michael Glenney
>> Sent from my iPhone
>>
>> On Aug 31, 2012, at 4:08 PM, Michael Glenney
>> < "> >
>> wrote:
>>
>> I had a problem with deep merge not working properly in 10.12.0.
>> Tested in
>> 0.10.10 and it didn't work there either. Instead of replacing my
>> array it
>> was actually putting "!merge:" into my array.
>>
>> After I reverted to 0.10.4 to get my deployment completed I
>> stumbled
>> on this
>> line from http://wiki.opscode.com/display/chef/Deep+Merge:
>>
>> "!merge (without the colon) can be used in Chef >= 0.10.10"
>>
>> I haven't had a chance to go back and test. Will have to wait
>> until
>> Monday.
>> But should that line read:
>>
>> "!merge (without the colon) MUST be used in Chef >= 0.10.10"
>>
>> Thanks,
>>
>> MG
>>
>>
>>
>>
>> --
>> Opscode, Inc.
>> Adam Jacob, Chief Customer Officer
>> T: (206) 619-7151 E: ">
>>
>>
>>
>>
>> --
>> Juanje
>>
>> http://about.me/juanje
>>
>>
>>
>>
>> --
>> Juanje
>>
>> http://about.me/juanje
>>
>>
>