[chef] Re: Re: Re: Re: Puppet+Hiera vs. Chef, or handling the occasional special snowflake server


Chronological Thread 
  • From: Fabien Delpierre < >
  • To: chef < >
  • Subject: [chef] Re: Re: Re: Re: Puppet+Hiera vs. Chef, or handling the occasional special snowflake server
  • Date: Wed, 7 Oct 2015 15:58:57 -0400

Attributes at the “normal” level belong to the node and are NOT wiped for each run. All of the other levels belong to an external thing (role, environment, policyfile) so they are wiped and recreated.

Attributes on the node object are only overwritten in the Chef run if there is something in the recipe code which specifically alters that attribute. The node object is both the desired state and current state of the object (although there's future work to split that planned I believe).

Alright, so I never bothered to look up normal attributes, but that makes sense then. Is it correct that, by default, knife node edit will set normal-level attributes, and if you want to set something at a different level as a one-off, you can use the -a flag?

Hiera can aggregate multiple hashes or arrays spread between multiple sources into a single attribute.
This is exactly what the chef attribute precedence system does. My understanding is Hiera was written in response to chef’s attributes system, though as a second system it has the benefit of learning from our limitations so it might be more flexible in some ways. In particular Chef has only one attribute level where you can set node-specific values, but these get merged with cookbook/role/environment attributes. Arrays are tricky since sometimes you need them to merge and sometimes override, I would recommend sticking to hashes and then join things yourself.

I guess I was confused. Admittedly I haven't had to worry about merging hashes in Chef attributes, but I thought that attributes were overwritten by sources higher in the food chain, period. I now see that's not the case with hashes: https://docs.chef.io/attributes.html#about-deep-merge
Well, good to know!

So I guess I'm still trying to figure out if key Hiera functionality is easily achieved in Chef or if you just have to use a totally different approach. The examples I used kind of steered the conversation in the wrong direction (although valuable things were still learned!).
Concretely, here's how I might have Hiera set up:
---
:backends:
  - yaml
:yaml:
  :datadir: /etc/puppet/hiera
:hierarchy:
  - %{::region}/%{::fqdn}
  - %{::region}/%{env}
  - %{::region}/common
  - global


That means I can store a whole bunch of attributes in /etc/puppet/hiera/global.yaml that will apply to all nodes.
Attributes stored in /etc/puppet/hiera/us-east/common.yaml can add to or override attributes stored in global.yaml.
Attributes stored in /etc/puppet/hiera/us-west/common.yaml will not interact/-fere with the ones in us-east.
Attributes stored in /etc/puppet/hiera/us-west/%{env}.yaml will be applied to all nodes in us-west whose names match %{env} (e.g. if %{env} = foo then all nodes called foo01, foo02, foo03, etc. will be matches.
Attributes stored in /etc/puppet/hiera/us-west/%{::fqdn}.yaml will only be applied to the particular node named by its FQDN (that special snowflake I was talking about) and will override attributes (or merge, if they are arrays or hashes) that are stored lower in the hierarchy.

I suppose that one way this can be achieved with Chef is with a set of roles, so I'd have common, us-east, us-west and <fqdn> roles, but it gets dicey with my %{env} thing above where settings can be applied to all nodes with similar names (I used 'web' in my example but it could be 'foo', 'bar', 'asdf'...). I don't know how I can do that with Chef as easily as with Hiera. Can I?


On Wed, Oct 7, 2015 at 11:39 AM, Daniel DeLeo < " target="_blank"> > wrote:
On Wednesday, October 7, 2015 at 6:36 AM, Fabien Delpierre wrote:
> > Generally you would try to use an existing grouping like a chef environment, role, or policy (if you are using any of those). For single node edits you would do it directly via `knife node edit`. For things that are between the scope of a role and environment, https://github.com/poise/poise-appenv can be a solution.
>
>
> Good to know about knife node edit. Wouldn't any change be overridden at the next chef-client run, though?
> I guess I ought to look into policies, too. I've never used them but I'm not finding much documentation about them. Are we talking about the new-ish Policyfile feature? That I can find documentation about.

Attributes at the “normal” level belong to the node and are NOT wiped for each run. All of the other levels belong to an external thing (role, environment, policyfile) so they are wiped and recreated.

And yeah, policies means policyfiles. We’re still hammering out the exact official terminology.
>
> I was including that when I mentioned "editing the run list" -- of course that works but if I have multiple nodes that use the same set of roles/recipes and I edit one of those roles or recipes, all of them will be affected, which I may not want.
>
> knife node edit does seem like it might help, but I'm concerned that the change wouldn't persist past a chef-client run.
knife node edit by default will only show you things that will be persisted across chef runs: run_list, environment, “normal” attributes.

>
> > As an example I have a recipe which installs a Newrelic agent if a certain attribute is set to true. In the production Chef environment I set the attribute so that all production nodes have this. If I want to use newrelic for a machine outside of production I can set that attribute on the individual node object instead, which could be in any environment.
>
> So that makes sense if you have a recipe that's designed to look for that attribute, but that still seems like it wouldn't accomplish what one might do with Puppet's Hiera.
>
> To use a concrete example, again with attaching YourKit to a JVM process -- like I mentioned, Hiera can aggregate multiple hashes or arrays spread between multiple sources into a single attribute.
This is exactly what the chef attribute precedence system does. My understanding is Hiera was written in response to chef’s attributes system, though as a second system it has the benefit of learning from our limitations so it might be more flexible in some ways. In particular Chef has only one attribute level where you can set node-specific values, but these get merged with cookbook/role/environment attributes. Arrays are tricky since sometimes you need them to merge and sometimes override, I would recommend sticking to hashes and then join things yourself.

> For example, in one YAML file I could have:
> foo::java_opts: "-Xms2g -Xmx2g"
> And in another:
> foo::java_opts: "-agentpath:/opt/yjp/bin/linux-x86-64/libyjpagent.so"
> Then I could invoke this foo::java_opts in a recipe or a template (e.g. for an init script) and Hiera would combine them into one so it produces an init script might read: java -Xms2g -Xmx2g -agentpath:/opt/yjp/bin/linux-x86-64/libyjpagent.so
> When I'm done with YourKit, I can just delete the YAML file that contains those bits and I never have to touch the nodes or the recipe since it only looks for what's in that java_opts attribute and spits it back out.
>
> While so far it doesn't seem like this would be possible in Chef, it does seem like tags would help, if accounted for within a recipe, e.g.:
> node.default['foo']['bar'] = false
>
> if tagged?('bar')
> ruby_block 'enable_bar' do
> block do
> node['foo']['bar'] = true
> end
> end
> end
>
> if node['foo']['bar'] == true
> (do unspeakable things)
> end

My recommendation would be to make the snowflake case part of your cookbook for the non-snowflake case. That way it’s always easy to find out what your system will look like for both cases, and your snowflake configs will be tracked in your git history just like the non-snowflake.


--
Daniel DeLeo





Archive powered by MHonArc 2.6.16.

§