[chef] Re: Re: Re: Re: on the general usage of chef

[Ranjib Dey < >]

this was also discussed in last summit. version everything (roles, environments.. etc) was an item in chef 12 wish list.. may be @btm have some update?

On Wed, Jan 15, 2014 at 7:59 AM, Sam Pointer < " target="_blank"> > wrote:

At the risk of continually pimping out the same talk to this list, we ran into a number of these issues at EA and discuss them here: http://lanyrd.com/2013/ldndevops-january/scbzkw/.

Along with disciplined source control: the concept of releases, etc. we built some tooling to:

• automatically handle cookbook versioning and version pinning in roles
• monitor for run_list conformity

A great number of problems in this area would be solved with the ability to version roles, something we raised 2 years ago I raised again with Chris Brown during a recent speaking engagement with him. Instead we have the cookbook wrapper pattern which in practice isn't especially great. Others in our circle have approached this from the point of view of having Jenkins move cookbooks through environments inspecting the output of chef spec and minitest runs, lints and whatnot before gating the release to live. In the round however the problem of continually deploying *code* to production is a development problem and one that is quite mature in terms of tooling and practice. Operations people should be looking to their development teams for guidance. Devops works both ways ;)

Sam Pointer

Lead Consultant

www.opsunit.com

On 14 January 2014 18:57, Lamont Granquist < " target="_blank"> > wrote:

Yeah, the answer is to use environments and to explicitly version pin all your cookbooks.  I'm not sure I'd suggest going down the road of explicitly version pinning in your cookbook deps, but instead your promotion from integration to production (or however you have your environments setup) should be to copy the exact version pins in the integration environment file and push them into the production one.  Ideally you have a full CD pipeline where you start with "devops"ers working on their laptops doing local changes to cookbooks and using test-kitchen to validate them against virtual images.  Then those changes are published and made available as a cookbook version on the chef server, and a jenkins environment that floats on cookbook versions picks them up and runs test-kitchen against them, and if they pass they're promoted to a full-sized integration environment where again there's a test suite that validates that the changes work.  If you swallow the whole koolaid then you will eventually do a push to production with any CRB approval since you've proved the change works via multiple testing steps in your different environments.

You don't even really need semantic versioning in this system, you just need to keep cookbook versions unique and frozen.  The set that gets deployed is the set that was tested together and works, you don't really want any version constraints in your cookbooks at all, you just test latest against latest and if its good you ship it.

The same basic workflow also works if you rip out all the CD, but then you've got manual Q/A process and humans pressing buttons to do the environment promotion, and a CRB involved at some point before it goes to prod.

On 1/14/14 8:15 AM, Dylan Northrup wrote:

On Tue, Jan 14, 2014 at 7:02 AM, Sam Darwin < " target="_blank"> > wrote:

it seems like the ideal case of using chef, would be to run it as a cronjob
every day on the servers, to keep the servers in their proper and perfect
configuration.

In practice, we are currently not doing this.   there is more than just one
simple reason for it.

- the recipes would need to always be in a perfect and functional state.
let's say that you were experimenting with something.   and then every single
server in your environment grabs that recipe and runs it.    this could affect
the entire system, every single server.

If you pin recipes (directly in run_lists or indirectly in environments or roles), you can upload new versions of recipes without fear of having them propagate to unintended servers.  An operational pattern I've used in the past is to have dev servers with versions that aren't pinned (so they get the latest version of whatever cookbooks are out there) and have every other type of server (integration, reference, production, etc) have explicitly pinned versions.  This allows "testing" and "development" of cookbooks in the same chef organization/server (insuring you're don't have any "Dev is different than ref is different than int is different than prod" issues) without concern that new cookbook versions will affect production nodes (or any other ones, for that matter).

Two things you'll want (or need) to do if you put this into practice:

1. Verify your pinning all the way down for non-dynamic cookbooks.  We distribute user information (logins, ssh keys, sudoers, etc) via a generated cookbook.  The version of this cookbook will change every time it's newly generated.  Since we want to use the most recent version of this cookbook every time, we do not do any pinning for this cookbook.  But for cookbooks that are not dynamic, everything gets pinned explicitly or via dependent pinning.  So, the platform cookbook explicitly pins apt, bash, system ruby, etc; nginx cookbook explicitly pins the lua cookbook and any module cookbooks it depends on; tomcat cookbook explicitly pins the java cookbook version; and so on.  Make sure it's pinned turtles all the way down.  It's a pain if you have to retrofit, but worth it IMO.

2. Every time you upload a cookbook that will be pinned, it is frozen.  Every time, no exceptions.  What you don't want, after going through all the trouble of making sure you know exactly what version is going to go out on a specific type of node, only to have that version change.

Still doesn't mean you can't propagate an error to many machines, but with additional gates in place, it makes it more difficult for this to happen unintentionally.