[chef] Re: Re: Re: Re: Re: Re: Environment Cookbook Patterns & Questions

[Nathan Williams < >]

we've been using an interesting pattern based on ideas that have been around a while, that's been working pretty well for us, but i haven't seen discussed much. i've been meaning to write up a blog post about (just as soon as i find that 25th hour in the day). here's a quick overview, happy to flesh out any details if you have questions.

we're a 2-person ops team in a medium-sized org with about 50 nodes, mixed about 90/10 between VMs and physical gear. not sure how well this would scale to a bigger setup, but not seeing any bottlenecks at present, except that cookbook promotion is more manual than i would like, which we're planning on fixing with some CI pipeline stuff. anyways... the details:

we refuse to set attributes in roles, and just use them for runlist management. nodes get roles in the runlist, roles get recipes (e.g. app::default and app::monitoring). roles roughly correlate to suites in our test-kitchen configuration (i.e. monitoring cookbook has both client and server suites and corresponding roles)

we use wrapper, library, and application cookbooks. our application cookbooks roughly map to roles, with a few exceptions.

we run our own berkshelf-api service that hooks into our hosted-chef org, and use a berksfile with just the metadata config in each cookbook repo. the cookbook-berksfile has our berks-api listed as a source, and the berksfile.lock is in the .gitignore, and uploading new cookbooks is done with berks upload from the cookbook repo.

we then have an infrastructure repo, where we keep documentation, runbooks, maintenance plans (ops stuff that isn't software), and "the" Berksfile that lists all of our org-specific cookbooks. the infrastructure repo has one branch per environment. promoting a cookbook is just a matter of berks updating a cookbook (berks update myface) after it's been uploaded, and berks applying to the environment that corresponds to the relevant branch (we initially envisioned promoting from staging->prod via PRs of the diff to the Berksfile.lock, but it turned out to be a hassle, so we just handle it manually for now). this also provides a nice audit-log in the form of the git-log if something should go wrong, and visibility into any dependent or contingent cookbooks that may also be changed.

overall, this gives us really good control (by locking the environment with berks apply) over what changes are released, without getting much in the way

that said, we don't have a way to release an updated library cookbook or community cookbook to a specific role. one version of each cookbook per environment, period. i've heard of people addressing this with role-cookbooks, but it hasn't been a problem for us in practice yet, so we're not using them. i expect we'll switch to policyfiles before introducing role-cookbooks, and we're eagerly watching policyfile support develop.

hope that helps!

-- nathan

Regards,

Nathan Williams

On Mon, Jan 5, 2015 at 11:59 AM, Matt Juszczak < " target="_blank"> > wrote:

Thanks for the suggestion!

> On Jan 5, 2015, at 6:02 AM, Torben Knerr < "> > wrote:
>
> My favorite approach is to use a "top-level cookbook". A top-level
> cookbook roughly represents a VM. It is essentially the same as an
> "environment cookbook" but uses metadata.rb for locking the dependency
> graph rather than environments.
>
> This lets you use environments for representing your infrastructure
> environments rather than (ab?)using them for locking dependency
> graphs. The only cookbook dependency you have in the environment is
> the one to the top-level cookbook (all others come in transitively via
> metadata.rb).
>
> Related discussions:
> * http://lists.opscode.com/sympa/arc/chef/2014-01/msg00419.html
> * http://lists.opscode.com/sympa/arc/chef/2014-06/msg00173.html
>
> Examples:
> * https://github.com/tknerr/sample-toplevel-cookbook
> * https://gist.github.com/tknerr/4e3236d00ceba917abea
>
> Word of caution: I'm using this with chef-solo and have not checked
> whether this way of keeping metadata.rb and Berksfile.lock in sync
> (https://gist.github.com/tknerr/4e3236d00ceba917abea) works with
> chef-client too. If not, you might want to us a generative approach
> instead.
>
> HTH,
> Torben
>
> On Mon, Jan 5, 2015 at 12:04 AM, Matt Juszczak < "> > wrote:
>> Agreed!! Thanks for the help! It's worth mentioning that "poise-appenv"
>> seems to try to solve this problem.
>>
>> Matt
>>
>> On Jan 4, 2015, at 7:08 AM, Christine Draper
>> < "> > wrote:
>>
>> Matt,
>>
>> Your analysis makes a lot of sense to me. We need a way to separate the
>> inherent variances between infrastructure environments as one dimension; and
>> the through-time release ("code") variances as another.
>>
>> Christine
>>
>> On Sat, Jan 3, 2015 at 10:09 PM, Matt Juszczak < "> > wrote:
>>>
>>> Christine,
>>>
>>>> It makes sense to me to create a base 'myface' cookbook that knows how
>>>> to set up the logical components of the application like database servers,
>>>> application servers, the actual application code etc.  I'd consider this to
>>>> be an application cookbook, one that you could then reuse in multiple
>>>> contexts (e.g. the different production data centers).  Perhaps it might
>>>> even be reused in different lifecycle phases (prod, staging, systest).
>>>>
>>>> The above might become an environment cookbook if you use it to lock
>>>> down the cookbook versioning for actual deployments.
>>>
>>> Gotcha. So in this case, the application cookbook and environment cookbook
>>> are *literally* the same thing and you don’t have BOTH. An application
>>> cookbook simply becomes an environment cookbook once a Berksfile.lock is
>>> checked in. Jamie mentions that application cookbooks and environment
>>> cookbooks don’t differ much, except to me it’s been unclear whether having
>>> an environment cookbook for an app replaces the functionality of the
>>> application cookbook all-together (simply by now having a Berksfile.lock in
>>> revision control). I think I got it now.
>>>
>>>> In your case it sounds like intend the east and west data centers to be
>>>> part of one overall production environment and want to keep them in sync
>>>> from a cookbook (and application software) version perspective. If so, the
>>>> environment cookbook could be 'myface-prod' and consist of little more than
>>>> a berksfile.lock plus attributes to customize the application cookbook
>>>> specifically for prod environment.
>>>
>>> I’m thinking we would make our environment cookbooks *code environments*
>>> only. In other words, while a prod environment might have static attributes
>>> that differ from a staging environment and will *always* differ, it seems
>>> the environment cookbook pattern that Jamie describes is about managing an
>>> SDLC. You apply version 1.0.1 of a cookbook to your staging environment, and
>>> eventually that makes its way to prod. Therefore, to me, it makes sense to
>>> have a “myface” environment cookbook that can be versioned, and utilize the
>>> berkflow tool to promote different versions of that cookbook through chef
>>> server environments.
>>>
>>> But that still doesn’t solve the problem of *infrastructure environments*
>>> (IE: configuration that applies to servers in the staging environment but
>>> never the prod environment and vice versa). For example, if you deploy
>>> version 1.0.0 of myface to staging, you’re eventually going to deploy that
>>> to production. However, if you deploy changes to “staging environment
>>> configuration” (perhaps network addresses?), you likely aren’t going to copy
>>> those configuration parameters to production any time soon. That
>>> configuration is strictly for the staging infrastructure environment!
>>>
>>> So…
>>>
>>>> The key question is how you can then add in the 'east' and 'west' data
>>>> center variances. Environment files for myface-prod-east and
>>>> myface-prod-west would seem a decent option, if the variances are fairly
>>>> simple attribute differences, and particularly if the variances were
>>>> relatively independent/orthogonal to the application (e.g. updating
>>>> networking information for one data center is relatively independent of
>>>> changes to the myface application). You'd likely want to version the
>>>> environment definitions externally in source control.  You may need to
>>>> coordinate cookbook changes with environment changes, in the case where an
>>>> application change requires networking changes in the two data centers.
>>>
>>> Based on your feedback, and the more I think about this, the more I think
>>> it does make sense to create more environment files in chef server and
>>> maintain them in revision control. These environment files would contain
>>> both application (managed by berkflow) and infrastructure environment
>>> configuration options (such as network configuration). Therefore, we would
>>> likely create something like:
>>>
>>> myface-prod-us-east
>>> myface-prod-us-west
>>> myface-staging-us-east
>>> myface-staging-us-west
>>>
>>> and use berkflow to manage version pinning:
>>>
>>> blo upgrade myface-prod-useast myface
>>> blo upgrade myface-prod-uswest myface
>>>
>>> ...and simply build a wrapper tool to upgrade all “prod” environments at
>>> once that calls blo upgrade numerous times. Meanwhile, the environments
>>> above (myface-prod-useast) have specific configuration for servers in that
>>> environment, such as network configuration.
>>>
>>> Thanks for your insight. I’m still curious to hear some other opinions but
>>> it’s good to know others have been confused on this and I’m not missing
>>> something simple.
>>>
>>> -Matt
>>
>>