[chef] Versioning roles (was: on the general usage of chef)

[Daniel DeLeo < >]

On Monday, January 20, 2014 at 2:24 AM, Ranjib Dey wrote:

this was also discussed in last summit. version everything (roles, environments.. etc) was an item in chef 12 wish list.. may be @btm have some update?

I think that allowing versioning of roles will require simplification elsewhere in the model. We already see “failure” cases with the server side dependency solver model where the dependency solver unexpectedly chooses an older version of a cookbook than the user expects because it produces what the solver sees as a more optimal solution. 

As a more concrete example of this, suppose you have a cookbook FOO, where FOO 1.0.0 has no dependencies, and FOO 2.0.0 depends on BAR (any version). If you delete BAR from the server, then the dependency solver will choose FOO 1.0.0 as the valid solution to the dependency constraints. This is only a simple example, and when you get more real-world complexity, the outcomes can get more confusing. 

Now, if you imagine a case like the above, except that the run list is different between different versions of roles, which causes different cookbooks with different dependency constraints to get pulled (oh, and roles can set dependency constraints on recipes, too, so that could factor in), you have a recipe for madness.

That said, roles do solve a handful of problems pretty well: they provide a way to compose run_lists so that you don’t need to edit 10s or 1000s of node objects to add a new recipe to those node’s run_lists; they allow you to customize attributes in a way that works much better than the role cookbook pattern (especially in chef 11). These are good things and we want to have them.

I’ve been privately discussing with some folks an alternative way to address this problem, which is to have nodes get their run lists from a separate document/resource/object called a Policyfile, which would replace both roles and environments. You would have a Berksfile-like file that would include a run_list (which could contain roles if you desire), attributes, and a set of cookbook version constraints. On your workstation, a tool would evaluate this file to produce a document, called a "policy lock file," containing the expanded run list, the exact versions of cookbooks to use, and a compiled set of attributes (including attributes from roles, if you used them). You’d then push this to the server (and upload any required cookbooks), and nodes assigned to this policy would use that run list, attributes, and cookbook set. Since all the factors affecting what code chef-client runs are statically defined in the policy lock file, you could diff your proposed version with the current version and see the exact changes that would be applied to your systems.  In this model, role versioning becomes a lot less important because role updates don’t take effect until you “recompile” your Policyfile and push it to the server. You would also be able to fetch roles from a variety of sources before compiling (such as on-disk or a git repo) so you would have a variety of options for adapting role sources to fit your desired workflow.

We built a demoware prototype of this tool/workflow. Note that it is 90% non-functional and is intended to function like a UI wireframe drawing, so none of the design decisions implied by this prototype are set in stone.  https://github.com/danielsdeleo/chef-workflow2-prototype

Any commentary or questions are welcome.

-- 

Daniel DeLeo