[chef] Re: Environments vs. Metadata vs. Policyfile for locking cookbook dependencies

[Daniel DeLeo < >]

On Friday, June 13, 2014 at 7:19 AM, Torben Knerr wrote:

> Hi everybody,
>  
> we recently started a discussion on the different ways you can lock the 
> cookbook dependency graph for a given node:  
>  
> 1. use Chef environments
> 2. use metadata.rb
> 3. use Policyfile (work in progress)
>  
> The discussion of environments vs. metadata started here on the list...
> http://lists.opscode.com/sympa/arc/chef/2014-05/msg00324.html
>  
> ...then continued on a github with @danielsdeleo, proposing a new mechanism 
> called Policyfile:
> https://github.com/berkshelf/berkshelf/issues/724#issuecomment-44980485
>  
>  
> Looking at the Policyfile approach, I like how the sketched terminal 
> session reads [1], but I'm also afraid that it will add even more 
> possibilities on how to center your workflow around Chef. It would be the 
> next competitor to "roles" vs. "environment cookbooks" vs. "metadata" vs. 
> "policyfile".
Long term, Policyfile should be the default workflow. It eliminates roles vs. 
role cookbooks as a point of contention by solving the primary flaw of roles, 
which is the fact that you can mutate them on live production nodes 
accidentally.
  
>  
>  
> So I'm wondering: can't we solve the problem with the tools at hand or do 
> we have to invent something new for it?
>  
> Almost everything that is described in the desgin principles [2] could be 
> easily solved by using cookbooks + metadata.rb today. You just have to make 
> sure that the contents of Berksfile.lock get translated into metadata.rb 
> depends statements. That's what I was calling "top-level" cookbook but you 
> could also call it "policy cookbook" if you will.  
>  
> Think of this:
> * for each node you have 1 top-level cookbook
> * this top-level cookbook has all the pinned versions from Berksfile.lock 
> via "depends" statements in its metadata.rb
> * in the the node's run_list is just the default recipe (or any other) of 
> the top-level cookbook, which defines the actual "run_list" that you would 
> otherwise have in roles via "include_recipe"
> * your top-level cookbook for sure has a version, as any other cookbook too
> * in your environments you only pin the top-level cookbook's version (all 
> transitive dependencies are pinned via the top-level cookbook's metadata)
>  
> What would be missing from this approach?
You give up a significant amount of control over the order in which cookbooks 
are run. Everyone using role cookbooks eventually hits the problem where 
computed attributes in attributes files have incorrect values because role 
cookbooks need dependencies to be loaded after themselves. This is 
unavoidable because cookbooks aren’t roles. That said, I won’t say that role 
cookbooks are bad per se, I just think Chef doesn’t give you the tools to 
compose behaviors without exposing you to unnecessary risk. Policyfiles fix 
that.

>  
> The only new thing proposed in [2] was the ability to uniquely identify 
> non-released development versions. The proposal suggests to use a hash over 
> the cookbook contents instead of name + version. I see the need for 
> properly identifying in-development versions as well, but IMO using a hash 
> would just obscure things. Instead I would rather see prerelease and build 
> identifiers being supported by the Chef cookbook versioning scheme (see 
> CHEF-4027 [3]) -> as we have for Ruby gems today.
What you’re saying isn’t actually how people really use rubygems today. If I 
need to run chef-client in development with a development version of ohai, I 
don’t release Ohai-7.5.0-dans_crazy_idea.5 to rubygems.org (also, you don’t 
have push rights for ohai on rubygems.org so you *can’t* do that), I point my 
Gemfile at a path or git source. You should be able to do the same with Chef 
and let Chef figure out how to put the pieces together on a remote node.

As I replied on the ticket, we understand that some random hex string is 
pretty meaningless to a human, which is why the Policyfile.lock will contain 
a fair amount of contextual information about cookbooks. This includes the 
source (local disk, community site, chef server, github), the semver version 
of the cookbook, and where relevant, git info such as the commit SHA, whether 
or not the repo is dirty, the git remote, and whether commits are 
synchronized to the remote. Before you apply this Policyfile.lock to any 
nodes, you can review a diff of all of this.

On the ticket, you said 'to me it sounds still like a "build identifier”’, 
which it is. chef-server and chef-client will talk to each other in terms of 
build identifiers. You also said, 'When talking to my colleagues I'd 
personally rather talk about apache-2.0.0 rather than 
f59ee7a5bca6a4e606b67f7f856b768d847c39bb’. You can totally have both. Part of 
the workflow we’re designing is that you compute the lockfile and can look at 
it, commit it to source control if desired, etc. before you apply it to 
anything. And the lockfile contains more than enough information for you to 
talk about the cookbook in human-understandable terms. For example, look at 
the lockfiles in the tests: 
https://github.com/opscode/chef-dk/pull/53/files#diff-db3efba452f4c4875514079ef3d4e7f7R320
 (there is a tiny bit of indirection there to make the tests more 
maintainable). You can talk with your colleagues about “apache 2.0.0 that 
came from this github repo” or a cookbook that was uploaded but not committed 
to source control, or you can see in a diff that you changed from the 
mainline version of a cookbook to your own fork.  

And all of this happens automatically. You don’t have to spend your own time 
editing version numbers to encode this information. That’s both crap work and 
error prone.
  
>  
>  
> Just saying: let's start simplifying and improve the tools around the 
> concepts we currently have, not inventing additional and competing concepts 
> that make everything more complex. Reuse the existing concepts, establish 
> conventions, and foster them by making sure the tools we use promote them.

I don’t think the existing concepts are good enough. I say this as someone 
who helped to design them. The way people used and thought about chef, 
cookbooks, etc. when we created environments, for example, was totally 
different than the understanding we have today. Based on what we’ve learned 
from other people and new tools, we can now see a way to make chef and 
chef-server work in a better, safer, and more humane way. So we should.
  
>  
> That was quite much to digest... hope that makes still sense :-)
>  
> Cheers,
> Torben
>  
>  
> [1] 
> https://github.com/danielsdeleo/chef-workflow2-prototype/blob/master/docs/demo-script.txt
>   
> [2] 
> https://github.com/danielsdeleo/chef-workflow2-prototype/blob/master/docs/design-principles.md
> [3] https://tickets.opscode.com/browse/CHEF-4027

--  
Daniel DeLeo