[chef] Re: Environments vs. Metadata vs. Policyfile for locking cookbook dependencies

[Daniel DeLeo < >]

On Friday, June 13, 2014 at 9:16 AM, Torben Knerr wrote:
>  
> On Fri, Jun 13, 2014 at 5:06 PM, Daniel DeLeo 
> <
 
>  
> (mailto:
 )>
>  wrote:
> >  
> >  
> > On Friday, June 13, 2014 at 7:19 AM, Torben Knerr wrote:
> >  
> > > Hi everybody,
> > >  
> > > we recently started a discussion on the different ways you can lock the 
> > > cookbook dependency graph for a given node:
> > >  
> > > 1. use Chef environments
> > > 2. use metadata.rb
> > > 3. use Policyfile (work in progress)
> > >  
> > > The discussion of environments vs. metadata started here on the list...
> > > http://lists.opscode.com/sympa/arc/chef/2014-05/msg00324.html
> > >  
> > > ...then continued on a github with @danielsdeleo, proposing a new 
> > > mechanism called Policyfile:
> > > https://github.com/berkshelf/berkshelf/issues/724#issuecomment-44980485
> > >  
> > >  
> > > Looking at the Policyfile approach, I like how the sketched terminal 
> > > session reads [1], but I'm also afraid that it will add even more 
> > > possibilities on how to center your workflow around Chef. It would be 
> > > the next competitor to "roles" vs. "environment cookbooks" vs. 
> > > "metadata" vs. "policyfile".
> > Long term, Policyfile should be the default workflow. It eliminates roles 
> > vs. role cookbooks as a point of contention by solving the primary flaw 
> > of roles, which is the fact that you can mutate them on live production 
> > nodes accidentally.
>  
>  
> ​Sounds good.  
>  
> Still I can not see the full picture of the Policyfile yet, so more 
> questions following inline...  
>  
> > >  
> > >  
> > > So I'm wondering: can't we solve the problem with the tools at hand or 
> > > do we have to invent something new for it?
> > >  
> > > Almost everything that is described in the desgin principles [2] could 
> > > be easily solved by using cookbooks + metadata.rb today. You just have 
> > > to make sure that the contents of Berksfile.lock get translated into 
> > > metadata.rb depends statements. That's what I was calling "top-level" 
> > > cookbook but you could also call it "policy cookbook" if you will.
> > >  
> > > Think of this:
> > > * for each node you have 1 top-level cookbook
> > > * this top-level cookbook has all the pinned versions from 
> > > Berksfile.lock via "depends" statements in its metadata.rb
> > > * in the the node's run_list is just the default recipe (or any other) 
> > > of the top-level cookbook, which defines the actual "run_list" that you 
> > > would otherwise have in roles via "include_recipe"
> > > * your top-level cookbook for sure has a version, as any other cookbook 
> > > too
> > > * in your environments you only pin the top-level cookbook's version 
> > > (all transitive dependencies are pinned via the top-level cookbook's 
> > > metadata)
> > >  
> > > What would be missing from this approach?
> > You give up a significant amount of control over the order in which 
> > cookbooks are run. Everyone using role cookbooks eventually hits the 
> > problem where computed attributes in attributes files have incorrect 
> > values because role cookbooks need dependencies to be loaded after 
> > themselves. This is unavoidable because cookbooks aren’t roles. That 
> > said, I won’t say that role cookbooks are bad per se, I just think Chef 
> > doesn’t give you the tools to compose behaviors without exposing you to 
> > unnecessary risk. Policyfiles fix that.
>  
>  
> ​Nice catch. Indeed it *looks* like you have full control over the odering 
> of cookbooks by just `include_recipe` them in the order you want, but this 
> is just the run order, not the load order that is relevant for the computed 
> attributes.  
>  
> In the meantime I know how to fix computed attributes (see 
> http://docs.opscode.com/chef/essentials_cookbook_recipes.html#reload-attributes)
>  but it's both a) not nice and b) a sure pitfall...  
>  
>  
> >  
> > >  
> > > The only new thing proposed in [2] was the ability to uniquely identify 
> > > non-released development versions. The proposal suggests to use a hash 
> > > over the cookbook contents instead of name + version. I see the need 
> > > for properly identifying in-development versions as well, but IMO using 
> > > a hash would just obscure things. Instead I would rather see prerelease 
> > > and build identifiers being supported by the Chef cookbook versioning 
> > > scheme (see CHEF-4027 [3]) -> as we have for Ruby gems today.
> > What you’re saying isn’t actually how people really use rubygems today. 
> > If I need to run chef-client in development with a development version of 
> > ohai, I don’t release Ohai-7.5.0-dans_crazy_idea.5 to rubygems.org 
> > (http://rubygems.org) (also, you don’t have push rights for ohai on 
> > rubygems.org (http://rubygems.org) so you *can’t* do that), I point my 
> > Gemfile at a path or git source. You should be able to do the same with 
> > Chef and let Chef figure out how to put the pieces together on a remote 
> > node.
>  
> ​
> You are talking about the dependencies of a "top-level" thing here (still 
> have no better name). In the PR linked below these would be `foo`, `bar`, 
> `baz` and `dep_of_bar` which are dependencies of `basic_example`. These 
> might be your own or other people's community cookbooks. And yes, you will 
> likely make modifications to them that are not released or published to 
> rubygems.org (http://rubygems.org) and just live in your fork of the git 
> repo.
>  
> It's still unclear to me how the Policyfile itself would be versioned, 
> published and being referenced (e.g. from within an environment).
>  
>  
> The Policyfile before compilation looks like a combination of Berksfile + 
> Role (see 
> https://github.com/danielsdeleo/chef-workflow2-prototype/blob/master/docs/demo-script.txt#L20-37)
>   
> After compilation the Policyfile.lock has everything we need to uniquely 
> identify these dependencies, looking quite similar to Berksfile.lock + Role
>  
> Can you say that Policyfile ~= Roles + Versioned Dependencies?
Policyfiles provide one of the features of roles, which is to delegate a 
node’s run list to another object. In the uncompiled Policyfile.rb, you’ll be 
able to specify a run list including roles if you want to use roles to define 
composable bits of functionality. These will be expanded when the compiled 
Policyfile.lock is generated which protects you from breaking unrelated 
systems when updating roles.

  
>  
>  
> > As I replied on the ticket, we understand that some random hex string is 
> > pretty meaningless to a human, which is why the Policyfile.lock will 
> > contain a fair amount of contextual information about cookbooks. This 
> > includes the source (local disk, community site, chef server, github), 
> > the semver version of the cookbook, and where relevant, git info such as 
> > the commit SHA, whether or not the repo is dirty, the git remote, and 
> > whether commits are synchronized to the remote. Before you apply this 
> > Policyfile.lock to any nodes, you can review a diff of all of this.
>  
> ​As a human I'm rather concerned on how to edit the uncompiled Policyfile. 
> I guess it's like in a Berksfile where you specify {name + version + 
> source} but then the compiled Policyfile.lock contains​ all the additional 
> info (e.g. the locked git rev for example)?
It’s just like a Berksfile, you ask for cookbooks using the concepts that 
make sense for the systems that store them. For a system that stores released 
artifacts according to a name and X.Y.Z version number, that’s version 
constraints, for git it’s SHA/branch/tag, for your filesystem it’s a path, 
etc.
  
>  
>  
> >  
> > On the ticket, you said 'to me it sounds still like a "build 
> > identifier”’, which it is. chef-server and chef-client will talk to each 
> > other in terms of build identifiers. You also said, 'When talking to my 
> > colleagues I'd personally rather talk about apache-2.0.0 rather than 
> > f59ee7a5bca6a4e606b67f7f856b768d847c39bb’. You can totally have both. 
> > Part of the workflow we’re designing is that you compute the lockfile and 
> > can look at it, commit it to source control if desired, etc. before you 
> > apply it to anything. And the lockfile contains more than enough 
> > information for you to talk about the cookbook in human-understandable 
> > terms. For example, look at the lockfiles in the tests: 
> > https://github.com/opscode/chef-dk/pull/53/files#diff-db3efba452f4c4875514079ef3d4e7f7R320
> >  (there is a tiny bit of indirection there to make the tests more 
> > maintainable). You can talk with your colleagues about “apache 2.0.0 that 
> > came from this github repo” or a cookbook that was uploaded but not 
> > committed to source control, or you can see in a diff that you changed 
> > from the mainline version of a cookbook to your own fork.
> >  
> > And all of this happens automatically. You don’t have to spend your own 
> > time editing version numbers to encode this information. That’s both crap 
> > work and error prone.
>  
> Thanks, that answered a lot.
>  
> In summary, what's still unclear to me is this:
>  
> 1. How do you specify a specific version of a cookbook dependency in the 
> (uncompiled) Policyfile? Simply {name + version + source} like in a 
> Berksfile or work with hashes here already?
You use version constraint operators in the uncompiled Policyfile.rb, just 
like with berks.
  
>  
>  
> 2. How would you specify a version for the top-level `basic_example` thing? 
> Can you assign a dotted numeric X.Y.Z version like for cookbooks, or will 
> it get a computed hash as well?
Policies are probably not going to have explicit versioning support. We want 
to have a strong relation between nodes and policies where a node 
unambiguously belongs to exactly one policy, which when combined with the 
strong relation between policies and cookbooks will make it a lot easier to 
answer questions like “what code will node ‘foo’ run when I run 
chef-client?”. If you want to apply an updated policy to some nodes, you can 
either have an explicit canary group in your process or create a new 
container and migrate nodes from the old one to the new one.
  
>  
>  
> 3. How are the Policyfiles versioned, published and being referenced?
Policies won’t have versions (aside from what you do on your own in your 
version control system). The exact mechanism by which a node is assigned to a 
policy hasn’t been decided yet. We have some prototype code that uses data 
bags to store the policies, see:

https://github.com/opscode/chef/blob/master/lib/chef/policy_builder/policyfile.rb
https://github.com/opscode/chef/blob/master/spec/unit/policy_builder/policyfile_spec.rb
https://github.com/opscode/chef/blob/master/lib/chef/config.rb#L343-351

In the current prototype, you’d name your policies something like 
$functional_role-$deployment_group where $functional_role is something like 
appserver/load balancer/database/etc. and $deployment group could map to your 
environments, groups within environments (for example, if you deploy to 
production on a cluster-by-cluster basis, you could have prod-cluster-a, 
prod-cluster-b, etc.) or whatever makes sense to you. Whether or not the 
deployment group concept becomes a bit more first class when we implement the 
APIs is something we haven’t decided yet.

  
>  
> 4. Can you reference a Policyfile from within an environment?
With policies, all the environment version specification stuff goes away. We 
may keep them around as a place to store attributes, and it’s possible that 
nodes will associate to policies by policyname plus environment. Contrarily, 
we might use a different name for policy containers, or design the system in 
such a way that the “containers” are implicit.
  
>  
> ​
> >  
> > >  
> > > Just saying: let's start simplifying and improve the tools around the 
> > > concepts we currently have, not inventing additional and competing 
> > > concepts that make everything more complex. Reuse the existing 
> > > concepts, establish conventions, and foster them by making sure the 
> > > tools we use promote them.
> >  
> > I don’t think the existing concepts are good enough. I say this as 
> > someone who helped to design them. The way people used and thought about 
> > chef, cookbooks, etc. when we created environments, for example, was 
> > totally different than the understanding we have today. Based on what 
> > we’ve learned from other people and new tools, we can now see a way to 
> > make chef and chef-server work in a better, safer, and more humane way. 
> > So we should.
>  
> ​+1
>  
> I assume (even though not explicitly mentioned) that the new Policyfile 
> mechanism would work for chef-solo as well, does it?
Exactly how it works with chef-solo is to be determined. Since chef-solo gets 
cookbooks from local disk, the question of supporting multiple versions with 
solo is pretty awkward.
  
>  
>  
> Thanks for all the lengthy details, the picture gets clearer... :-)
>  
> Cheers,
> Torben
>  
>  
>  
> > >  
> > > That was quite much to digest... hope that makes still sense :-)
> > >  
> > > Cheers,
> > > Torben
> > >  
> > >  
> > > [1] 
> > > https://github.com/danielsdeleo/chef-workflow2-prototype/blob/master/docs/demo-script.txt
> > > [2] 
> > > https://github.com/danielsdeleo/chef-workflow2-prototype/blob/master/docs/design-principles.md
> > > [3] https://tickets.opscode.com/browse/CHEF-4027
> >  
> >  
> > --
> > Daniel DeLeo
>  



--  
Daniel DeLeo