General discussion about Chef

[chef] Re: Environments vs. Metadata vs. Policyfile for locking cookbook dependencies


Chronological Thread 
  • From: Torben Knerr < >
  • To: Daniel DeLeo < >, " " < >
  • Subject: [chef] Re: Environments vs. Metadata vs. Policyfile for locking cookbook dependencies
  • Date: Fri, 13 Jun 2014 18:16:30 +0200

On Fri, Jun 13, 2014 at 5:06 PM, Daniel DeLeo < " target="_blank"> > wrote:


On Friday, June 13, 2014 at 7:19 AM, Torben Knerr wrote:

> Hi everybody,
>
> we recently started a discussion on the different ways you can lock the cookbook dependency graph for a given node:
>
> 1. use Chef environments
> 2. use metadata.rb
> 3. use Policyfile (work in progress)
>
> The discussion of environments vs. metadata started here on the list...
> http://lists.opscode.com/sympa/arc/chef/2014-05/msg00324.html
>
> ...then continued on a github with @danielsdeleo, proposing a new mechanism called Policyfile:
> https://github.com/berkshelf/berkshelf/issues/724#issuecomment-44980485
>
>
> Looking at the Policyfile approach, I like how the sketched terminal session reads [1], but I'm also afraid that it will add even more possibilities on how to center your workflow around Chef. It would be the next competitor to "roles" vs. "environment cookbooks" vs. "metadata" vs. "policyfile".
Long term, Policyfile should be the default workflow. It eliminates roles vs. role cookbooks as a point of contention by solving the primary flaw of roles, which is the fact that you can mutate them on live production nodes accidentally.


​Sounds good. 

Still I can not see the full picture of the Policyfile yet, so more questions following inline... 

 
>
>
> So I'm wondering: can't we solve the problem with the tools at hand or do we have to invent something new for it?
>
> Almost everything that is described in the desgin principles [2] could be easily solved by using cookbooks + metadata.rb today. You just have to make sure that the contents of Berksfile.lock get translated into metadata.rb depends statements. That's what I was calling "top-level" cookbook but you could also call it "policy cookbook" if you will.
>
> Think of this:
> * for each node you have 1 top-level cookbook
> * this top-level cookbook has all the pinned versions from Berksfile.lock via "depends" statements in its metadata.rb
> * in the the node's run_list is just the default recipe (or any other) of the top-level cookbook, which defines the actual "run_list" that you would otherwise have in roles via "include_recipe"
> * your top-level cookbook for sure has a version, as any other cookbook too
> * in your environments you only pin the top-level cookbook's version (all transitive dependencies are pinned via the top-level cookbook's metadata)
>
> What would be missing from this approach?
You give up a significant amount of control over the order in which cookbooks are run. Everyone using role cookbooks eventually hits the problem where computed attributes in attributes files have incorrect values because role cookbooks need dependencies to be loaded after themselves. This is unavoidable because cookbooks aren’t roles. That said, I won’t say that role cookbooks are bad per se, I just think Chef doesn’t give you the tools to compose behaviors without exposing you to unnecessary risk. Policyfiles fix that.

​Nice catch. Indeed it *looks* like you have full control over the odering of cookbooks by just `include_recipe` them in the order you want, but this is just the run order, not the load order that is relevant for the computed attributes. 

In the meantime I know how to fix computed attributes (see http://docs.opscode.com/chef/essentials_cookbook_recipes.html#reload-attributes) but it's both a) not nice and b) a sure pitfall...



>
> The only new thing proposed in [2] was the ability to uniquely identify non-released development versions. The proposal suggests to use a hash over the cookbook contents instead of name + version. I see the need for properly identifying in-development versions as well, but IMO using a hash would just obscure things. Instead I would rather see prerelease and build identifiers being supported by the Chef cookbook versioning scheme (see CHEF-4027 [3]) -> as we have for Ruby gems today.
What you’re saying isn’t actually how people really use rubygems today. If I need to run chef-client in development with a development version of ohai, I don’t release Ohai-7.5.0-dans_crazy_idea.5 to rubygems.org (also, you don’t have push rights for ohai on rubygems.org so you *can’t* do that), I point my Gemfile at a path or git source. You should be able to do the same with Chef and let Chef figure out how to put the pieces together on a remote node.

​
You are talking about the dependencies of a "top-level" thing here (still have no better name). In the PR linked below these would be `foo`, `bar`, `baz` and `dep_of_bar` which are dependencies of `basic_example`. These might be your own or other people's community cookbooks. And yes, you will likely make modifications to them that are not released or published to rubygems.org and just live in your fork of the git repo.

It's still unclear to me how the Policyfile itself would be versioned, published and being referenced (e.g. from within an environment).


The Policyfile before compilation looks like a combination of Berksfile + Role (see https://github.com/danielsdeleo/chef-workflow2-prototype/blob/master/docs/demo-script.txt#L20-37)
After compilation the Policyfile.lock has everything we need to uniquely identify these dependencies, looking quite similar to Berksfile.lock + Role

Can you say that Policyfile ~= Roles + Versioned Dependencies?


As I replied on the ticket, we understand that some random hex string is pretty meaningless to a human, which is why the Policyfile.lock will contain a fair amount of contextual information about cookbooks. This includes the source (local disk, community site, chef server, github), the semver version of the cookbook, and where relevant, git info such as the commit SHA, whether or not the repo is dirty, the git remote, and whether commits are synchronized to the remote. Before you apply this Policyfile.lock to any nodes, you can review a diff of all of this.

​As a human I'm rather concerned on how to edit the uncompiled Policyfile. I guess it's like in a Berksfile where you specify {name + version + source} but then the compiled Policyfile.lock contains​ all the additional info (e.g. the locked git rev for example)?

 

On the ticket, you said 'to me it sounds still like a "build identifier”’, which it is. chef-server and chef-client will talk to each other in terms of build identifiers. You also said, 'When talking to my colleagues I'd personally rather talk about apache-2.0.0 rather than f59ee7a5bca6a4e606b67f7f856b768d847c39bb’. You can totally have both. Part of the workflow we’re designing is that you compute the lockfile and can look at it, commit it to source control if desired, etc. before you apply it to anything. And the lockfile contains more than enough information for you to talk about the cookbook in human-understandable terms. For example, look at the lockfiles in the tests: https://github.com/opscode/chef-dk/pull/53/files#diff-db3efba452f4c4875514079ef3d4e7f7R320 (there is a tiny bit of indirection there to make the tests more maintainable). You can talk with your colleagues about “apache 2.0.0 that came from this github repo” or a cookbook that was uploaded but not committed to source control, or you can see in a diff that you changed from the mainline version of a cookbook to your own fork.

And all of this happens automatically. You don’t have to spend your own time editing version numbers to encode this information. That’s both crap work and error prone.

Thanks, that answered a lot.

In summary, what's still unclear to me is this:

1. How do you specify a specific version of a cookbook dependency in the (uncompiled) Policyfile? Simply {name + version + source} like in a Berksfile or work with hashes here already?

2. How would you specify a version for the top-level `basic_example` thing? Can you assign a dotted numeric X.Y.Z version like for cookbooks, or will it get a computed hash as well?

3. How are the Policyfiles versioned, published and being referenced?

4. Can you reference a Policyfile from within an environment?

​
 
>
> Just saying: let's start simplifying and improve the tools around the concepts we currently have, not inventing additional and competing concepts that make everything more complex. Reuse the existing concepts, establish conventions, and foster them by making sure the tools we use promote them.

I don’t think the existing concepts are good enough. I say this as someone who helped to design them. The way people used and thought about chef, cookbooks, etc. when we created environments, for example, was totally different than the understanding we have today. Based on what we’ve learned from other people and new tools, we can now see a way to make chef and chef-server work in a better, safer, and more humane way. So we should.


​+1

I assume (even though not explicitly mentioned) that the new Policyfile mechanism would work for chef-solo as well, does it?


Thanks for all the lengthy details, the picture gets clearer... :-)

Cheers,
Torben



 
--
Daniel DeLeo





Archive powered by MHonArc 2.6.16.

§