[chef] Re: Re: Re: Re: Re: Re: Centralized cookbook-library repos vs distributed cookbook repos

[Adam Jacob < >]

Okay, in my head, I see two different workflows here:

1) The Consumer Workflow

This is the workflow where you find a cookbook that works for you, and
you want to use it in your infrastructure, and track it in your own
source control (probably a chef-repo.)  You may then want to make
changes to the upstream cookbook, and you then want to be able to
easily see the diff between the changes you have made and any new
versions the upstream has published.

The important thing here is that you care about the cookbook having
it's changes tracked in your local repository (since you use it to
build your infrastructure), but you don't care so much about having
the entirety of the upstream revision history (indeed, you might not
be able to, in the cases where you want to use a different source
control system from the upstream.)  So you want to easily track the
upstream, and apply your changes.

2) The Developer/Collaborator Workflow

In this case, you want to collaborate with the upstream on development
of the cookbook directly - you want your changes to be situated in the
upstream.  To me, this workflow is actually defined by whomever is
managing the upstream - for Opscode, it's tickets and pull requests to
the opscode/chef repo, for 37signals it's different, and for someone
else it may require using mercurial or svn.

I've been thinking about the consumer workflow for a while now, and I
spent some of today working on getting a good pass at getting it down
to essentially a single command.  For the developer/collaborator
workflow, we may also want to make a best practice, but I'm less
concerned about it right now.

The pattern I'm following is called "vendor branching", and it's been
around forever.  The basic method is that you are tracking an upstream
source release, and you want to keep a certain set of patches in sync,
and be able to move between local versions easily.  It was pretty
popular in the CVS days, if any of you can get into the way-back
machine with me.  Here is a good description of doing this with Git:

http://happygiraffe.net/blog/2008/02/07/vendor-branches-in-git/

I've adapted this pattern to be integrated with knife and the
cookbooks.opscode.com site.  If you already have knife configured to
work with your local chef repo, from the current opscode/chef, you can
do:

$ knife cookbook site vendor apache2

And you'll get the latest version of the apache2 cookbook in a local
vendor branch.  You can then make changes directly to the cookbook in
your local repository - you don't need to build a site-cookbook, or do
anything else.  Just make changes like normal.  When the upstream
releases a new version, just repeat the command:

$ knife cookbook site vendor apache2

And the new version will be downloaded, the vendor branch updated, and
then merged into your master branch.  The resulting diff can be easily
viewed, and if there are any merge conflicts, you get a chance to
review them.  If you don't like the changes, you can just 'git reset
--hard HEAD~1'.

You can always get a diff between your current cookbook and the upstream with:


 
 git diff chef-vendor-apache2-0.10.1 HEAD
diff --git a/cookbooks/apache2/attributes/apache.rb
b/cookbooks/apache2/attributes/apache.rb
index c733d5e..d6a8f76 100644
--- a/cookbooks/apache2/attributes/apache.rb
+++ b/cookbooks/apache2/attributes/apache.rb
@@ -75,4 +75,3 @@ set_unless[:apache][:worker][:minsparethreads] = 64
 set_unless[:apache][:worker][:maxsparethreads] = 192
 set_unless[:apache][:worker][:threadsperchild] = 64
 set_unless[:apache][:worker][:maxrequestsperchild] = 0
-# whee, vendor merge.

Check this picture of my local git history out - it shows going from
the 0.9.1 version of the apache2 cookbook to 0.10.1, with a local
change in the middle:

http://skitch.com/adamjacob/n6246/chef-repo-branch-master

Check out this gist for some in-action shots:

http://gist.github.com/365452

In order to make this complete, a few things still need to be done:

1) We need to enable the API for uploading to the cookbook site
easily, to make it simple for cookbook authors to publish their
cookbooks.
2) We need to enable tracking of the upstream source repository in the
cookbook site, along with an optional path, which should enable you to
both vendor HEAD and easily collaborate on development.

What do you think?

Adam

-- 
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: