[chef] Re: Re: Re: Re: Centralized cookbook-library repos vs distributed cookbook repos

[dreamcat four < >]

On Sun, Apr 11, 2010 at 1:18 AM, Hedge Hog 
<
 >
 wrote:
>> collectd module. It's not clear how to handle these inter-app
>> dependencies or where they should live.

Hi,
I use git to make my own life with chef recipes relatively painless.
Typcially, there is one reasonable workflow which I repeatedly use:

1) Fork opscode/cookbooks -> Your fork.

2) Rename "your/cookbooks" -> "your/site-cookbooks". Now clear all the
files, so that the site-cookbooks are empty.

3) add the opscode and 37 signals cookbooks as remotes to your local
repo. If you have multiple official or semi-official source repos,
then it may be worth to write a script to help initialize your
remotes. If you sit inside an organisation which has a site-wide
site-cookbooks, then ass that too. We assume here that you are
commiting to a local / individual repository.

4) We also assume that you are developing solely in "site-cookbooks".
And that underlying "cookbooks" is kept as "opscode/cookbooks" (master
branch). So we are never are comiting directly to the "cookbooks"
base. Only the "site-cookbooks".

3) Checkout git subtrees with `git checkout <remote>/<branch> --
<coobook_name>. They will land up in your local master / working tree.
This way you can positively select those cookbooks you intend need to
edit / manipulate. This can be done on an "as-needed" basis, at any
time the editing of a cookbook is called for. (To only edit a
cookbooks's attributes, we dont need thid. Just override the cookbook
attrs with json fragment / couchdb).

4) Finding the dependencies of the cookbooks you wish to work on. Run
this command in you site-cookbooks repo:  `$ grep -R -1 depends
*/metadata.rb`. This should provide of all of the cookbooks which your
(individually checked-out) selected cookbooks depend upon. Of course,
once you checkout those direct dependencies, your git tree will become
more */metadata.rb to parse. Hmm.

^^ This is a part of the process that could benefit from some kind of
recursive automation. And admittedly we dont need to actually check
out these files to peek what they are in the git tree. (or even
locally). Git should be flexible enough to also offer other / better
ways to help list a cookbook's dependencies.

5) Work on your cookbooks. When the time is right, then merge, and
push back to the higher repos as required. One of your git remotes
will also be your production / test servers. So it should be a
faimiliar git process.

To me, this workflow is reasonable and workable. Currently I perform
all of these steps manually on the commandline. Perhaps a rubygem
could help to automate some of the steps, namely setting up the list
of remotes, tracking branch, and recurisively finding / checking out
all a cookbook's dependencies. Just to make it all a bit more
consistent, and fewer repetitive steps.

To find the cookbook dependencies automatically:

If we want to use the "depends(cb)" declarations from the metadata.rb
file, we would need to somehow ensure that the dependencies are
allways being properly listed in there. Currently chef does not work
that way. If there are missing depends(cb) dependencies, chef will
continue to run a cookbook, and not raise any error. Chef also has the
"include_recipe()" directive which is used inside the
"recipes/recipe.rb" files. So I guess we could alternatively use a
more complex search algorithm to dig those up directly. Or make the
"include_recipe(cb)" fail unless its also accompanied by a
"depends(cb)" in the metadata.rb. I dont know, perhaps the chef
library already has some better way to figure this out.

However in chef-world, chef isnt aware of multiple cookbooks
repositories in a way that lets 2 cookbooks with the same name
co-exist. In part this is a feature, as it allows you to override
arbitrary files of a cookbook in your other git repo. However its not
inconceivable to think that 37's cookbook called "apache" might be
utterly diverged from the generic opscode version. If both such
popular cookbooks repos (the 37 signals and opscode) are to be
included as (default) remotes / cookbook sources, then we might have a
problem. Wouldnt you kindda need to scope / namespace for the
situation where 2 different upstream cookbooks share the same name?
Such scoping may be gleaned from various information lying around in
the cookbooks eco system. I wouldnt know if it could be make into a
de-facto standard. Such that a commandline tool (a "repo-helper-gem")
could rely upon it.

Which precise mechanisms would people want to support and maintain as
a standard? How would you decide to sacrifice some of the existing
flexibility in order to get those new features? And of course, thats
assuming that enough people would want to manage their cookbooks with
that exact same git workflow. There are many other git workflows too.


dreamcat4