[chef] Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Centralized cookbook-library repos vs distributed cookbook repos

[Adam Jacob < >]

On Wed, Apr 14, 2010 at 7:08 PM, Hedge Hog 
<
 >
 wrote:
> This approach still preserves different cookbook-libraries as separate 
> silo's.
> Simply becasue it still ties dependency management to repo organisation.

Yes - the places where a given cookbook lives as a unit of
collaboration is a separate silo.

There is also a dependency management problem, but we have the
ground-work already to solve that.

More on it in a sec.

> OK, this is getting clearer to me :)
>
> I think we are getting side tracked by the fact that cookbook library
> repo organization is being used to proxy for a dependency management
> solution.
> However, library repos are being justified/defended as easy to
> maintain in terms of VCS effort and _not_ dependency management
> effort!
> In my mind the two activities should be separate but have become conflated 
> :(
> I don't think VCS used has anything to do with dependency management.

I agree.

> In reality the real reason for prefering library repos is _not_ that
> library repos are easier to _maintain_, rather cookbook dependencies
> are easier to _enforce_.
> Changing to cookbook repos will raise the same issues in CVS, hg, etc.
> - If you are not going to enforce cookbook-library silos, how do you
> manage cookbook dependencies?
>
> The cost of cookbook library repos is giving up distributed cookbook
> development and maintenance/admin burdens, having cookbook library
> silos develop indepently, etc all outlined in earlier emails
> The benefit  of cookbook library repos is trivial dependency magement:
> Solve dependencies by getting all cookbooks from this cookbook-library
> and no others.
>
> The proposal to elevate cookbooks to repos is to try and create a low
> pain path for cookbook contributions to flow _around_ the community,
> rather than just up-and-down different cookbook-library silos.
> To do this we'd need a dependency management approach that works for
> Chef's users and use cases.

I think there is a third way, which is that we already have a
publishing platform for cookbooks, and the cookbooks themselves have
dependencies.  I've extended the vendor stuff I wrote before, so that
it now downloads and vendors the first cookbook, and then reads the
metadata and repeats the process recursively for all the dependencies:

% ./knife cookbook site vendor wordpress 0.5.0 -d

Now grabs wordpress and all it's dependencies, giving you a tree like this:

% ls -la cookbooks
total 8
drwxr-xr-x   8 adam  adam  272 Apr 15 14:32 ./
drwxr-xr-x  11 adam  adam  374 Apr 15 14:32 ../
-rw-r--r--   1 adam  adam  151 Apr 15 14:32 README
drwxr-xr-x  10 adam  adam  340 Mar 10 16:43 apache2/
drwxr-xr-x  11 adam  adam  374 Apr 15 14:32 mysql/
drwxr-xr-x   7 adam  adam  238 Feb 26 14:35 openssl/
drwxr-xr-x   8 adam  adam  272 Apr 15 14:32 php/
drwxr-xr-x   8 adam  adam  272 Apr 15 14:32 wordpress/

% git branch
  chef-vendor-apache2
  chef-vendor-mysql
  chef-vendor-openssl
  chef-vendor-php
  chef-vendor-wordpress
* master

% git tag
chef-vendor-apache2-0.10.1
chef-vendor-mysql-0.15.0
chef-vendor-openssl-0.1.0
chef-vendor-php-0.7.0
chef-vendor-wordpress-0.5.0

If we start publishing cookbooks and specifying dependencies in
metadata, and extend the cookbooks site to allow the upstream
developers to put the source control end-point for development in,
we'll have versioned cookbooks, with dependency management, along with
the mechanisms required to live on the bleeding edge.  And it'll be
discoverable.

Sound good?

> So it seems that the debate over cookbook repo approach really has
> little (nothing?) to do with the effort involved in managing repos and
> everything to do with the effort required by the dependency magaement
> solution adopted.
> Which really should be discussed in a separte thread:
> 'Freeing cookbook dependency management from source repo organization'
>
> Thoughts?

I think we agree.  There are (at least) 3 things here:

1) How do you discover cookbooks you want to use
2) How do you track them over time, and potentially make site-specific
changes, and track *those* over time.
3) How do you track and resolve the dependencies one cookbook has on another

I think I've got the 'knife cookbook site vendor' stuff far enough
that it does all 3 of these based on the cookbook site information
today, and we can focus on adding in the functionality we need to
cover the remaining use cases.

You?

Adam

-- 
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: