[chef] Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Centralized cookbook-library repos vs distributed cookbook repos

[Hedge Hog < >]

On Thu, Apr 15, 2010 at 7:04 AM, Adam Jacob 
<
 >
 wrote:
> On Wed, Apr 14, 2010 at 1:08 PM, Alex Soto 
> <
 >
>  wrote:
>> My comment was triggered by you mentioning that:
>>
>> The assumption that everyone in the chain is using Git, for example,
>> is a false one.  Most of us are - but not all of us, and if someone
>> wants to use Launchpad and Mercurial for source control, I want to
>> include them, not force them to use a different source control system.
>>
>> I read too much into that to think you meant to support all these other 
>> scm's which is not necessarily what you meant.  My bad, I should have 
>> re-read things.  My off the cuff remark, was because it just raised the 
>> flag in my head to point out that we don't want to over engineer this to 
>> make things super flexible.  keep it simple :)
>
> Absolutely.  The pattern I proposed in an earlier message was using
> 'vendor branches', which exist in every VCS since CVS, to deal with
> tracking your local changes against an up-stream release.

This approach still preserves different cookbook-libraries as separate silo's.
Simply becasue it still ties dependency management to repo organisation.

> The
> integration with various source code control systems is easy here,
> since it's just 'implement that VCS systems vendor branch pattern'.
>
> Check out the source here for an example:
>
> http://github.com/adamhjk/chef/blob/master/chef/lib/chef/knife/cookbook_site_vendor.rb
>
> You can imagine how that could get re-factored to support multiple VCS
> systems without much pain.
>

OK, this is getting clearer to me :)

I think we are getting side tracked by the fact that cookbook library
repo organization is being used to proxy for a dependency management
solution.
However, library repos are being justified/defended as easy to
maintain in terms of VCS effort and _not_ dependency management
effort!
In my mind the two activities should be separate but have become conflated :(
I don't think VCS used has anything to do with dependency management.

In reality the real reason for prefering library repos is _not_ that
library repos are easier to _maintain_, rather cookbook dependencies
are easier to _enforce_.
Changing to cookbook repos will raise the same issues in CVS, hg, etc.
- If you are not going to enforce cookbook-library silos, how do you
manage cookbook dependencies?

The cost of cookbook library repos is giving up distributed cookbook
development and maintenance/admin burdens, having cookbook library
silos develop indepently, etc all outlined in earlier emails
The benefit  of cookbook library repos is trivial dependency magement:
Solve dependencies by getting all cookbooks from this cookbook-library
and no others.

The proposal to elevate cookbooks to repos is to try and create a low
pain path for cookbook contributions to flow _around_ the community,
rather than just up-and-down different cookbook-library silos.
To do this we'd need a dependency management approach that works for
Chef's users and use cases.

>> With respect to the developer workflow:
>>
>> Here's my specific problem.
>>
>> My previous chef deployment used my own chef server, which was cool 
>> because it allowed me to have multiple cookbook repos that the server 
>> pulled from.  I maintained an internal repo that had my specific cookbooks 
>> for my infrastructure and I had a fork of the opscode repo that I could 
>> declare dependencies on from my 'internal repo'.  My fork of the opscode 
>> repo also allowed me to update those cookbooks when I found problems.
>>
>> Now I'm trying to use the platform (I'm still just barely scratched the 
>> surface of it, so I may be mistaken with how I'm using it), but it looks 
>> like I have effectively one cookbook repo which is the 'platform' where I 
>> have to push cookbooks into.  So if I want to use some opscode cookbooks, 
>> I have to copy them and push them up individually.  Because the 
>> granularity of the repo is not at the cookbook level, I don't see how I 
>> could have just a copy of an individual opscode cookbook and still track 
>> upstream changes easily.
>
> The message I posted earlier is my answer to this - check it out and
> see if it works.  The idea is that you would still have your own git
> repository full of all your own cookbooks (you want it this way,
> because it lets you rebuild everything from scratch with only a copy
> of what you care about,) that you're publishing to the platform or a
> chef server.
>
> It lets you track the upstream easily, and provides a more flexible
> way for you to have custom modifications that travel forward in your
> repository.  I think it's pretty close.
>
>> I 'think' I like the one repo per cookbook strategy because those are the 
>> logical modules I think of when developing cookbooks/recipes.  Then I 
>> would be able to fork and submodule them into my repo.  However there is a 
>> nagging feeling that makes me feel that's a bit too granular and may get 
>> cumbersome to manage.
>
> It would be significantly harder to manage for the maintainers of a
> collection of complex, interdependent cookbooks.
>

Maybe.
Initially, there will be some work to do in resolving dependencies
when merging some cookbook's contents from libraries that are
currently treated as 'off-limits'.
But that work will be there _only_ if you _choose_ to change your
cookbooks, or accept changes into them.
Additional cookbook repo admin can be pushed out to cookbook
maintainers, and they will only make the effort to bring in changes if
they think they add value.
Updating and pushing a library of cookbook submodules involves 3 git commands

       $ # git commit all cookbooks changes in superproject
       $ git add .
       $ git commit -m "Updated submodules."
       $ # git submodule update to push change to the individual
repositories that predate the superproject.
       $ git push  # only if you have write access

Worth the effort if there are a wider set of cookbook maintainers to
pick and choose from - of course they would have to have described
their cookbook's dependencies correctly - usual open-source issues.

So it seems that the debate over cookbook repo approach really has
little (nothing?) to do with the effort involved in managing repos and
everything to do with the effort required by the dependency magaement
solution adopted.
Which really should be discussed in a separte thread:
'Freeing cookbook dependency management from source repo organization'

Thoughts?

Regards

> Adam
>
> --
> Opscode, Inc.
> Adam Jacob, CTO
> T: (206) 508-7449 E: 
> 
 
>



-- 
πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
  Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://wiki.hedgehogshiatus.com