[chef] Re: Re: Re: Re: Re: Re: Re: Re: [chef-dev] Re: ideas for chef-hackday at Chefconf, Tuesday afternoon, May 15

[Hedge Hog < >]

On Thu, Apr 26, 2012 at 6:32 AM, Jay Feldblum 
<
 >
 wrote:
> Bryan,
>
> Great question. Indeed, because Bundler rocked the gems world, @hedgehog
> kicked off the idea of bundling up your cookbooks (props!) with his fork of
> Bundler and a convention for treating cookbooks as
> gems (http://hedgehogshiatus.com/carving-chefs-cookbooks). This was a great
> solution, especially in Chef's early days, and was an inspiration for me
> thinking that we should be bundling the cookbooks we use.
>
> But cookbooks are not gems. They don't have lib directories, the gemspecs do
> not expose enough of the metadata that cookbooks provide, they would
> conflict with actual gems, etc. The principle is that form must follow
> function: any packaging/bundling system for cookbooks must follow the actual
> characteristics of cookbooks in order to be useful. Indeed, this is what I
> have tried to do with Librarian-Chef: deal with cookbooks as they are while
> following Bundler's overarching principles (of course, that work is ongoing
> and far from complete, whereas @hedgehog's solution has been out for a few
> years already).
>

Hi,
Just to recap and clarify, from the early discussions it transpired that:

 - Collective/European style cookbooks were used initially to warranty
inter-cookbook compatibility, i.e. it was the only workaround for the
fact that the 37signals apache cookbook was not guaranteed to work
with the opscode varnish cookbook, etc. etc.

 - Individual/American style cookbooks needed a dependency resolution
mechanism.  Bundler seemed the natural fit.  However, no criticism
implied, Bundler's core team does not have git sourced repos as a
priority (and have now clearly indicated, e.g. see issue #67 among
others).  After a few months it became clear that Bundlers test spec
suite made it impossible to neatly fix its Git support, and Bundler
dev's had no pressing need to change that.  To be clear the use of
gemspecs was simply to try to move the version/dependency information
into Bundlers scope - not because cookbooks map well to gems.  They
definitely do not.

- Librarian is where all the real action is :).  It is the users
responsibility to ensure that the cookbook repos+branches they feed to
Librarian play-nice together.  This, along with the fact Librarian is
not trying to use Bundler's code means gemspec files are unnecessary -
unless you want to feed them into some rugygems type server.  But that
is a cookbook visibility issue that Github solves, as does Opscode's
community portal, or any other git server UI.

- The github cookbooks account[0] is intended to be a multi-vendor
umbrella where people try to ensure that the cookbooks under that
account work together, see[1].  Right now it hasn't set the world
alight ;) and needs some love.  My suspicion is, like the Bundler
idea, this is very early and adoption might increase over time as
cookbook vendors specialize in various stacks, or non-trivial
opensoure projects start producing cookbooks, and they want/need their
cookbooks to 'just-work' with other specialist vendor/project
cookbooks as well as cookbooks from more general shops like Opscode.

HTH

[0]: https://github.com/cookbooks
[1]: https://github.com/cookbooks/about


> Having multiple packaging/bundling systems for different kinds of
> packagables isn't really a problem to me - at least, the price is worth it.
> The world is distributed. The things we make and use are replicated and
> forked. Everybody learns from everybody else. Sure, now there's a lot more
> to learn and remember, but that's the price you pay for continual innovation
> everywhere. That's why we have Chef, when there already was a plethora of
> configuration management systems.
>
> We should certainly package up (and distribute) ohai plugins, knife plugins,
> and some chef report & exception handlers as gems, though.
>
> Apologies for the length of my replies.
>
> Cheers,
> Jay
>
>
> On Wed, Apr 25, 2012 at 3:47 PM, Bryan Berry 
> <
 >
>  wrote:
>>
>> dammit as usual, when I disagree w/ Jay, it's only because I misunderstand
>> him ;)
>>
>> you deserve the Golden Spatula for sure!
>>
>> this also begs the question of why we don't package our cookbooks as ruby
>> gems. I am not familiar enough with gems to know their weaknesses. Adam
>> Jacob told me on Twitter once that "we can do better" than gems, and I hope
>> he is right.
>>
>>
>> On Wed, Apr 25, 2012 at 9:34 PM, Jay Feldblum 
>> <
 >
>> wrote:
>>>
>>> Bryan,
>>>
>>> The original structure I outlined was a proposed structure for a single
>>> cookbook repo - a repo whose purpose is the development and testing of a
>>> single cookbook -, and was not a structure for a repo hosting multiple
>>> cookbooks or for a full infrastructure.
>>>
>>> In that proposed structure, the single "core" cookbook lives in a
>>> subdirectory, while the surrounding boilerplate/tests/caches live either 
>>> at
>>> the top-level (e.g. the Rakefile) or in different subdirectories (e.g. the
>>> test cases). That structure means you can pull out and tarball a single
>>> subdirectory of the cookbook's repo in order to capture the self-contained
>>> cookbook, while skipping the boilerplate/tests/caches for the cookbook 
>>> which
>>> are outside of that single subdirectory.
>>>
>>> There would of course be no roles, environments, data-bags, etc., in that
>>> proposed structure, since it's for developing & testing a single cookbook 
>>> in
>>> isolation rather than developing & testing an infrastructure.
>>>
>>> The proposed structure mirrors the usual structure for most Ruby gems,
>>> but replacing lib/GEM-NAME/ with cookbook/COOKBOOK-NAME/ (and replacing
>>> Gemfile with Cheffile for those who are so inclined).
>>>
>>> Cheers,
>>> Jay
>>>
>>> On Wed, Apr 25, 2012 at 2:55 PM, Bryan Berry 
>>> <
 >
>>> wrote:
>>>>
>>>> I feel pretty strongly that everything required to test a cookbook
>>>> live within the cookbook's git repository, that each cookbook has to
>>>> be its own project. Isn't every gem its own project?
>>>>
>>>> Once something is common to various parts of your infrastructure, the
>>>> chance that it will be shared to the broader Chef community rapidly
>>>> approaches zero. This is just my experience. I have shared multiple
>>>> cookbooks but yet to share a single role, data_bag, environment or
>>>> even knife script w/ the broader Chef community.
>>>>
>>>> Let me know if I have misunderstood you.
>>>>
>>>> On Wed, Apr 25, 2012 at 8:47 PM, Jay Feldblum 
>>>> <
 >
>>>> wrote:
>>>> > Bryan,
>>>> >
>>>> > Thanks for the comments.
>>>> >
>>>> > The way I was thinking of it is that the individual cookbook is
>>>> > embedded in
>>>> > a larger surrounding project, so that the "core" cookbook directory
>>>> > could be
>>>> > pulled out and zipped up without much logic concerning which
>>>> > files/directories belong to the "core" cookbook and which
>>>> > files/directories
>>>> > belong to the larger surrounding project. This would make the work of
>>>> > a
>>>> > "rake upload" type task much easier.
>>>> >
>>>> > If, instead, everything is at the toplevel, that will require extra
>>>> > logic
>>>> > when zipping and pushing a release. But it will make the project
>>>> > structure
>>>> > simpler.
>>>> >
>>>> > I wouldn't embed everything deep down in cookbook/COOKBOOK-NAME;
>>>> > unless
>>>> > there's something I'm missing, I would either embed just the "core"
>>>> > cookbook
>>>> > there or keep everything at the top-level. This is for repos
>>>> > containing
>>>> > exactly one cookbook, following the repo-per-cookbook pattern.
>>>> >
>>>> > Either way, let me know how easy/hard it is to integrate Vagrant &
>>>> > Toft with
>>>> > the Cheffile (in case you or others plan to have one), and based on
>>>> > your
>>>> > experimental results what can be done to improve that integration.
>>>> >
>>>> > One thing I am considering is a metadata DSL method that would read
>>>> > the
>>>> > metadata.rb from the same directory, so you wouldn't have to spell out
>>>> > all
>>>> > the dependencies in the metadata.rb and then do so again in the
>>>> > Cheffile.
>>>> > This would follow the pattern of Bundler's gemspec DSL method. I can
>>>> > see
>>>> > this being useful for the Cheffile belonging to an individual cookbook
>>>> > repo.
>>>> >
>>>> >     site "http://community.opscode.com/api/v1";
>>>> >
>>>> >     # apply all the dependencies listed in ./metadata.rb
>>>> >     # these all use the default source listed above
>>>> >     metadata
>>>> >
>>>> >     # override the source for one of the cookbooks listed in
>>>> > ./metadata.rb
>>>> >     # e.g. in case we need a specific fix when developing & testing
>>>> > the
>>>> > cookbook
>>>> >     cookbook "apache2",
>>>> >         :git => "https://github.com/opscode-cookbooks/apache2";,
>>>> >         :ref => "be12fede2e99aff3029360df38366a3b8a34171f"
>>>> >
>>>> > Cheers,
>>>> > Jay
>>>> >
>>>> > On Wed, Apr 25, 2012 at 2:14 PM, Bryan Berry 
>>>> > <
 >
>>>> > wrote:
>>>> >>
>>>> >> hmm, I have something different in mind
>>>> >>
>>>> >> Jay, like we talked on the podcast, I think cookbooks need to be
>>>> >> self-contained. Here is how I think it should be laid out
>>>> >>
>>>> >> I definitely think that the Vagrantfile, Toftfile (my own invention),
>>>> >> and Cheffile should live at the cookbook/COOKBOOK-NAME/  level
>>>> >>
>>>> >> cookbook/COOKBOOK-NAME/
>>>> >>                                              attributes/
>>>> >>                                              definitions/
>>>> >>                                              files/
>>>> >>                                              libraries/
>>>> >>                                              providers/
>>>> >>                                              recipes/
>>>> >>                                              resources/
>>>> >>                                              temp/      # cache
>>>> >> cookbooks sourced by Cheffile
>>>> >>                                              tests/              #
>>>> >> tests or test? don't know the convention
>>>> >>                                                        minitest/
>>>> >>
>>>> >> recipe1/
>>>> >>
>>>> >>          some_test.rb
>>>> >>
>>>> >>          some_test2.rb
>>>> >>
>>>> >>  recipe2/
>>>> >>
>>>> >>         . . .
>>>> >>                                                        cucumber/
>>>> >>                                                        rspec/
>>>> >>                                              Cheffile
>>>> >>                                              Cheffile.lock
>>>> >>                                              Rakefile
>>>> >>                                              Toftfile
>>>> >> # same as Vagrantfile but for toft
>>>> >>                                              Vagrantfile
>>>> >>
>>>> >> The Cheffile at the at COOKBOOK-NAME level would be used by
>>>> >> Vagrantfile or Toftfile to source the other cookbooks needed to run
>>>> >> integration tests.
>>>> >>
>>>> >> that's my euro 0,02
>>>> >>
>>>> >> On Wed, Apr 25, 2012 at 7:53 PM, Jay Feldblum 
>>>> >> <
 >
>>>> >> wrote:
>>>> >> > It may be wise to structure the Git repository of tested cookbooks
>>>> >> > according
>>>> >> > to the following convention:
>>>> >> >
>>>> >> > /
>>>> >> >     cookbooks/
>>>> >> >     #   .gitignore'd
>>>> >> >     #   for cookbooks installed via librarian-chef
>>>> >> >     #   this is an assumption librarian-chef currently makes
>>>> >> >     cookbook/
>>>> >> >         COOKBOOK-NAME/
>>>> >> >             attributes/
>>>> >> >             definitions/
>>>> >> >             files/
>>>> >> >             libraries/
>>>> >> >             providers/
>>>> >> >             recipes/
>>>> >> >             resources/
>>>> >> >             templates/
>>>> >> >             metadata.rb
>>>> >> >     test/
>>>> >> >         integration/
>>>> >> >             some_integration_test.rb
>>>> >> >             some_other_integration_test.rb
>>>> >> >     .gitignore
>>>> >> >     #   includes:
>>>> >> >     #       cookbooks/
>>>> >> >     #       Cheffile.lock
>>>> >> >     Cheffile
>>>> >> >     #   to support Vagrantfile
>>>> >> >     #   the cookbook under test may depend on other cookbooks
>>>> >> >     Cheffile.lock
>>>> >> >     #   .gitignore'd
>>>> >> >     #   don't add to git in a cookbook repo
>>>> >> >     #   only add to git in an infrastructure repo
>>>> >> >     Rakefile
>>>> >> >     #   has a task for zipping up the content of
>>>> >> > cookbooks/COOKBOOK-NAME
>>>> >> >     #   and optionally for releasing to the Opscode Community Site
>>>> >> >     Vagrantfile
>>>> >> >     #   if using Vagrant for testing
>>>> >> >     #   references cookbooks located in cookbooks/ (dependencies)
>>>> >> > and
>>>> >> > cookbook/ (under test)
>>>> >> >
>>>> >> > If people think this is a good idea, I may add intelligence to
>>>> >> > Librarian-Chef's git source to try to find the cookbook inside
>>>> >> > cookbook/COOKBOOK-NAME/ if it can't find the cookbook at the
>>>> >> > top-level
>>>> >> > of
>>>> >> > the git repo. However, for now, people can always use the format:
>>>> >> >
>>>> >> > cookbook "COOKBOOK-NAME",
>>>> >> >     :git => "git://github.com/user/COOKBOOK-NAME",
>>>> >> >     :path => "cookbook/COOKBOOK-NAME"
>>>> >> >
>>>> >> > Cheers,
>>>> >> > Jay
>>>> >> >
>>>> >
>>>> >
>>>
>>>
>>
>



-- 
πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
  Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://hedgehogshiatus.com