[chef] Re: Re: Re: Re: Re: Re: Re: [chef-dev] Re: ideas for chef-hackday at Chefconf, Tuesday afternoon, May 15

[Jay Feldblum < >]

Bryan,

Great question. Indeed, because Bundler rocked the gems world, @hedgehog kicked off the idea of bundling up your cookbooks (props!) with his fork of Bundler and a convention for treating cookbooks as gems (http://hedgehogshiatus.com/carving-chefs-cookbooks). This was a great solution, especially in Chef's early days, and was an inspiration for me thinking that we should be bundling the cookbooks we use.

But cookbooks are not gems. They don't have lib directories, the gemspecs do not expose enough of the metadata that cookbooks provide, they would conflict with actual gems, etc. The principle is that form must follow function: any packaging/bundling system for cookbooks must follow the actual characteristics of cookbooks in order to be useful. Indeed, this is what I have tried to do with Librarian-Chef: deal with cookbooks as they are while following Bundler's overarching principles (of course, that work is ongoing and far from complete, whereas @hedgehog's solution has been out for a few years already).

Having multiple packaging/bundling systems for different kinds of packagables isn't really a problem to me - at least, the price is worth it. The world is distributed. The things we make and use are replicated and forked. Everybody learns from everybody else. Sure, now there's a lot more to learn and remember, but that's the price you pay for continual innovation everywhere. That's why we have Chef, when there already was a plethora of configuration management systems.

We should certainly package up (and distribute) ohai plugins, knife plugins, and some chef report & exception handlers as gems, though.

Apologies for the length of my replies.

Cheers,

Jay

On Wed, Apr 25, 2012 at 3:47 PM, Bryan Berry < " target="_blank"> > wrote:

dammit as usual, when I disagree w/ Jay, it's only because I misunderstand him ;)

you deserve the Golden Spatula for sure!

this also begs the question of why we don't package our cookbooks as ruby gems. I am not familiar enough with gems to know their weaknesses. Adam Jacob told me on Twitter once that "we can do better" than gems, and I hope he is right.

On Wed, Apr 25, 2012 at 9:34 PM, Jay Feldblum < " target="_blank"> > wrote:

Bryan,

The original structure I outlined was a proposed structure for a single cookbook repo - a repo whose purpose is the development and testing of a single cookbook -, and was not a structure for a repo hosting multiple cookbooks or for a full infrastructure.

In that proposed structure, the single "core" cookbook lives in a subdirectory, while the surrounding boilerplate/tests/caches live either at the top-level (e.g. the Rakefile) or in different subdirectories (e.g. the test cases). That structure means you can pull out and tarball a single subdirectory of the cookbook's repo in order to capture the self-contained cookbook, while skipping the boilerplate/tests/caches for the cookbook which are outside of that single subdirectory.

There would of course be no roles, environments, data-bags, etc., in that proposed structure, since it's for developing & testing a single cookbook in isolation rather than developing & testing an infrastructure.

The proposed structure mirrors the usual structure for most Ruby gems, but replacing lib/GEM-NAME/ with cookbook/COOKBOOK-NAME/ (and replacing Gemfile with Cheffile for those who are so inclined).

Cheers,

Jay

On Wed, Apr 25, 2012 at 2:55 PM, Bryan Berry < " target="_blank"> > wrote:

I feel pretty strongly that everything required to test a cookbook
live within the cookbook's git repository, that each cookbook has to
be its own project. Isn't every gem its own project?

Once something is common to various parts of your infrastructure, the
chance that it will be shared to the broader Chef community rapidly
approaches zero. This is just my experience. I have shared multiple
cookbooks but yet to share a single role, data_bag, environment or
even knife script w/ the broader Chef community.

Let me know if I have misunderstood you.

On Wed, Apr 25, 2012 at 8:47 PM, Jay Feldblum < " target="_blank"> > wrote:
> Bryan,
>
> Thanks for the comments.
>
> The way I was thinking of it is that the individual cookbook is embedded in
> a larger surrounding project, so that the "core" cookbook directory could be
> pulled out and zipped up without much logic concerning which
> files/directories belong to the "core" cookbook and which files/directories
> belong to the larger surrounding project. This would make the work of a
> "rake upload" type task much easier.
>
> If, instead, everything is at the toplevel, that will require extra logic
> when zipping and pushing a release. But it will make the project structure
> simpler.
>
> I wouldn't embed everything deep down in cookbook/COOKBOOK-NAME; unless
> there's something I'm missing, I would either embed just the "core" cookbook
> there or keep everything at the top-level. This is for repos containing
> exactly one cookbook, following the repo-per-cookbook pattern.
>
> Either way, let me know how easy/hard it is to integrate Vagrant & Toft with
> the Cheffile (in case you or others plan to have one), and based on your
> experimental results what can be done to improve that integration.
>
> One thing I am considering is a metadata DSL method that would read the
> metadata.rb from the same directory, so you wouldn't have to spell out all
> the dependencies in the metadata.rb and then do so again in the Cheffile.
> This would follow the pattern of Bundler's gemspec DSL method. I can see
> this being useful for the Cheffile belonging to an individual cookbook repo.
>
>     site "http://community.opscode.com/api/v1"
>
>     # apply all the dependencies listed in ./metadata.rb
>     # these all use the default source listed above
>     metadata
>
>     # override the source for one of the cookbooks listed in ./metadata.rb
>     # e.g. in case we need a specific fix when developing & testing the
> cookbook
>     cookbook "apache2",
>         :git => "https://github.com/opscode-cookbooks/apache2",
>         :ref => "be12fede2e99aff3029360df38366a3b8a34171f"
>
> Cheers,
> Jay
>
> On Wed, Apr 25, 2012 at 2:14 PM, Bryan Berry < " target="_blank"> > wrote:
>>
>> hmm, I have something different in mind
>>
>> Jay, like we talked on the podcast, I think cookbooks need to be
>> self-contained. Here is how I think it should be laid out
>>
>> I definitely think that the Vagrantfile, Toftfile (my own invention),
>> and Cheffile should live at the cookbook/COOKBOOK-NAME/  level
>>
>> cookbook/COOKBOOK-NAME/
>>                                              attributes/
>>                                              definitions/
>>                                              files/
>>                                              libraries/
>>                                              providers/
>>                                              recipes/
>>                                              resources/
>>                                              temp/      # cache
>> cookbooks sourced by Cheffile
>>                                              tests/              #
>> tests or test? don't know the convention
>>                                                        minitest/
>>
>> recipe1/
>>
>>          some_test.rb
>>
>>          some_test2.rb
>>
>>  recipe2/
>>
>>         . . .
>>                                                        cucumber/
>>                                                        rspec/
>>                                              Cheffile
>>                                              Cheffile.lock
>>                                              Rakefile
>>                                              Toftfile
>> # same as Vagrantfile but for toft
>>                                              Vagrantfile
>>
>> The Cheffile at the at COOKBOOK-NAME level would be used by
>> Vagrantfile or Toftfile to source the other cookbooks needed to run
>> integration tests.
>>
>> that's my euro 0,02
>>
>> On Wed, Apr 25, 2012 at 7:53 PM, Jay Feldblum < " target="_blank"> >
>> wrote:
>> > It may be wise to structure the Git repository of tested cookbooks
>> > according
>> > to the following convention:
>> >
>> > /
>> >     cookbooks/
>> >     #   .gitignore'd
>> >     #   for cookbooks installed via librarian-chef
>> >     #   this is an assumption librarian-chef currently makes
>> >     cookbook/
>> >         COOKBOOK-NAME/
>> >             attributes/
>> >             definitions/
>> >             files/
>> >             libraries/
>> >             providers/
>> >             recipes/
>> >             resources/
>> >             templates/
>> >             metadata.rb
>> >     test/
>> >         integration/
>> >             some_integration_test.rb
>> >             some_other_integration_test.rb
>> >     .gitignore
>> >     #   includes:
>> >     #       cookbooks/
>> >     #       Cheffile.lock
>> >     Cheffile
>> >     #   to support Vagrantfile
>> >     #   the cookbook under test may depend on other cookbooks
>> >     Cheffile.lock
>> >     #   .gitignore'd
>> >     #   don't add to git in a cookbook repo
>> >     #   only add to git in an infrastructure repo
>> >     Rakefile
>> >     #   has a task for zipping up the content of cookbooks/COOKBOOK-NAME
>> >     #   and optionally for releasing to the Opscode Community Site
>> >     Vagrantfile
>> >     #   if using Vagrant for testing
>> >     #   references cookbooks located in cookbooks/ (dependencies) and
>> > cookbook/ (under test)
>> >
>> > If people think this is a good idea, I may add intelligence to
>> > Librarian-Chef's git source to try to find the cookbook inside
>> > cookbook/COOKBOOK-NAME/ if it can't find the cookbook at the top-level
>> > of
>> > the git repo. However, for now, people can always use the format:
>> >
>> > cookbook "COOKBOOK-NAME",
>> >     :git => "git://github.com/user/COOKBOOK-NAME",
>> >     :path => "cookbook/COOKBOOK-NAME"
>> >
>> > Cheers,
>> > Jay
>> >
>
>