[chef] Re: Re: Re: Re: Re: Re: Re: RE: Single Repo vs Repo per cookbook


Chronological Thread 
  • From: Ranjib Dey < >
  • To: " " < >
  • Subject: [chef] Re: Re: Re: Re: Re: Re: Re: RE: Single Repo vs Repo per cookbook
  • Date: Tue, 14 Apr 2015 16:16:10 -0700

at PagerDuty, we use a mix of both approaches. Our main infra repo hosts all cookbooks inside site-cookbooks, we consume a lot of community cookbooks. Berkshelf assemble all of these for us. Local cookbooks are assembled via looping over Dir['site-cookbooks/*'] and the declaring path.

Over the time, we have faced several issues including versioning, circular dependencies, performance and vendoring. But none of them was show stopper. Above all, though opinionated berkshelf authors never left known bugs open, or limit the tools usage (\m/). But this also means you should do some tooling around berks vendoring. i.e. never point any cookbooks upload machinary against site-cookbooks. All should point to berks vendor, and keep the vendoring process ephemeral. i.e nuke and re-create the folder before every cookbook upload process (we do it via CI or chatbots).

My recommendation will be to start with the single repo approach and fan out as and when need, if they are being consumed by other groups/tools beyond the initial monolithic repo. That way you can incrementally fine tune your own versioning policy/convention etc, as and when you encounter problems. If you break the repo upfront, it will add lot more cognitive load as u'll switch back and forth between repos to get anything useful. Which is the case unless you are maintaining community cookbooks. Most of the internal cookbooks generally makes sense together, not in isolation. when we write them we think of addressing a specific use case [like ldap server to back service foo], we dont really start with the requirement of 'lets write a cookbook for ldap which can be re-used across a wide range of platform'. The  re-usable bit involves lot more discovery, and needs lot more maintenance (you have to think of chef version specific features etc)

hope this help
ranjib    

On Tue, Apr 14, 2015 at 3:32 PM, Yoshi Spendiff < " target="_blank"> > wrote:
Thanks Mike, I didn't know about the path option actually.

Though In my case I would keep a Berksfile in each cookbook still, which is another way of doing it.

On Tue, Apr 14, 2015 at 3:09 PM, Mike < " target="_blank"> > wrote:
As to the "how does berks work with a monolithic repo with multiple cookbooks", here's a couple of lines from a Berksfile we use that points to a github repo that has a bunch of cookbooks in it, and are referenced by paths.

    # External cookbooks to <THIS> repo
    cookbook 'dd-base', git: ' :DataDog/ourmonolithicrepo.git', branch: 'master', rel: 'site-cookbooks/dd-base'
    cookbook 'dd-consul', git: ' :DataDog/ourmonolithicrepo.git', branch: 'prod', rel: 'site-cookbooks/dd-consul'

     # cookbooks in <THIS> repo, relative to the Berksfile
     cookbook 'workspace', path: './site-cookbooks/workspace'

Does this clutter up the local ~/.berkshelf/ cache directory with multiple copies of the same git repo? Maybe. But who cares? It works. :)

It's pretty flexible - so as long as you figure out "when does my team need to create a new cookbook repo" vs "when does my team need to create a new cookbook in the existing repo" - that's a workflow problem, and berks will play nice either way.

-M


On Tue, Apr 14, 2015 at 7:45 PM, Yoshi Spendiff < " target="_blank"> > wrote:
Yeah that's the thought train I was heading down too, which is why I was wondering if anyone had any experience to the contrary. There's obvious benefits to having cookbooks as independent objects but that doesn't mean you need to follow the whole preachy philosophy.

On Tue, Apr 14, 2015 at 11:27 AM, Lamont Granquist < " target="_blank"> > wrote:

I've never needed to point into a multiple-cookbook repo so not familiar with using the rel: option.  I'd still argue that you shouldn't need to do that, but yeah that renders that question moot.

So, really it sounds like Berkshelf fully supports this method of arranging cookbooks.  Period.  Full Stop.  You do not need to rearrange your cookbooks into single git repos to use Berkshelf.

The problem is that Berkshelf authors and users were noisy about the single git repo model being superior, and that's been conflated with the tool not being usable if you didn't drink all the kool aid.


On 4/14/15 11:15 AM, Yoshi Spendiff wrote:
Isn't that only a problem if you have multiple repos with multiple cookbooks?  If you've got one repo with all your role and wrapper cookbooks, and then peel off individual library cookbooks which are one-cookbook-per-repo then it should work okay and you can just use `path: ../some_cookbook` in the Berksfile for the internal dependencies of the monolithic repo, and avoid having the library cookbooks having deps back into your monolithic repo (which would smell like you're starting to create circular dependencies).

Berkshelf should work fine for this model as well.

I think this is a bit of a moot point isn't it? You CAN actually specify a git repository that has multiple cookbooks, using the rel: option. You just can't do this in a single Berksfile declaration, which is never going to be supported.

I was more asking if you SHOULD rather than if you COULD. It's definitely possible.

On Tue, Apr 14, 2015 at 11:11 AM, Lamont Granquist < " target="_blank"> > wrote:
On 4/13/15 1:17 PM, Kevin Keane Subscription wrote:


The problem with berkshelf is that you can specify a git repository as source per cookbook, but you can't really specify a git repository that contains multiple cookbooks.


Isn't that only a problem if you have multiple repos with multiple cookbooks?  If you've got one repo with all your role and wrapper cookbooks, and then peel off individual library cookbooks which are one-cookbook-per-repo then it should work okay and you can just use `path: ../some_cookbook` in the Berksfile for the internal dependencies of the monolithic repo, and avoid having the library cookbooks having deps back into your monolithic repo (which would smell like you're starting to create circular dependencies).

Berkshelf should work fine for this model as well.




--




--




--




Archive powered by MHonArc 2.6.16.

§