[chef] Re: Re: Re: Re: Re: in-house berks-api cookbook naming conflicts

[Koert Kuipers < >]

yeah i see the issues with priority, since we also upload general supermarket cookbooks into our private supermarket as part of the process (this is not the case with java, where the private maven repo only holds private artifacts). so since say the yum cookcook is also sitting in our private repo this would get messy quickly: we would never see updates for the yum cookbook since the inhouse always has precedence!

priorities would work well if our private supermarket only held the cookbooks we uploaded to it, not their dependencies. then the separation would be clean and priorities would work. that seems doable.

On Wed, Jan 7, 2015 at 12:11 PM, Justin Dossey < " target="_blank"> > wrote:

From a naive persective it would seem that adding a "priority" keyword to the Berksfile format seems like it would be a simple way to address this issue.  Then, we could say:

source 'https://supermarket.chef.io', priority: 100

source 'https://my-private-supermarket.example.com', priority: 200

and then Berkshelf would choose the first matching cookbook in my-private-supermarket, or, if no matches, look at supermarket.chef.io.

If you were to add such a feature, you open up a much more difficult situation.  The issue that comes up when you abandon the idea of a flat cookbook namespace is dependency resolution on the Chef server.  If cookbook "foo" uses the "apache" cookbook from the community supermarket and cookbook "bar" (in the same chef org) uses the "apache" cookbook from the private supermarket, I don't think there's currently a mechanism to prioritize one over the other in each context. 

So ultimately, this isn't an Berkshelf issue at all. 

On Wed, Jan 7, 2015 at 7:57 AM, Koert Kuipers < " target="_blank"> > wrote:

"path:", "git:" and "github:" always felt strange to me in Berks simply because they point to only one release. this seems to be at odds with the idea of berkshelf, which is to have berkshelf pick the most suitable version based on metadata release constraints.

for example if i release a new version of inhouse cookbook x inhouse that can handle yum >= 3, then i dont want that one to be applied (picked) until all other cookbooks can also handle yum >= 3, which means i want to make it available and let berkshelf pick it when the time is ready. i dont see how i can do that with path:, github: or git: in Berkshelf file.

so to me an in-house supermarket is the way to go. however i certainly do not want to exclude the public supermarket, because that seems way too complex and tedious. it should not be that way.

so right now i am going through the process or renaming all inhouse cookbooks, which is far from easy, since the name is all over the place (comments, attributes, recipes, lwrps, etc.). i find it very frustrating.

it seems only a few very basic things are needed to not have to do this futile renaming exercise: support for release tags such as 0.1.2-mycompany, and maybe a simple blacklist to keep a few cookbooks from public supermarket out if i need to, or a way to set precendence perhaps. however i think just simple release tags would get us 95% of the way there.

note that in java we deal with artifacts and transitive dependencies all the time, and we have in-house versions of public artifacts all the time too, and it certainly does not involve renaming all artifacts and modules.

On Tue, Jan 6, 2015 at 10:45 PM, Koert Kuipers < " target="_blank"> > wrote:

that's an ticket interesting. having SNAPSHOT releases (with upload checks accordingly that allow overwrites) would be a great thing to have as well.

On Tue, Jan 6, 2015 at 6:46 PM, Torben Knerr < " target="_blank"> > wrote:

There was https://tickets.opscode.com/browse/CHEF-4027 once, but it
seems it had not been migrated to the Github issues (at least I
couldn't find it there).

And what's so bad about referencing Git repos from a Berksfile?

It's much more explicit and transparent than overloading namespaces
globally by pointing to a different "source" location.

Cheers,
Torben

On Tue, Jan 6, 2015 at 9:38 PM, Noah Kantrowitz < " target="_blank"> > wrote:
>
> On Jan 6, 2015, at 11:35 AM, Koert Kuipers < " target="_blank"> > wrote:
>
>> renaming internal cookbooks is not really an option, since it would also involve renaming all attributes and it would all together create a giant mess.
>>
>> it seems odd that i am forced to rename cookbook "x" just because someone managed to upload an "x" to the central supermarket, even if the "x" on the central supermarket is barely used.
>>
>> it is also unclear to me how we can go about modifying an existing cookbook. typically we do this and then a forced to temporarily use an in-house version (until our pullreq is accepted and incorporated in a release). for this we do not want to rename the cookbook since that is way too disruptive and the change is only temporary. again this problem would be solved if our inhouse supermarket could take precendence. with that option not available i looked at other alternatives. one would be to reference a git repo, since a git repo can take precendence over supermarket, but everyone says that is bad practice. ok then i thought, what if we modify cookbook "x" version 1.0.0 and publish it inhouse as cookbook "x" version 1.0.0-mycompany, and pin the version we depend on? again no luck, since chef and berks dont support such pre-release versions.
>
> Indeed, the minimal support for complex version numbers is a bit painful here. There are a few major options:
>
> The most robust solution is to stop using "source 'https://supermarket.chef.io'" in your Berksfiles in favor of your own berks-api server pointed at your own Chef server and nothing else. This allows you to create an isolated universe where you control the namespace "entirely". The air quotes are because if you import any cookbooks from the community site, you may run in to issues where things depend on the community version of X instead of your incompatible version of X. This can be worked around at times, but you'll either have to not use community cookbooks as much, or eat the renaming pain on the community cookbook side eventually.
>
> In cases where it is truly a temporary fork, I'll try really hard to keep the logic in a wrapper cookbook via chef-rewind or monkey patching but that isn't always going to be possible. If a fork has to happen, I'll generally set the version to 99.99.99 so it is unlikely to ever collide with a real release version, and then lock that version in the wrapper cookbook or environment (or both).
>
> This in turn sometimes won't work if a ton of other cookbooks outside my control have version dependencies on the things I'm modifying. At this point a lot of people might be tempted to just leave the version number the same or bump it by 0.0.1 and upload to their Chef server, but here be dragons. When using non-git sources, Berks stores cookbooks by "name-version", not content hash, so you risk effectively a self-induced cache poisoning attack where some people might get your modified version in their cache and others might get the unmodified community version, and the only way you can really tell is when things break mysteriously despite using the same Berksfile.lock. If you must do a local fork with the same version number, definitely keep it in the Berksfile as a git reference so you don't clobber your cache.
>
> If you don't like any of these options, sometimes discretion is the better part of valor and you should just wait for a new release. For cookbooks maintained by volunteers on their own time, offering to pay for some development time can greatly accelerate things but tread lightly as some devs might not like that idea. Good luck, you may need it :-)
>
> --Noah
>

--

Justin Dossey

Practice Owner

New Context Services, Inc