[chef] Re: Re: Re: Re: Re: Berksfile question

[Jamie Winsor < >]

Torben,

I maybe misunderstood what you meant by determining "the best branch to use". I thought you meant that we should go back in history until we hit something in the commit log that contained a metadata.rb with the version number we were looking for. Your suggestion makes more sense now.

We tag our software in Git whenever we make a release, however, it is not part of the SemVer convention do this. It's a good bet that you would see something named "v1.0.0" or "1.0.0", but tagging releases and this format is not a standardized thing. We could create some sort of Github default location which would pull the tar.gzs made available from the presence of tags, but we would have to assume this tagging naming convention (and whatever else pops up that is "common") and there's still the question of "Why?". Do we really need to cling to the idea of Git/Github as the canonical place to pull artifacts from when our nodes do it every single day out of the actual artifact server (The Chef server)?

It's not currently debatable whether a given non-tagged branch should contain a final-looking version number because it's not possible to use anything other than SemVer 1.0 in cookbooks. It's also not a standardized rule to change your version number to the next target release. It's understood that any commits which happen after the version was tagged but before the version in the actual version file is incremented are part of a future release.

To answer your reasons to use Git locations:

1. You can still setup a Chef Server which only serves the purpose of acting as an artifact server

2. Berkshelf was written in a way to enable people to get as much out of it as they wanted. You definitely don't need to follow things 100% but when you talk to the guys who made Berkshelf, you're probably going to get their best advice which happens to be The Berkshelf Way.

3. I am a huge supporter of namespaces, but we just aren't there yet. The fact that Github namespaces them does not mean that the cookbook is actually namespaced. When you upload 1.0.0 of "Me/Apache2" to a Chef server it looks identical to 1.0.0 of "You/Apache2". It would be safer to include the namespacing in the name of the cookbook: "you_apache2" and "me_apache2". We've seen this has been the best and least confusing way to handle forks in a larger organization.

-- 

Jamie Winsor

@resetexistence

https://github.com/reset

On Wednesday, May 22, 2013 at 10:43 PM, Torben Knerr wrote:

Hi Jamie,

See my responses inline:

On May 23, 2013 1:57 AM, "Jamie Winsor" < "> > wrote:
>
> Hey Torben,
>
> Yeah you'd need to specify the branch that you want to use. I can see why it would seem to make sense to determine "the best branch to use", but there's a problem with that. Technically version 1.0.0 may exist as many, many, more commits than just the very first SHA where the metadata file contains the numbers "1.0.0".

That's why it should specifically look for a tag named "v1.0.0" (which this is the convention if you follow SemVer afaik) rather than doing a deep search through all git revisions.

It's debatable whether the master branch (or any other branch that is not final per convention) should contain a final-looking version number:
- In the git repos for some ruby gems I have seen versions like "1.0.1-dev" indicating that you are working *towards* version "1.0.1", but you don't expect "1.0.1-dev" to be stable/final.
- In maven it's the same but the pattern is "1.0.1-SNAPSHOT"
- In Chef that's atm not possible unfortunately, see CHEF-4027 :-/

> This is because artifacts are not "finalized" when they exist in our Git repositories, and this is another reason why it is strongly suggested to avoid using the Git source in a Berksfile to manage each of your  cookbooks.
>
> It is a much better approach to instead point to an artifact server that contains finalized artifacts. In this case the Chef server is actually an artifact server containing finalized artifacts. Berkshelf actually by default freezes cookbooks on upload to further enforce this idea.
>

I see some valid use cases and advantages in using git locations:
1) you might not have an artifact repository (e.g. I'm working mostly chef-solo, thus I don't need/want a chef server)
2) you might not want to follow the Berkshelf way to 100%
3) with git locations you have namespacing! Using 'github: "you/apache2"' clearly identifies this dependency as your fork of the apache2 community cookbook. If you don't specify this in the Berksfile this information gets totally lost and you end up with whatever version/fork is currently happening to be on the chef server

The last one is the killer argument for using git locations for me.

> Since there is at any one time more than one version of a cookbook as identified by the string present in the metadata.rb, it is impossible to properly identify the "best branch to use".
>
> -- 
> Jamie Winsor
> @resetexistence
> https://github.com/reset
>
> On Wednesday, May 22, 2013 at 2:33 PM, Torben Knerr wrote:
>>
>> Hey Jamie,
>>
>> picking up your last example:
>>
>> site :opscode
>> chef_api :Config
>>
>> cookbook 'my-app', github: 'RiotGames/my-app'
>> cookbook 'some-dependency', github: 'RiotGames/some-dependency'
>>
>> If RiotGames/my-app/metadata.rb says:
>>
>>        depends 'some-dependency', '= 1.0'
>>
>> ...then Berkshelf would still try to use the HEAD rev in the master branch, right? 
>>
>> So if `some-dependency` lives in a git repo but not on the community site (e.g. if you maintain a fork of a community cookbook) and you want to get v1.0, then you would have to specify the branch explicitly in that Berksfile, i.e.:
>>
>>     cookbook 'some-dependency', github: 'RiotGames/some-dependency', branch: 'v1.0'
>>
>> Wouldn't it make sense if Berkshelf per convention tries to detect the "best branch to use"?
>>
>> In the above example '= 1.0' this is simple, its actaully a bit more complicated due to the various version specifiers (e.g. '=> 1.0', '~>1.1', etc)
>>
>> Cheers, 
>> Torben 
>>
>>
>>
>>
>> On Wed, May 22, 2013 at 10:15 PM, Jamie Winsor < "> > wrote:
>>>
>>> Hey Mark,
>>>
>>> Given the Berksfile:
>>>
>>> site :opscode
>>> cookbook 'my-app', github: 'RiotGames/my-app'
>>>
>>> This will point to HEAD of the Git repository located at that Github repo. It will reflect on the metadata.rb of that file and use one of the 'default sources' to find, and download, any cookbooks found in the metadata of 'my-app'. Let's say that 'my-app' depended on the NGINX cookbook. Berkshelf would go out to the Opscode community site to find the cookbook because you have only one default source, 'site :opscode'.
>>>
>>> Given the Berksfile:
>>>
>>> site :opscode
>>> chef_api :config
>>>
>>> cookbook 'my-app', github: 'RiotGames/my-app'
>>>
>>> Now we have two default sources. First we will go out to the community site and attempt to find the version of the NGINX cookbook that 'my-app' depends on. If it is not found then we will go to your configured Chef Server using the credentials in  your Berkshelf configuration (or Knife configuration if you don't have a Berkshelf configuration) and attempt to find it there.
>>>
>>> Let's say that 'my-app' depended on a cookbook which is not found on your Chef server or on the Community site, or let's say you want to pull this other dependency that out of Git. You would then have to explicitly list the location of that cookbook in your Berksfile. Here is an example:
>>>
>>> site :opscode
>>> chef_api :Config
>>>
>>> cookbook 'my-app', github: 'RiotGames/my-app'
>>> cookbook 'some-dependency', github: 'RiotGames/some-dependency'
>>>
>>> It is strongly suggested that you avoid managing all of your dependencies through Git and instead rely on the second example I have given you. Instead upload those cookbooks into your Chef Server and allow the chef_api default source to find them.
>>>
>>> -- 
>>> Jamie Winsor
>>> @resetexistence
>>> https://github.com/reset
>>>
>>> On Wednesday, May 22, 2013 at 6:32 AM, Mark Pimentel wrote:
>>>>
>>>> If I specify a git location without a branch and the metadata.rb specifies the version required, does berkshelf pull master and check the version in that dependencies' metadata?
>>>>
>>>> If I want to pull a specific tag, I understand you would have to specify branch: "1.0.0" in berks but then also in metadata?  That seems somewhat redundant.  
>>>>
>>>> Am I missing something?
>>>>
>>>> --
>>>> Thanks,
>>>>
>>>> Mark
>>>
>>>
>>
>