[chef] Re: An deep topic

[Lamont Granquist < >]

On 7/7/14, 8:05 PM, John E. Vincent (lusis) wrote:
> FWIW I didn't have a problem with Berkshelf originally. One: I didn't 
> use it and didn't HAVE to use it. Two: It didn't have this entirely 
> new service it depended on.
>
> Now I can't avoid it and it requires a separate internet service 
> (though now apparently rolled into the supermarket) just to be used.
Just addressing this one issue...

I hated the Berkshelf API with a passion initially.  I still think its a 
hack, but its a hack around problems that needed to be solved.  The 
first problem is just speed.  Berkshelf needs to depsolve against all 
the cookbooks on the community site, and doing that with a single 
request for every cookbook was slow.   So the /universe endpoint was 
created to slurp it all down.  That endpoint is now integrated with the 
community site, and what is going on is essentially exactly what happens 
when you use 'gem install' and it goes and hits rubygems to pull down 
the list of current gems so that the rubygems depsolver can figure out 
what gems to install.  Its that problem, and its now solved in a very 
similar fashion.  There's another obvious source of cookbooks -- the 
chef server -- that you should be able to depsolve against as well, and 
we should add a universe endpoint (probably 
/organizations/<org>/cookbook_universe or something) to the chef-server 
to solve it.

The other problem is one dealing with metadata.rb and metadata.json that 
has its roots in configuration-as-code.  The problem is that if you ship 
around metadata.rb and it has pure-ruby code in it that references 
external data you will not necessarily get the same metadata.json result 
when you evaluate it.  There are two chef tickets on this:

https://tickets.opscode.com/browse/CHEF-4810
https://tickets.opscode.com/browse/CHEF-4811

There's also Jamie's blog post on it:

http://blog.vialstudios.com/the-importance-of-compiled-metadata/

And there's my issue that I opened against berkshelf where you can see 
it tied all together:

https://github.com/berkshelf/berkshelf/issues/1126

The result of that leads to the lack of being able to use transitive 
github dependencies and the need to serve your own API server.  The use 
most important case which is driving it is an important one which is 
that people are using ruby code in metadata.rb to drive their version 
numbers off of CI/CD (e.g. jenkins) and in that case the artifacts need 
to have compiled metadata.json so that all the actors involved in the 
system agree on the version number in the metadata.  Since CI/CD is 
important, we don't want to break that.

Eliminating the need for 'finalized artifacts' and API servers may not 
be a use case that you care about, but we can't just break the users 
that do care about it because you don't.  My initial reaction to the 
problem is just to do a (tableflip) on configuration-as-code and think 
to myself that we should deprecate metadata.rb and mandate using 
metadata.json (back-to-the-future time).  There might be a better way to 
address the problem, though, which looks at the CD/CI use case and fixes 
their version numbering use case a better way. I'm somewhat optimistic 
that in the future that the berks API server will have completely gone 
away.

I'd also be interested in solutions like baking supermarket into every 
chef server deploy.  Then everyone would have a local repo to push 
cookbooks to and a /universe endpoint.  Then that just becomes your 
local berks API server.

So, if I've got any point here its that I'm very grouchy about the berks 
API server because I see it as something which is a barrier to entry of 
berkshelf adoption and its a problem that need to get solved.   But 
being grouchy about it without understanding why its there (Jamie is 
pretty smart, he didn't just write it because he felt like writing some 
server code) is not being productive.  We need to understand the reasons 
why things are there, and the use cases that drive them and come up with 
solutions to the problems, rather than just complain.  Just complaining 
about the berks API is bad signalling since it just causes flak to the 
people working on berkshelf.   What you want to signal is the chef 
server team that they should integrate the /universe endpoint, or that 
supermarket should be bundled with chef server, or other solutions like 
that. What the chef server team hears right now is that Berkshelf sucks, 
which sounds entirely like a Berkshelf problem.  That's not useful in 
helping them to determine what they could do better with the server code 
or packaging to help solve that problem -- and I'm fairly convinced that 
this is an ecosystem problem, and that Berkshelf is what it is because 
its been developed in isolation from work going on in the client and 
server, and that needs to change in order to make it better.