[chef] Re: Chef Server vs. Not Guidance

[Daniel DeLeo < >]

On Tuesday, August 4, 2015 at 12:24 PM, Kelly Brownsberger wrote:

> My team has been developing recipes with Chef for several months now and 
> we've made significant progress in automating our infrastructure, but we're 
> now getting to a point where we need to scale this up and make it a first 
> class part of our CI pipeline where many teams are maintaining their own 
> cookbooks sitting side by side the software code that's running on this 
> infrastructure. Up until now we've been focused entirely on recipe 
> development using chef-client and Vagrant for spinning up nodes and running 
> the cookbooks via Vagrant's built in provisioners. As you'd expect, things 
> are starting to become unwieldy with managing attributes and common shared 
> cookbooks, etc. We've started looking at Berkshelf for managing 
> dependencies and environments and roles for putting some structure to our 
> attributes.
>  
> Here's where it gets interesting... to put it bluntly, there are several 
> within my team that aren't bought into the idea of Chef Server. Our entire 
> enterprise orbits around our version control system. And we already have a 
> large investment in Artifactory which is a home for all of the technical 
> artifacts for all of our products across several platforms and it works 
> quite well as a general purpose distribution hub. As we've started playing 
> with the chef concepts of environments and roles while running chef-client 
> in local mode, I think I see the value in these abstractions, however.... 
> I'm starting to get the sense that using these outside the context of a 
> Chef Server is going to be a very bumpy road.

If you really want to avoid using a Chef Server, you can just run local mode 
in production, using a local mode formatted repo as your distribution 
artifact.
  
>  
> Right now I'm able to get our cookbook using environments and roles and 
> have chef-client find them by pointed to a directory on disk where these 
> json files list under version control. This works, but the next use case 
> we're looking to support is running these outside the context of Vagrant 
> and the current thought is using --recipe-url to point to a tarball that we 
> published to artifactory from our CI builds in Jenkins (by way of Berks 
> package). This use case would be for running these cookbooks are production 
> machines that do not have connectivity to our version control system and 
> will not be running in the context of Vagrant. As best I can tell, this 
> will not work as I do not believe there is any way to include environments 
> and roles json files in this tarbar and have chef-client recognize it via 
> --recipe-url. Can someone confirm my suspicion?

Not 100% sure, but I think you’d basically need to make your own script to 
replace berks package. But this shouldn’t be hard.
  
>  
> Questions
> 1) are roles and environments even recommended at all? I'm finding folks 
> out there that really feel strongly they are an anti-pattern [1][2], though 
> I'm not sure I agree with them.

The big issue most folks have with roles is that the Chef Server doesn’t have 
any way to version them, so a change to a role would go live on all systems 
that use it at once. If you solve for this in the way you do your 
distribution, they’re fine.

Environments mostly just exist to provide version pinning for cookbooks, but 
they do have attributes you could use if you want. But you could also just 
use roles for that.


> 2) I keep hearing about how there are loads of people out there using chef 
> without chef-server and doing so happily. are there folks using 
> environments and roles?
> 3) similar to environments and roles, how are nodes managed in a 
> chef-server-less world? I see that you can define these in json and even 
> knife these into a chef server, but when I do a chef-client run in local 
> mode, the nodes/ directory is written to with data from that run. In other 
> words files in nodes/ doesn't seem to be something that's considered a 
> "source" of anything, but rather a runtime output dir.

The node data there will be loaded the next time Chef runs, which gives you 
persistence for ‘normal’ level attributes. It’s also possible to create other 
nodes there so that search works (if you use a 3rd party cookbook that 
hardcodes search, for example).
  
>  
> 4) I'm torn with chef-server because it feels like something like this 
> shouldn't be necessary, but the deeper I get into these concepts more it 
> feels like we're going against the grain not fully embracing chef-server. 
> Can anyone point me to some detailed information on what the full 
> development production lifecycle would look liek with a chef-server 
> oriented world? What about the same in a chef-server-less world?

Since you already have Ci in the mix, you’d have Ci push your cookbooks to 
the server. The tricky bit is making sure each set of nodes gets the correct 
cookbook versions and you do not overwrite any existing cookbooks. There’s 
several ways to do this; you can either put the responsibility on developers 
to update cookbook versions when submitting changes, or you can have Ci 
automatically bump the version number. In either case, you can then generate 
an environment JSON file and then upload that to pin the cookbooks.

Alternatively, you could look at policyfiles, which solve the issues with 
roles being unversioned and also with cookbook version conflicts. You can 
read more about them here: 
https://github.com/chef/chef-dk/blob/master/POLICYFILE_README.md
  
>  
> Sorry for the long rambling email, but was hoping some kind soul could give 
> me some guidance on how to proceed - every direction we look right now feel 
> less than ideal.
>  
> [1] http://dougireton.com/blog/2013/02/16/chef-cookbook-anti-patterns/
> [2]http://devopsanywhere.blogspot.com/2012/11/how-to-write-reusable-chef-cookbooks.html


HTH, I would like to have gone into more depth but it’s getting to be late 
here. Feel free to follow up with more questions.

--  
Daniel DeLeo