[chef] Re: Managing multiple discrete projects

["steve ." < >]

Splitting out your repo to one cookbook per repository (and using a workflow tool Berkshelf or Librarian-Chef) might help scope your changes a little more tightly.

Other than that, we faced basically the same problem you did.  The answer was to set up CI to handle the testing.  In the open source community, you could take a look at using test-kitchen... unfortunately, it wasn't a thing yet so I set up a whole Jenkins install to do Vagrant and OpenStack-based testing of cookbooks from a central GitHub Enterprise repository.  (I set the whole thing up using Chef cookbooks, too!)

As far as I know, test-kitchen is kind of the de facto standard out there, so I've never seriously considered going through the vetting process that would result in my having permission to open-source the cookbook.  After presenting my work at ChefConf this year, I did have a couple people come up to me and say they were doing something similar using Jenkins.  (Unfortunately the presentation isn't available online but I'm happy to answer questions about it...)

A couple of the workflowy problems I've faced with this is that pull requests/commits against a combined repository took ages to perform because the build mechanism is, "Build all the roles contained in the repository in all environments on all platforms and make sure none of the tests/Chef runs fail."  So you really want to have a way to build a dependency graph of all the cookbooks you care about so that you can map file changes to the recipes and roles that are impacted by the change.  That way, if a pull request that affects the Java cookbook comes in, Jenkins knows to test that cookbook but also your Tomcat and Solr cookbooks as well.  The other big problem has been tracking dependencies that are outside of your source tree.  It's easy to trigger a build when a commit gets pushed to one of your repos, but in order to catch changes outside of that you're stuck writing a poll trigger that checks those repos every so often...

Honestly, if I had to do it all over again I think I'd try to find a way to hack multi-provider multi-platform cookbook testing into Travis-CI.  (Although in my case it would have to be Travis-CI Pro Enterprise Behind-Your-Firewall Turbo Hyper Fighting Edition... )

On Tue, Oct 15, 2013 at 7:45 PM, Clif Smith < " target="_blank"> > wrote:

We've been using Chef for a couple of years now and love it.  Thankfully our business is growing to the point where it's becoming a bit onerous to support multiple products with 1 Chef repo in 1 Git repo.  For example, we're currently supporting 2 products.  When we update a cookbook, data bag, etc. that's used by both products, we should always test that those changes don't cause any issues.  So far that's been fine, but this latest product is causing many changes throughout my chef repo and the overhead in testing across 3 products is a bit more than we'd like to take on.  We're pondering standing up a couple more Chef servers (prod/non-prod) and utilizing a copy of the current git repo but I hate the idea of enhancements not easily making it into the other repo(s).  We're also looking at a git repo for common artifacts which would alleviate some of my worries.

Having said all of that…how are others handling similar situations?