[chef] Integration testing Chef -- seriously, what is the best approach?

[Jay Pipes < >]

Ohai Chefs!

Alrighty, so over in OpenStack + Chef world, we've recently made a lot
of progress in consolidating and cleaning up the myriad cookbook
repositories, adding ChefSpec unit tests for the cookbooks, and getting
all the cookbooks linted with FoodCritic and gated using Gerrit and
Jenkins [1]. All good stuff.

The next big piece of the puzzle is integration testing. Now, I'm
neither a Chef nor a Ruby expert, so I decided to use the Google machine
to determine what the best practices were for doing multi-machine
integration testing of a large set of interrelated cookbooks.

So far, I've been struggling. I found chef-workflow, and that seemed to
be a great solution. It's advantages are:

 * pretty well doc'd
 * able to use multiple provisioners
 * uses minitest to define test cases
 * just makes sense for integration testing -- define a workflow (set of
tasks) that simulate your deployment, spin up a set of boxes or machines
based on Chef roles, and validate stuff

But it has some downsides that I'm unable to get around:

 * does not support Chef 11 -- and the OpenStack + Chef community has
decided to standardize on Chef 11 and not spend time supporting 10.x
 * many of the tools we use -- ChefSpec, Strainer, and others -- depend
on much newer versions of libs (minitest, json, etc) than chef-workflow
depends on, which makes it impossible to use in our environment [2]
 * lots of "magic" that seems to get glossed over -- for instance, I
cannot understand if Chef Solo is a requirement and if not, why it gets
installed on all the boxes I provision -- this is probably just my fault
in not understanding Chef/Ruby 100%, but when magic seems to happen, my
coder ears tend to perk up -- and not in a good way ;)

I've also read Josh Timberman's article on Test Kitchen [3] but I
believe we must have different ideas of what integration testing is. My
definition of integration testing is tests that a) don't mock anything
out and b) stress the points of interaction (integration) between
various dependent things (in this case, interrelated cookbooks). I don't
really see how Test Kitchen makes this easy, but of course, I may be
misunderstanding something. Josh, please feel free to enlighten me! :)

Anyway, the point of this email is to get feedback on what exactly the
state of integration testing of Chef cookbooks is today, where it's
heading, and what people feel should be the direction the OpenStack +
Chef community should go in.

Best,
-jay


[1] For more info, see here:
http://www.joinfu.com/2013/05/working-with-the-openstack-code-review-and-ci-system-chef-edition/
[2] Issue logged in chef-workflow for this... if you have suggestions or
workarounds, please do comment:
https://github.com/chef-workflow/chef-workflow/issues/11
[3]
http://jtimberman.housepub.org/blog/2012/12/13/cookbook-integration-testing-with-real-examples/