[chef] Re: Integration testing Chef -- seriously, what is the best approach?

[Jesse Nelson < >]

  I've been down this road with our Infrastructure, and I will share my thoughts. What I started with was workflow, but as the chef11 thing came up I couldn't use that.  I've been a fan and continue to be of test-kitchen 1.0  for individual cookbook testing, but the lack of being able to describe infrastructures and run chef-server to test hurts for integration. I believe this function will come to kitchen tho as many people have talked about it, and I am sure Fletcher is on board with the idea just not 100% ready to go a direction yet.  

 What we use now:

   Thor tasks that manage knife-server and spice weasel + knife-vagrant to build out arbitrary infrastructures. I ripped this out of our repo and put it on github during chef hack-day. Example @ github.com/spheromak/chef-infra.  It's less than pretty, but it has been working for us to get our dev work up and running.  Just add server-spec[1] tests and a test task to get some testing in there (or whatever you want/need).  I am happy to talk more about it if there is interest, but I will just give highlights here. 

 

 This approach using a mishmash of tools allows us to swap the infrastructures descriptions out from vagrant to openstack or something else supported by knife. That was an initial requirement, but our testing needs have been driving me down a different path due to vbox being a pos, and the tests taking too long to spin up. 

What I am trying to make work now:

At chef-conf I spent some time talking with Chris Roberts, and he introduced me to the thing he was working on in this realm: vagabond[2].  Which is a lxc management framework that integrates with kitchen and has its own hooks for running tests. It handles building a chef-server or chef-zero container, and making your repo assets available to it. Basically it is a better more concise version of what I had done with thor + spice + knife. Over the last week or so I have been working on getting things working with that instead.  The vagabond system is much faster and easier to add tests for. Using kitchen for cooks and vagabond for integration testing is working out really nicely for me right now. There are still a lot of edges to be worked out but Chris has been hacking on vagabond pretty heavily. 

Caveat emptor: Everything I've been doing in vagabond land has been against development or my own custom hacks. This thing isn't quite ready for general consumption, but It is already useful. I also believe its the right way to move on the integration testing front (at least for my needs). 

[1]( http://serverspec.org/ ) 

[2]( https://github.com/chrisroberts/vagabond )

- 

Jesse 

On May 31, 2013, at 8:11 AM, Jay Pipes < "> > wrote:

Ohai Chefs!

Alrighty, so over in OpenStack + Chef world, we've recently made a lot
of progress in consolidating and cleaning up the myriad cookbook
repositories, adding ChefSpec unit tests for the cookbooks, and getting
all the cookbooks linted with FoodCritic and gated using Gerrit and
Jenkins [1]. All good stuff.

The next big piece of the puzzle is integration testing. Now, I'm
neither a Chef nor a Ruby expert, so I decided to use the Google machine
to determine what the best practices were for doing multi-machine
integration testing of a large set of interrelated cookbooks.

So far, I've been struggling. I found chef-workflow, and that seemed to
be a great solution. It's advantages are:

* pretty well doc'd
* able to use multiple provisioners
* uses minitest to define test cases
* just makes sense for integration testing -- define a workflow (set of
tasks) that simulate your deployment, spin up a set of boxes or machines
based on Chef roles, and validate stuff

But it has some downsides that I'm unable to get around:

* does not support Chef 11 -- and the OpenStack + Chef community has
decided to standardize on Chef 11 and not spend time supporting 10.x
* many of the tools we use -- ChefSpec, Strainer, and others -- depend
on much newer versions of libs (minitest, json, etc) than chef-workflow
depends on, which makes it impossible to use in our environment [2]
* lots of "magic" that seems to get glossed over -- for instance, I
cannot understand if Chef Solo is a requirement and if not, why it gets
installed on all the boxes I provision -- this is probably just my fault
in not understanding Chef/Ruby 100%, but when magic seems to happen, my
coder ears tend to perk up -- and not in a good way ;)

I've also read Josh Timberman's article on Test Kitchen [3] but I
believe we must have different ideas of what integration testing is. My
definition of integration testing is tests that a) don't mock anything
out and b) stress the points of interaction (integration) between
various dependent things (in this case, interrelated cookbooks). I don't
really see how Test Kitchen makes this easy, but of course, I may be
misunderstanding something. Josh, please feel free to enlighten me! :)

Anyway, the point of this email is to get feedback on what exactly the
state of integration testing of Chef cookbooks is today, where it's
heading, and what people feel should be the direction the OpenStack +
Chef community should go in.

Best,
-jay


[1] For more info, see here:
http://www.joinfu.com/2013/05/working-with-the-openstack-code-review-and-ci-system-chef-edition/
[2] Issue logged in chef-workflow for this... if you have suggestions or
workarounds, please do comment:
https://github.com/chef-workflow/chef-workflow/issues/11
[3]
http://jtimberman.housepub.org/blog/2012/12/13/cookbook-integration-testing-with-real-examples/

Attachment: smime.p7s
Description: S/MIME cryptographic signature