[chef] Re: Integration testing Chef -- seriously, what is the best approach?

[Peter Donald < >]

Hi,

We use a heavily customised multi-vm Vagrantfile that is driven by
simple testing infrastructure.

Essentially our tests start by spinning up a chef-server, uploading
the full set of cookbooks/roles/data bags and then instantiating a set
of nodes in a particular order. We then do a few simple tests, just
getting stuff to converge successfully is often a good first step.

We went with the Vagrantfile approach as many of the current test
infrastructure did not exist when we started. Also we use windows that
is not well support in other infra.


On Sat, Jun 1, 2013 at 1:11 AM, Jay Pipes 
<
 >
 wrote:
> Ohai Chefs!
>
> Alrighty, so over in OpenStack + Chef world, we've recently made a lot
> of progress in consolidating and cleaning up the myriad cookbook
> repositories, adding ChefSpec unit tests for the cookbooks, and getting
> all the cookbooks linted with FoodCritic and gated using Gerrit and
> Jenkins [1]. All good stuff.
>
> The next big piece of the puzzle is integration testing. Now, I'm
> neither a Chef nor a Ruby expert, so I decided to use the Google machine
> to determine what the best practices were for doing multi-machine
> integration testing of a large set of interrelated cookbooks.
>
> So far, I've been struggling. I found chef-workflow, and that seemed to
> be a great solution. It's advantages are:
>
>  * pretty well doc'd
>  * able to use multiple provisioners
>  * uses minitest to define test cases
>  * just makes sense for integration testing -- define a workflow (set of
> tasks) that simulate your deployment, spin up a set of boxes or machines
> based on Chef roles, and validate stuff
>
> But it has some downsides that I'm unable to get around:
>
>  * does not support Chef 11 -- and the OpenStack + Chef community has
> decided to standardize on Chef 11 and not spend time supporting 10.x
>  * many of the tools we use -- ChefSpec, Strainer, and others -- depend
> on much newer versions of libs (minitest, json, etc) than chef-workflow
> depends on, which makes it impossible to use in our environment [2]
>  * lots of "magic" that seems to get glossed over -- for instance, I
> cannot understand if Chef Solo is a requirement and if not, why it gets
> installed on all the boxes I provision -- this is probably just my fault
> in not understanding Chef/Ruby 100%, but when magic seems to happen, my
> coder ears tend to perk up -- and not in a good way ;)
>
> I've also read Josh Timberman's article on Test Kitchen [3] but I
> believe we must have different ideas of what integration testing is. My
> definition of integration testing is tests that a) don't mock anything
> out and b) stress the points of interaction (integration) between
> various dependent things (in this case, interrelated cookbooks). I don't
> really see how Test Kitchen makes this easy, but of course, I may be
> misunderstanding something. Josh, please feel free to enlighten me! :)
>
> Anyway, the point of this email is to get feedback on what exactly the
> state of integration testing of Chef cookbooks is today, where it's
> heading, and what people feel should be the direction the OpenStack +
> Chef community should go in.
>
> Best,
> -jay
>
>
> [1] For more info, see here:
> http://www.joinfu.com/2013/05/working-with-the-openstack-code-review-and-ci-system-chef-edition/
> [2] Issue logged in chef-workflow for this... if you have suggestions or
> workarounds, please do comment:
> https://github.com/chef-workflow/chef-workflow/issues/11
> [3]
> http://jtimberman.housepub.org/blog/2012/12/13/cookbook-integration-testing-with-real-examples/



-- 
Cheers,

Peter Donald