[chef] Re: Infrastructure Testing

[Bryan McLellan < >]

On Thu, Mar 1, 2012 at 12:13 PM, Jamie van Dyke 
<
 >
 wrote:
> Again, I must stress that I feel like this si such a simple thing that needs
> to be in place that I must have missed some vital information I need to get
> cracking.  Apologies if it comes across as whiney, I'm just frustrated and
> need to know if I embark on such a project, would I be re-inventing a wheel
> I missed somewhere.

The problem here is that not everyone is solving the same problem and
we have different preferences about tools and language. TMTOWTDI.

Robert Berger started another mailing list for this kind of discussion
recently, but I'm not compelled to have the conversation on a
different list myself:
https://groups.google.com/forum/?fromgroups#!forum/chef-testing

Also, it is one of those things that most everyone thinks they should
do, but we don't have to do it, so we spend more time thinking about
it and talking about it than implementing it.

> Minitest-Handler
> ----------------
>
> There are 2 variants on this idea, 1 is strictly for chef-solo and the other
> is for chef-client.  I don't run chef-solo so I opted to try out the
> cookbook_file based method that was recently discussed on the mailing list
> (http://lists.opscode.com/sympa/arc/chef/2012-03/msg00000.html).  I'm
> struggling to understand why you'd follow this method.  Here's why:  A
> cookbook is generic and can be passed around between users/environments, for
> example the 'generic-users' cookbook.  To add a
> files/default/tests/minitest/moo_test.rb to this cookbook and upload it,
> means I've just made that test specific to my environment but included it
> within a generic cookbook.  It's completely possible that I've not grasped
> the methodology on this, but it seems fundamentally wrong because the 2
> should be separate.

Calavera's minitest-handler is the really same for chef-solo or
chef-server. In my example we use the chef_handler because lets us
install it to a node by simply using the run list rather than having
to edit the node configuration.

Opscode does a lot of heavy lifting trying to test a cookbook after
merging a community contribution to ensure the cookbook still does
what it is supposed to do. This gets more complex as the cookbook
becomes more complex, as well as when it supports multiple
distributions and operating systems. If the apache2 cookbook contained
integration tests for many use cases on many distributions we could
easily spin up cloud instances or virtual machines to verify that
these tests pass. It has proven harder to choose a framework for
integration testing than unit testing however, where we have already
landed solidly on rspec.

While the apache2 cookbook makes a more straight forward example, the
generic-users cookbook can still have quality tests written for it
that are not dependent on the user because the cookbook is data
driven. In fact, the minitest-handler solution is a particularly
strong choice here because we still have access to data in the chef
run, and can create a data driven test. For all of the users in the
data bag that should have been created, check on the system that they
actually made it there.

Bryan