[chef] Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Continuous Integration

[Jesse Nelson < >]

Single cbook testing doesn't interest me as much as the complex interactions 
of various cookbooks. If you have many people writing cookbooks realistically 
testing every permutation and output doesn't seem logistically feasible. 
Pulling a subset of the runlist from prod chef and running it over that would 
be good. perhaps cookbook meta data could help with knowing what a cookbook 
should be able to run with or can't run with. 

I agree with andrew that we are already declaring a lot of the state, but 
simply asserting against that state seems a little redundant. Especially when 
we are dynamically configuring something.  I suppose the assumption is that 
if chef says a package installed then it is installed.  At some point we have 
to trust that the tool is doing what it says, is erroring if its not/can't, 
and is doing so in a graceful non destructive manner  (thats the job of the 
framework).  

Some simple code that could take chef recipes and magic up some cucumber or 
similar testing framework would be infinitely useful for most people i 
suspect.  If it could generate basic test for cookbooks that plug into 
cucumber-nagios or something similar. you have some sweet sauce there. 

Generic service for testing functional output of a run was also something we 
were talking about.  being able to have confidence in the output being 
correct for an application at the very minimum  being  syntactically correct 
is at least a step.   I.e. i want to take output form the apache cookbook 
which has dynamic inputs and know that on the other end what the human has 
put in is going to be parsed by httpd without syntax errors.  I believe there 
is value there. 

my apologies if this comes off a little scatter brained im just into my first 
cup of coffee this morning. 




On Jul 9, 2010, at 12:20 PM, Jeremy Deininger wrote:

> Hey guys,
> Thought I'd chime in with my experience testing system configuration code @ 
> RightScale so far.  What we've been building are integration style cucumber 
> tests to run a cookbook through it's paces on all platforms and OSs that we 
> support.  
> 
> First we use our API to spin up 'fresh' server clusters in EC2, one for 
> every platform/OS (variation) that the cookbook will be supporting.  The 
> same could be done using other cloud APIs (anyone else doing this with 
> VMware or etc?)  Starting from scratch is important because of chef's 
> idempotent nature.
> 
> Then a cucumber test is run against every variation in parallel.  The 
> cucumber test runs a series of recipes on the cluster then uses what we 
> call 'spot checks' to ensure the cluster is configured and functional.  The 
> spot checks are updated when we find a bug, to cover the bug.  An example 
> spot check would be, sshing to every server and checking the mysql.err file 
> for bad strings.
> 
> These high level integration tests are long running but have been very 
> useful flushing out bugs.
> -J