[chef] Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Continuous Integration

[Jeremy Deininger < >]

Hey guys,
Thought I'd chime in with my experience testing system configuration code @ RightScale so far.  What we've been building are integration style cucumber tests to run a cookbook through it's paces on all platforms and OSs that we support.  

First we use our API to spin up 'fresh' server clusters in EC2, one for every platform/OS (variation) that the cookbook will be supporting.  The same could be done using other cloud APIs (anyone else doing this with VMware or etc?)  Starting from scratch is important because of chef's idempotent nature.

Then a cucumber test is run against every variation in parallel.  The cucumber test runs a series of recipes on the cluster then uses what we call 'spot checks' to ensure the cluster is configured and functional.  The spot checks are updated when we find a bug, to cover the bug.  An example spot check would be, sshing to every server and checking the mysql.err file for bad strings.

These high level integration tests are long running but have been very useful flushing out bugs.
-J