[chef] automated testing of cookbooks

["Marcel M. Cary" < >]

I noticed that Bryan Berry suggested "cookbook testing" as a topic for the 
community summit.

    "Cookbook testing and how to automate it, esp. for opscode/cookbooks"

    
http://wiki.opscode.com/pages/diffpagesbyversion.action?pageId=20414482&selectedPageVersions=3&selectedPageVersions=2

I'm also interested in that topic.  As someone responsible for a 
relatively small number of nodes, testability (and the prerequisite 
repeatability) have been my biggest benefits from Chef.

So far, what we've done for testing is to use rspec for implementing 
tests.  Here's an example:

    it "should respond on port 80" do
      lambda {
        TCPSocket.open(@server, 'http')
      }.should_not raise_error
    end

Before running the tests, I have to manually bootstrap a node using knife. 
If my instance is the only one in its environment, the spec can find it 
using knife's search feature.  The bootstrap takes a few minutes, and the 
20 or so tests take about half a minute to run.

While I'm iteratively developing a recipe, my work cycle is to edit 
source, upload a cookbook, and rerun chef-client (usually by rerunning 
knife boostrap, because the execution environment is different from 
invoking chef-client directly).  This feels a bit slower than the cycle 
I'm used to when coding in Ruby because of the upload and bootstrap steps.

I like rspec over other testing tools because of how it generates handy 
reports, such as this one, which displays an English list of covered test 
cases:

    $ rspec spec/ -f doc

    Foo role
      should respond on port 80
      should run memcached
      should accept memcached connections
      should have mysql account
      should allow passwordless sudo to user foo as user bar
      should allow passwordless sudo to root as a member of sysadmin
      should allow key login as user bar
      should mount homedirs on ext4, not NFS
      should rotate production.log
      should have baz as default vhost
      ...

That sample report also gives a feel for sort of things we check.  So far, 
nearly all of our checks are non-intrusive enough to run on a production 
system.  (The exception is testing of local email delivery 
configurations.)

Areas I'd love to see improvement:

    * Shortening the edit-upload-bootstrap-test cycle
    * Automating the bootstrap in the context of testing
    * Adding rspec primitives for Chef-related testing, which might
      include support for multiple platforms

As an example of rspec primitives, instead of:

    it "should respond on port 80" do
      lambda {
        TCPSocket.open(@server, 'http')
      }.should_not raise_error
    end

I'd like to write:

    it { should respond_on_port(80) }

Rspec supports the the syntactic sugar; it's just a matter of adding some 
"matcher" plugins.

How do other chef users verify that recipes work as expected?

I'm not sure how applicable my approach is to opscode/cookbooks because 
it relies on having a specific server configuration and can only test a 
cookbook in the context of that single server.  If we automated the 
boostrap step so it could be embedded into the rspec setup blocks, it 
would be possible to test a cookbook in several sample contexts, but the 
time required to setup each server instance might be prohibitive.

Marcel