[chef] Re: automated testing of cookbooks

[Ranjib Dey < >]

I have used both Rspec + ssh as well as fujin's minitest report handler, now pretty happy with the report handler based approach, i guess rspec instead of minitest will be a good stuff try out.

On Wed, Nov 23, 2011 at 9:30 AM, Marcel M. Cary < "> > wrote:

I noticed that Bryan Berry suggested "cookbook testing" as a topic for the community summit.

   "Cookbook testing and how to automate it, esp. for opscode/cookbooks"

   http://wiki.opscode.com/pages/diffpagesbyversion.action?pageId=20414482&selectedPageVersions=3&selectedPageVersions=2

I'm also interested in that topic.  As someone responsible for a relatively small number of nodes, testability (and the prerequisite repeatability) have been my biggest benefits from Chef.

So far, what we've done for testing is to use rspec for implementing tests.  Here's an example:

   it "should respond on port 80" do
     lambda {
       TCPSocket.open(@server, 'http')
     }.should_not raise_error
   end

Before running the tests, I have to manually bootstrap a node using knife. If my instance is the only one in its environment, the spec can find it using knife's search feature.  The bootstrap takes a few minutes, and the 20 or so tests take about half a minute to run.

While I'm iteratively developing a recipe, my work cycle is to edit source, upload a cookbook, and rerun chef-client (usually by rerunning knife boostrap, because the execution environment is different from invoking chef-client directly).  This feels a bit slower than the cycle I'm used to when coding in Ruby because of the upload and bootstrap steps.

I like rspec over other testing tools because of how it generates handy reports, such as this one, which displays an English list of covered test cases:

   $ rspec spec/ -f doc

   Foo role
     should respond on port 80
     should run memcached
     should accept memcached connections
     should have mysql account
     should allow passwordless sudo to user foo as user bar
     should allow passwordless sudo to root as a member of sysadmin
     should allow key login as user bar
     should mount homedirs on ext4, not NFS
     should rotate production.log
     should have baz as default vhost
     ...

That sample report also gives a feel for sort of things we check.  So far, nearly all of our checks are non-intrusive enough to run on a production system.  (The exception is testing of local email delivery configurations.)

Areas I'd love to see improvement:

   * Shortening the edit-upload-bootstrap-test cycle
   * Automating the bootstrap in the context of testing
   * Adding rspec primitives for Chef-related testing, which might
     include support for multiple platforms

As an example of rspec primitives, instead of:

   it "should respond on port 80" do
     lambda {
       TCPSocket.open(@server, 'http')
     }.should_not raise_error
   end

I'd like to write:

   it { should respond_on_port(80) }

Rspec supports the the syntactic sugar; it's just a matter of adding some "matcher" plugins.

How do other chef users verify that recipes work as expected?

I'm not sure how applicable my approach is to opscode/cookbooks because it relies on having a specific server configuration and can only test a cookbook in the context of that single server.  If we automated the boostrap step so it could be embedded into the rspec setup blocks, it would be possible to test a cookbook in several sample contexts, but the time required to setup each server instance might be prohibitive.

Marcel