[chef] Re: Re: Re: automated testing of cookbooks

[Andrew Crump < >]

Integration tests that exercise the service you are building
definitely give you the most bang for buck.

We found the feedback cycle slow as well so I wrote chefspec which
builds on RSpec to support unit testing cookbooks:
https://github.com/acrmp/chefspec

This basically fakes a convergence and allows you to make assertions
about the created resources. At first glance Chef's declarative nature
makes this less useful, but once you start introducing conditional
execution I've found this to be a time saver.

If you're looking to do CI (which you should be) converging twice goes
some way to verifying that your recipes are idempotent.

knife cookbook test is a useful first gate for CI:
http://wiki.opscode.com/display/chef/Knife#Knife-test

Cheers,

Andrew.

On Wed, Nov 23, 2011 at 4:21 AM, AJ Christensen 
<
 >
 wrote:
> As was mentioned, I have been using my MiniTest cookbook [0] (w/
> Report handler) to provide a minimal DSL around the Test Unit style
> tests that MiniTest supports. It would be pretty easy to support
> MiniTests' Spec style, as well as potentially abstracting it to run
> RSpec tests..
>
> Some examples are here [1]
>
> --AJ
>
> [0] https://github.com/fujin/minitest-cookbook
> [1] 
> https://github.com/fujin/minitest-cookbook/blob/develop/cookbooks/minitest/recipes/examples.rb#L57
>
> On 23 November 2011 17:07, Ranjib Dey 
> <
 >
>  wrote:
>> I have used both Rspec + ssh as well as fujin's minitest report handler, 
>> now
>> pretty happy with the report handler based approach, i guess rspec instead
>> of minitest will be a good stuff try out.
>>
>> On Wed, Nov 23, 2011 at 9:30 AM, Marcel M. Cary 
>> <
 >
>> wrote:
>>>
>>> I noticed that Bryan Berry suggested "cookbook testing" as a topic for the
>>> community summit.
>>>
>>>    "Cookbook testing and how to automate it, esp. for opscode/cookbooks"
>>>
>>>
>>>  http://wiki.opscode.com/pages/diffpagesbyversion.action?pageId=20414482&selectedPageVersions=3&selectedPageVersions=2
>>>
>>> I'm also interested in that topic.  As someone responsible for a
>>> relatively small number of nodes, testability (and the prerequisite
>>> repeatability) have been my biggest benefits from Chef.
>>>
>>> So far, what we've done for testing is to use rspec for implementing
>>> tests.  Here's an example:
>>>
>>>    it "should respond on port 80" do
>>>      lambda {
>>>        TCPSocket.open(@server, 'http')
>>>      }.should_not raise_error
>>>    end
>>>
>>> Before running the tests, I have to manually bootstrap a node using knife.
>>> If my instance is the only one in its environment, the spec can find it
>>> using knife's search feature.  The bootstrap takes a few minutes, and the 
>>> 20
>>> or so tests take about half a minute to run.
>>>
>>> While I'm iteratively developing a recipe, my work cycle is to edit
>>> source, upload a cookbook, and rerun chef-client (usually by rerunning 
>>> knife
>>> boostrap, because the execution environment is different from invoking
>>> chef-client directly).  This feels a bit slower than the cycle I'm used to
>>> when coding in Ruby because of the upload and bootstrap steps.
>>>
>>> I like rspec over other testing tools because of how it generates handy
>>> reports, such as this one, which displays an English list of covered test
>>> cases:
>>>
>>>    $ rspec spec/ -f doc
>>>
>>>    Foo role
>>>      should respond on port 80
>>>      should run memcached
>>>      should accept memcached connections
>>>      should have mysql account
>>>      should allow passwordless sudo to user foo as user bar
>>>      should allow passwordless sudo to root as a member of sysadmin
>>>      should allow key login as user bar
>>>      should mount homedirs on ext4, not NFS
>>>      should rotate production.log
>>>      should have baz as default vhost
>>>      ...
>>>
>>> That sample report also gives a feel for sort of things we check.  So far,
>>> nearly all of our checks are non-intrusive enough to run on a production
>>> system.  (The exception is testing of local email delivery 
>>> configurations.)
>>>
>>> Areas I'd love to see improvement:
>>>
>>>    * Shortening the edit-upload-bootstrap-test cycle
>>>    * Automating the bootstrap in the context of testing
>>>    * Adding rspec primitives for Chef-related testing, which might
>>>      include support for multiple platforms
>>>
>>> As an example of rspec primitives, instead of:
>>>
>>>    it "should respond on port 80" do
>>>      lambda {
>>>        TCPSocket.open(@server, 'http')
>>>      }.should_not raise_error
>>>    end
>>>
>>> I'd like to write:
>>>
>>>    it { should respond_on_port(80) }
>>>
>>> Rspec supports the the syntactic sugar; it's just a matter of adding some
>>> "matcher" plugins.
>>>
>>> How do other chef users verify that recipes work as expected?
>>>
>>> I'm not sure how applicable my approach is to opscode/cookbooks because it
>>> relies on having a specific server configuration and can only test a
>>> cookbook in the context of that single server.  If we automated the 
>>> boostrap
>>> step so it could be embedded into the rspec setup blocks, it would be
>>> possible to test a cookbook in several sample contexts, but the time
>>> required to setup each server instance might be prohibitive.
>>>
>>> Marcel
>>
>>
>