[chef] Re: Infrastructure Testing

["Wagner, Jason" < >]

Several folks got in a conversation off of Twitter and ended up starting http://groups.google.com/group/chef-testing to discuss this topic.

Here's a combination of the tools that we've found and my rough thoughts on how they might fit together: http://groups.google.com/group/chef-testing/browse_thread/thread/523fde7ce4f9eaf2?hl=en

The conversation started with how we'd like to incorporate these tools together with Ironfan to have some truly end-to-end workflow for testing all the way from recipes to a full cluster. The Infochimps guys have been doing a lot of stuff on Ironfan's new release, so my position has been "wait and see" what they come up with.

The tool that's been working best for me is a combination of 
cucumber-nagios and toft (https://github.com/exceedhl/toft). It uses LXC containers to quickly fire up instances, apply the current repository to it as a chef-solo recipe, and use cucumber to check it. It does have a Vagrant setup
 that I haven't tried.

Toft still 
needs a bit of work, though, to be a graceful solution. Since it uses chef-solo, anything with a search will fail. I'm sure that can be mocked and incorporated as part of the harness. The whole thing could be easily extended so that the cucumber features run against EC2 nodes, etc. I'm thinking of extending Ironfan to have a Toft cloud service, pending the cooldown of that project.

Any thoughts on the subject would be greatly appreciated in the google group.

On Thu, Mar 1, 2012 at 12:13 PM, Jamie van Dyke < "> > wrote:

Hi all,

I've been beating myself up over the last few days trying to figure out the best way of testing our build automation.  I've played with cucumber-chef, minitest_handler, rspec-chef and more.  All of them were either broken in implementation or theory.  Here are my thoughts:

Cucumber-chef
-------------

I fought for 2 days with the gem trying to get it to work, and after multiple gem incompatibilities I thought I had it sorted.  Until I got to the part where it builds an EC2 instance, which then failed miserably as well.  In the end I dumped it because if it's this hard to get running initially, how can I trust it to continue to work in the future without another 2+ days worth of headaches.  In theory this was probably what I want, but over-complicated and brittle.

Rspec-chef
----------

I've possibly misunderstood the concept behind this one, but it looks like it's testing that I've written the correct pieces in my recipes rather than my infrastructure has turned out correctly after a run.

Minitest-Handler
----------------

There are 2 variants on this idea, 1 is strictly for chef-solo and the other is for chef-client.  I don't run chef-solo so I opted to try out the cookbook_file based method that was recently discussed on the mailing list (http://lists.opscode.com/sympa/arc/chef/2012-03/msg00000.html).  I'm struggling to understand why you'd follow this method.  Here's why:  A cookbook is generic and can be passed around between users/environments, for example the 'generic-users' cookbook.  To add a files/default/tests/minitest/moo_test.rb to this cookbook and upload it, means I've just made that test specific to my environment but included it within a generic cookbook.  It's completely possible that I've not grasped the methodology on this, but it seems fundamentally wrong because the 2 should be separate.

What I'm looking for is something as simple as this workflow, and if it comes to light that nobody else has done this and I need to write my won then I'm happy to do so:

1 - I write my infrastructure test, describing what change I am about to implement
2 - I run my test suite
 a - a vagrant/ec2/heroku/whatever instance is started
 b - chef-client is run on that server as I would expect in production
 c - my test suite looks to see if changes I want are in place
 d - it fails
3 - I write my cookbook changes to implement the small change I want
4 - is our test instance running?
 a - no, GOTO 2a
 b - yes, GOTO 2b
5 - did it pass?
 a - yes, GOTO 1
 b - no, GOTO 3


I would expect my test suite to simply describe what should and should not be in place, running, or open.  Have I completely missed the boat here and this already exists in a nice simple structure, or are we at the stage in Chef specific DevOps where we need to write something that fulfils this?

To be clear, I would expect such a tool to have a generic 'setup' step that isn't automatic and needs a little work.  Something like 'start a vagrant instance with the role X' or similar.  I would also expect to have to write my own 'teardown'.

Again, I must stress that I feel like this si such a simple thing that needs to be in place that I must have missed some vital information I need to get cracking.  Apologies if it comes across as whiney, I'm just frustrated and need to know if I embark on such a project, would I be re-inventing a wheel I missed somewhere.

Cheers
Jamie van Dyke

-- 
Jason Wagner

" target="_blank">