[chef] Re: Re: Testing and Determining Test Coverage in Cookbooks

["steve ." < >]

Hi Torbin!

Durf.  Never occurred to me to crib off of cucumber-nagios.  That's a
pretty neat idea, and it sure solves the problem of "How do you
explain test-writing?" as Cucumber tests are about as easy to read as
they come.  I'm going to give that a try next week.

I like the chefspec/fauxhai concept but I'm lucky enough to be testing
in an environment where spinning up and immediately terminating a
bunch of images is relatively painless.  So I will probably just laze
out on that step and write unit/basic integration tests with the
minitest handler, then write up some Cucumber features to test each VM
after convergence and before teardown.  One place where I do see value
in that would be for our LWRPs, a few of which are getting pretty
complicated and could probably do with some specs running that could
catch regressions.

The only problem I have (and this is due to my particular
implementation) is that I'm testing role/environment/platform
combinations.  Can't really test a multi-tiered app this way.  It's
one of the things test-kitchen does that my little skunkworks project
doesn't.  I guess the answer is to bind some features to a special
full-stack role that we would only use for testing/development, though
that wouldn't work for apps that require multiple hosts (Mongo springs
to mind).

Your documentation and the test-writing docs on the Chefspec and
Fauxhai sites are pretty comprehensive, it seems (or have I just been
reading too much poorly-documented code lately? ;) ).  I'm not sure
how many people are interested in converging Vagrant VMs against a
Chef server endpoint (i.e. using the chef-client provisioner instead
of chef-solo), but jtimberman wrote a couple of pretty good blog posts
on the subject (starting with
http://jtimberman.housepub.org/blog/2012/03/18/multivm-vagrantfile-for-chef/
, again, HI GOOGLE! ) and I could perhaps write something up as well
if the community desires it and existing resources are found wanting.

Anyway, I think this kind of testing is a neat idea and I'm going to
see how far I can get with it.  I try to follow this list but I'm
awash in work email most of the time.  Comments/off-list conversations
on this subject with interested parties are also welcome.

On Thu, Oct 4, 2012 at 10:22 PM, Torben Knerr 
<
 >
 wrote:
> Love the idea of properly tested cookbooks and the fact that you are pushing
> your buddies into that direction :-)
>
> I've taken some notes and collected some links about test-driven cookbook
> development:
> https://github.com/tknerr/bills-kitchen/blob/master/COOKBOOK_DEVELOPMENT.md
>
> It covers cookbook testing on several levels:
> * syntax & long checking -> foodcritic
> * unit-level spec testing -> chefspec and fauxhai
> * "smoke tests" from the inside -> minitest-handler
> * acceptance-level testing from the outside -> cucumber-nagios
>
> Maybe you can make some use of it or you have some ideas how to improve it.
>
> Cheers, Torben
>
> Am 05.10.2012 03:17 schrieb "steve ." 
> <
 >:
>
>> Continuous cookbook testing framework (analogous to test-kitchen, but
>> built on Jenkins, Foodcritic, RVM and Vagrant) ... check.
>> Chef-minitest handler running on VMs and reporting test results back
>> to CI host ... check.
>>
>> Now I just have to get all my cookbook authors to write tests!
>>
>> It seems like a lot of us are in this situation, so I thought I'd
>> share my findings and the direction in which I seem to be headed.  We
>> have to get a bunch of people who are maybe not so used to TDD/BDD or
>> writing integration tests to suddenly start writing integration tests
>> in minitest (spec tests, in my exact case).
>>
>> There's a growing amount of source code out there, but the best
>> documentation I've found is still in Bryan's example cookbook (found
>> here, if this somehow pops up on Google):
>>
>> https://github.com/btm/minitest-handler-cookbook
>>
>> So you can use node data, but only within the scope of a particular
>> test definition.  I understand why that is, but folks who know "enough
>> Ruby for Chef" will probably get tripped up by that.
>>
>> Still, I need a way to tell my budding cooks what needs testing and
>> what doesn't.  Can I use existing code coverage tools to do this
>> *outside of converging a node and running the minitest handler?*  As
>> far as I can tell the answer is, no, I can't.
>>
>> (I base this preliminary conclusion on an indexed conversation in
>> #chef from July:
>>
>> http://community.opscode.com/chat/chef/2012-07-18#id-165241
>>
>> ... would love to be proven wrong, BTW)
>>
>> My current idea for drawing attention to which cookbooks are
>> adequately tested in our Chef repository involves writing some
>> Foodcritic rules to check for the presence of minitests in each
>> cookbook and then critique them.  I believe I can get started with
>> this using three rules:
>> "Does your cookbook have a files/default/tests/minitest directory?"
>> "Do you have a test/spec.rb file describing each recipe in the cookbook?"
>> "Does your test/spec for recipe X cover all of the resources specified
>> in recipe X?".
>>
>> Since Foodcritic testing is already in the early stage of my cookbook
>> testing pipeline, I can easily turn those warnings into pretty graphs.
>>  I'm worried that writing those rules will be a bit brain-melty,
>> though.
>>
>> Mission accomplished: I am shaming all my repo contributors for not
>> writing tests for their cookbook contributions.  I still need to make
>> writing those tests easier for them to pick up or I'm just being mean.
>>  My idea here is to write and exhaustively document a helper gem that
>> provides convenience functions beyond the resource checks that are
>> already included, then distribute that along with the recipe that
>> installs the chef-minitest-handler gem and sets it up as an event
>> handler on the system.  If the helper gem is well-documented and the
>> tests in the repository leverage the helper properly, I think that
>> will be enough signposts to get us started.  Then It'll be a lot
>> easier to accept all these pull requests...
>>
>> Is anyone else proceeding along these lines?  Is there any interest in
>> adding that functionality to mainline Foodcritic?  Is there some other
>> way of ensuring test coverage on cookbooks?  Any fellow travelers know
>> of any pitfalls or open source projects that might help?
>>
>> (besides test-kitchen, that is.  ;) )