[chef] Re: Testing and Determining Test Coverage in Cookbooks

[Torben Knerr < >]

Love the idea of properly tested cookbooks and the fact that you are pushing your buddies into that direction :-)

I've taken some notes and collected some links about test-driven cookbook development:
https://github.com/tknerr/bills-kitchen/blob/master/COOKBOOK_DEVELOPMENT.md

It covers cookbook testing on several levels:
* syntax & long checking -> foodcritic
* unit-level spec testing -> chefspec and fauxhai
* "smoke tests" from the inside -> minitest-handler
* acceptance-level testing from the outside -> cucumber-nagios

Maybe you can make some use of it or you have some ideas how to improve it.

Cheers, Torben

Am 05.10.2012 03:17 schrieb "steve ." < "> >:

Continuous cookbook testing framework (analogous to test-kitchen, but
built on Jenkins, Foodcritic, RVM and Vagrant) ... check.
Chef-minitest handler running on VMs and reporting test results back
to CI host ... check.

Now I just have to get all my cookbook authors to write tests!

It seems like a lot of us are in this situation, so I thought I'd
share my findings and the direction in which I seem to be headed.  We
have to get a bunch of people who are maybe not so used to TDD/BDD or
writing integration tests to suddenly start writing integration tests
in minitest (spec tests, in my exact case).

There's a growing amount of source code out there, but the best
documentation I've found is still in Bryan's example cookbook (found
here, if this somehow pops up on Google):

https://github.com/btm/minitest-handler-cookbook

So you can use node data, but only within the scope of a particular
test definition.  I understand why that is, but folks who know "enough
Ruby for Chef" will probably get tripped up by that.

Still, I need a way to tell my budding cooks what needs testing and
what doesn't.  Can I use existing code coverage tools to do this
*outside of converging a node and running the minitest handler?*  As
far as I can tell the answer is, no, I can't.

(I base this preliminary conclusion on an indexed conversation in
#chef from July:

http://community.opscode.com/chat/chef/2012-07-18#id-165241

... would love to be proven wrong, BTW)

My current idea for drawing attention to which cookbooks are
adequately tested in our Chef repository involves writing some
Foodcritic rules to check for the presence of minitests in each
cookbook and then critique them.  I believe I can get started with
this using three rules:
"Does your cookbook have a files/default/tests/minitest directory?"
"Do you have a test/spec.rb file describing each recipe in the cookbook?"
"Does your test/spec for recipe X cover all of the resources specified
in recipe X?".

Since Foodcritic testing is already in the early stage of my cookbook
testing pipeline, I can easily turn those warnings into pretty graphs.
 I'm worried that writing those rules will be a bit brain-melty,
though.

Mission accomplished: I am shaming all my repo contributors for not
writing tests for their cookbook contributions.  I still need to make
writing those tests easier for them to pick up or I'm just being mean.
 My idea here is to write and exhaustively document a helper gem that
provides convenience functions beyond the resource checks that are
already included, then distribute that along with the recipe that
installs the chef-minitest-handler gem and sets it up as an event
handler on the system.  If the helper gem is well-documented and the
tests in the repository leverage the helper properly, I think that
will be enough signposts to get us started.  Then It'll be a lot
easier to accept all these pull requests...

Is anyone else proceeding along these lines?  Is there any interest in
adding that functionality to mainline Foodcritic?  Is there some other
way of ensuring test coverage on cookbooks?  Any fellow travelers know
of any pitfalls or open source projects that might help?

(besides test-kitchen, that is.  ;) )