Hi Steve,
I'm still in the learning phase for testing with Chef, but I consider it an essential part, thats why I spent some time on it lately.
See comments inline.
Am 07.10.2012 10:38 schrieb "steve ." <
">
>:
>
> Hi Torbin!
>
> Durf. Never occurred to me to crib off of cucumber-nagios. That's a
> pretty neat idea, and it sure solves the problem of "How do you
> explain test-writing?" as Cucumber tests are about as easy to read as
> they come. I'm going to give that a try next week.
>
> I like the chefspec/fauxhai concept but I'm lucky enough to be testing
> in an environment where spinning up and immediately terminating a
> bunch of images is relatively painless.
I like chefspec/fauxhai too. From my experience they are especially (if not only) useful for testing how combinations of different platforms and attributes affect the resources that you create.
I quickly felt the urgent need to spin up some images and test real node convergence as well :-)
Vagrant works well for me here, but the roundtrip times are naturally longer. I wanted to play with LXC containers instead of using VirtualBox (ideally Re-using my Vagrantfile), but haven't come to testing this yet.
Anyone here on the list using LXC containers for testing Chef runs already?
So I will probably just laze
> out on that step and write unit/basic integration tests with the
> minitest handler, then write up some Cucumber features to test each VM
> after convergence and before teardown. One place where I do see value
> in that would be for our LWRPs, a few of which are getting pretty
> complicated and could probably do with some specs running that could
> catch regressions.
>
> The only problem I have (and this is due to my particular
> implementation) is that I'm testing role/environment/platform
> combinations. Can't really test a multi-tiered app this way. It's
> one of the things test-kitchen does that my little skunkworks project
> doesn't.
Yeah, that's where test-kitchen really rocks.
I guess the answer is to bind some features to a special
> full-stack role that we would only use for testing/development, though
> that wouldn't work for apps that require multiple hosts (Mongo springs
> to mind).
>
For testing a complex setup like this I would go for a multivm Vagrantfile and test the real scenario with multiple VMs - otherwise you are testing something that dosen't reflect the real situation.
Still it would be useful to have something like a special role for development in addition.
> Your documentation and the test-writing docs on the Chefspec and
> Fauxhai sites are pretty comprehensive, it seems (or have I just been
> reading too much poorly-documented code lately? ;) ). I'm not sure
> how many people are interested in converging Vagrant VMs against a
> Chef server endpoint (i.e. using the chef-client provisioner instead
> of chef-solo), but jtimberman wrote a couple of pretty good blog posts
> on the subject (starting with
> http://jtimberman.housepub.org/blog/2012/03/18/multivm-vagrantfile-for-chef/
> , again, HI GOOGLE! ) and I could perhaps write something up as well
> if the community desires it and existing resources are found wanting.
>
I think thats useful and will probably do that for my current project. I'll probably be using knife-server for painlessly setting up dedicated chef Servers for my test runs on the fly.
Thanks for the pointer to the blog post, found some valuable tips there!
Cheers, Torben
> Anyway, I think this kind of testing is a neat idea and I'm going to
> see how far I can get with it. I try to follow this list but I'm
> awash in work email most of the time. Comments/off-list conversations
> on this subject with interested parties are also welcome.
>
> On Thu, Oct 4, 2012 at 10:22 PM, Torben Knerr <
">
> wrote:
> > Love the idea of properly tested cookbooks and the fact that you are pushing
> > your buddies into that direction :-)
> >
> > I've taken some notes and collected some links about test-driven cookbook
> > development:
> > https://github.com/tknerr/bills-kitchen/blob/master/COOKBOOK_DEVELOPMENT.md
> >
> > It covers cookbook testing on several levels:
> > * syntax & long checking -> foodcritic
> > * unit-level spec testing -> chefspec and fauxhai
> > * "smoke tests" from the inside -> minitest-handler
> > * acceptance-level testing from the outside -> cucumber-nagios
> >
> > Maybe you can make some use of it or you have some ideas how to improve it.
> >
> > Cheers, Torben
> >
> > Am 05.10.2012 03:17 schrieb "steve ." <
">
>:
> >
> >> Continuous cookbook testing framework (analogous to test-kitchen, but
> >> built on Jenkins, Foodcritic, RVM and Vagrant) ... check.
> >> Chef-minitest handler running on VMs and reporting test results back
> >> to CI host ... check.
> >>
> >> Now I just have to get all my cookbook authors to write tests!
> >>
> >> It seems like a lot of us are in this situation, so I thought I'd
> >> share my findings and the direction in which I seem to be headed. We
> >> have to get a bunch of people who are maybe not so used to TDD/BDD or
> >> writing integration tests to suddenly start writing integration tests
> >> in minitest (spec tests, in my exact case).
> >>
> >> There's a growing amount of source code out there, but the best
> >> documentation I've found is still in Bryan's example cookbook (found
> >> here, if this somehow pops up on Google):
> >>
> >> https://github.com/btm/minitest-handler-cookbook
> >>
> >> So you can use node data, but only within the scope of a particular
> >> test definition. I understand why that is, but folks who know "enough
> >> Ruby for Chef" will probably get tripped up by that.
> >>
> >> Still, I need a way to tell my budding cooks what needs testing and
> >> what doesn't. Can I use existing code coverage tools to do this
> >> *outside of converging a node and running the minitest handler?* As
> >> far as I can tell the answer is, no, I can't.
> >>
> >> (I base this preliminary conclusion on an indexed conversation in
> >> #chef from July:
> >>
> >> http://community.opscode.com/chat/chef/2012-07-18#id-165241
> >>
> >> ... would love to be proven wrong, BTW)
> >>
> >> My current idea for drawing attention to which cookbooks are
> >> adequately tested in our Chef repository involves writing some
> >> Foodcritic rules to check for the presence of minitests in each
> >> cookbook and then critique them. I believe I can get started with
> >> this using three rules:
> >> "Does your cookbook have a files/default/tests/minitest directory?"
> >> "Do you have a test/spec.rb file describing each recipe in the cookbook?"
> >> "Does your test/spec for recipe X cover all of the resources specified
> >> in recipe X?".
> >>
> >> Since Foodcritic testing is already in the early stage of my cookbook
> >> testing pipeline, I can easily turn those warnings into pretty graphs.
> >> I'm worried that writing those rules will be a bit brain-melty,
> >> though.
> >>
> >> Mission accomplished: I am shaming all my repo contributors for not
> >> writing tests for their cookbook contributions. I still need to make
> >> writing those tests easier for them to pick up or I'm just being mean.
> >> My idea here is to write and exhaustively document a helper gem that
> >> provides convenience functions beyond the resource checks that are
> >> already included, then distribute that along with the recipe that
> >> installs the chef-minitest-handler gem and sets it up as an event
> >> handler on the system. If the helper gem is well-documented and the
> >> tests in the repository leverage the helper properly, I think that
> >> will be enough signposts to get us started. Then It'll be a lot
> >> easier to accept all these pull requests...
> >>
> >> Is anyone else proceeding along these lines? Is there any interest in
> >> adding that functionality to mainline Foodcritic? Is there some other
> >> way of ensuring test coverage on cookbooks? Any fellow travelers know
> >> of any pitfalls or open source projects that might help?
> >>
> >> (besides test-kitchen, that is. ;) )
Archive powered by MHonArc 2.6.16.