[chef] Re: Meet test-kitchen

[Joshua Timberman < >]

On Aug 18, 2012, at 8:45 AM, Torben Knerr 
<
 >
 wrote:

> Absolutely Nice!

Thanks! We're all pretty excited about this. Andrew did a great job with it.

> 1) Concerning cookbooks with dependencies to other cookbooks:
> 
> Having noticed the librarian Cheffile in the
> apache2/test/kitchen/cookbooks directory, I wonder whether the
> cookbook dependencies therein are automatically resolved by
> test-kitchen, are they? Would a Berksfile be supported as well?

Yes - test-kitchen wraps up librarian-chef in the base runner:

https://github.com/opscode/test-kitchen/blob/master/lib/test-kitchen/runner/base.rb#L95-106

The reason Berkshelf isn't supported is simply because it didn't exist 
(publicly available) when test-kitchen was created and developed. 

> If these dependencies are indeed automatically resolved then that
> would be worth mentioning in the README.md

Would you please open a ticket in the KITCHEN project for that?

> 2) Do you have an example with test-kitchen tests running on Travis
> CI? Btw: can you actually spin up a vagrant VM on Travis?

I'm not sure if Travis would support that. I haven't done much with Travis 
besides wire it up with foodcritic and knife cookbook test. And by that I 
mean I've merged pull requests that do that :).

We have Jenkins internally for other projects, like building our omnibus 
packages, and the services that run Hosted/Private Chef. We also have the 
cookbooks tested under test-kitchen, though it is very simplistic right now.

I'd like to see Travis.CI support in each of our cookbooks, whether they're 
just doing the foodcritic/cookbook test, or running test-kitchen, that'd be 
great.

> 3) Concerning fast running tests:
> 
> It would be great to have fast running unit-like tests (which don't
> require to spin up a vagrant VM first) covered by test-kitchen as
> well. Chefspec comes to my mind, but I haven't found it referenced in
> the README.
> 
> Oh wait, isn't that already supported by the cookbook / script
> parameter which defaults to 'rspec spec'? Or would I do that within
> the cookbook / preflight command to make sure this runs _before_ the
> vagrant VM is spun up?

Yes, that should be done by the "script" parameter. You should be able to 
just run that in your cookbook directory without test-kitchen if you don't 
want the full provisioning cycle.

> 4) Concerning integration_test:
> 
> The README "Kitchenfile DSL" section talks about a features dir
> underneath the <cookbook>/test/kitchen dir, but in the examples I see
> only <cookbook>/test/features?
> 
> Also, is the "integration_test" block in the Kitchenfile really
> necessary to run the features? From the "Cucumber" section above,
> which says "Test Kitchen will run any features in the test/features
> subdirectory tagged with the configuration name in order to check that
> the service is working as expected.", it sounds like you wouldn't need
> to do anything extra to make the cucumber features run. In general,
> the "integration_test" section really confused me...

The cookbook features will automatically get run by test-kitchen if the 
directory exists.

The integration_test block is for non-cookbook projects (like 
mixlib-shellout) that have unit or integration testing. While mixlib-shellout 
is kind of a contrived example, a better one would be a Rails application 
that has its own test suite that should be run in a fully deployed scenario 
with its database, etc.

> 5) Runtimes?
> 
> I don't get what the cookbook and integration_test runtimes parameter
> is good for. Why does RVM need to be installed on the base box? Are
> you running the features locally on the target box? I would have
> expected that the features are run from the outside (e.g. on my
> workstation) against the provisioned target vm (the "black box"), so
> why would I care that RVM is installed inside that black box?!?

Rvm was used as a relatively simple way to get specific versions of Ruby 
installed for performing the tests. Think of a non-cookbook project, like a 
Ruby library that you want to test on different versions of Ruby (like 1.8.7 
vs 1.9.3, or even JRuby). My personal preference for cookbook testing is that 
we make the RVM part optional, and use the Ruby that is included in Chef on 
the base boxes already.

I created a ticket to track this:

http://tickets.opscode.com/browse/KITCHEN-4

-- 
Opscode, Inc
Joshua Timberman, Technical Program Manager
IRC, Skype, Twitter, Github: jtimberman