[chef] Re: Re: Serverspec vs Mintest Feedback

[Torben Knerr < >]

I always thought of minitest-handler and test-kitchen / serverspec as
being two orthogonal things:

- minitest-handler for "smoke tests" that run from the inside
everytime the system converges (e.g. check if processes are up and
running, etc.)
- kitchenci / serverspec for "integration tests" that still run from
the inside but not necessarily after each run (i.e. on demand via
`kitchen test`)

Actually, now that I'm writing it... the kitchenci tests actually run
inside the node, right?

What are the preferred frameworks for acceptance-level testing from the 
outside?

I had played with cucumber-nagios [0] once, but it doesn't seem
widespread in the chef community. Anyone having experience with
leibniz [1] btw?

Cheers,
Torben

[0] http://auxesis.github.io/cucumber-nagios/
[1] https://github.com/Atalanta/leibniz

On Mon, Jan 5, 2015 at 4:48 PM, Joshua Timberman 
<
 >
 wrote:
> Ohai!
>
> On Sat, Jan 3, 2015 at 1:00 PM, David Petzel 
> <
 >
>  wrote:
>>
>> TLDR - I'm interesting in knowing why folks are moving away from Minitest,
>> toward ServerSpec.
>
>
> As Julian said, there's an element of not wanting to "contaminate" the
> tests. However, there's a couple other reasons.
>
>>
>> I've been noticing a lot of cookbooks recently have been migrating their
>> testings from minitest to ServerSpec. I've got nothing against ServerSpec,
>> but I'm curious why/where this trend is coming from.
>
>
> Back in the before times, that is, before `test kitchen` and the notion of
> `bussers`, the best (only?) way of reasonably testing nodes post convergence
> was through the minitest chef handler. The early version of test kitchen
> (pre 1.0) did run those, but the rewritten version, with the bussers made it
> more flexible.
>
> I think Serverspec is popular because Rspec is popular. It's not a value
> judgment, just more widely used - Rspec is what CHEF uses for chef itself,
> Ruby and Ruby on Rails programmers are familiar with Rspec. While minitest
> is in the Ruby stdlib (and I like it more than Rspec for that reason), it's
> simply not widespread. Also, Serverspec does an "outside in" kind of test
> approach. I have more confidence that the tests are doing the right thing. I
> can defer configurable options to attributes, and then set those attributes
> on a per-suite basis.
>
>> Maybe my question would be better asked in the context of an example...
>> Lets assume I am consuming a community cookbook which sets up some
>> application that has an HTTP port. There is an attribute that controls what
>> that HTTP port is.
>> In ServerSpec land, I think you need to hard code the HTTP port to test?
>> In Minitest, I have access to the node object, so my test is not hard
>> coded to a port, but instead uses the value of the node attribute.
>>
>>
>> For the sake of the discussion, lets say I run that application on port 78
>> in dev, 79 in qa, and 80 in production. While using minitests I'm able to
>> assert the application is working properly, because the minitest test would
>> be reading the node attribute. However from what I've seen with ServerSpec,
>> I don't have that same ability to adapt the test to use the right port,
>> without manual intervention?
>
>
> This is why I like, and use test kitchen. I can set up two suites.
>
> suites:
>   - name: default
>     run_list:
>       - recipe[webapp]
>     attributes:
>   - name: port78
>     run_list:
>       - recipe[webapp]
>     attributes:
>       www:
>         port: 78
>
> Then I can have tests for each suite, one that checks that my web app
> responds on port 80 (presuming its the default), and responds on port 78. I
> don't actually care about the different ports in different environments, I
> want to test that the non-default port does the right thing. Adding more
> suites that only change this attribute just tests that Chef works, which
> we've already proved in the first case. In fact, if the port is an
> attribute, I don't think the `default` suite is really *really* necessary.
>
> Cheers,
> Joshua
>