[chef] Re: Re: Re: Re: Re: Re: Testing Cookbooks vs Testing Infrastructure

[Mike < >]

Torben,

I think it's easy to get really deep in how one wants to perform acceptance testing.

In our deployment cycle, we deploy the changes to a node, then run a series of phantomjs tests that load a web page, log in, clock a few things, assert behavior.

This is totally customized for our app, but there's no reason anyone couldn't write their own tests.

I don't think it's ever going to be the responsibility of Test-Kitchen/Serverspec - rather these methods could use bussers to launch your acceptance testing, or not - that's up to you.

-M

On Wed, Apr 22, 2015 at 2:28 PM, Torben Knerr < " target="_blank"> > wrote:

Sorry, I was probably not specific enough :-)

Serverspec is asserting state by ssh'ing into the VM, then checking
for processes, files being present, services being started, etc... (I
call that "from the inside")

However, if all that works, it doesn't necessarily mean that it works
for the end user (lets say the webapp is served on port 80 and
accessible "from outside" of the VM. That's probably a grey zone where
infrastructure testing ends and you the acceptance testing for a
specific application starts.

A more concrete example: say I have a cookbook for setting up
JenkinsCI with Slaves on multiple nodes, with nginx in front, etc..
I'd like to have a test suite that:
- sets up the multi-node scenario
- then verifies that the front page is served...
- ...and that you can create a basic job that is executed on a slave
- ...and so on (some basic acceptance level smoke tests)

That's what I'm up to... did it get clearer?

Cheers,
Torben

On Wed, Apr 22, 2015 at 3:54 PM, Tensibai < "> > wrote:
> We wish to have integration tests launched from jenkins too, that's not
> really outside, but the idea is to test the functional state in IC before
> promotion to QA with human testers behind load balancer and reverse proxy.
>
> I may misunderstand what you think of under acceptance term, sorry if it's
> the case :)
>
> Le 2015-04-22 15:37, Torben Knerr a écrit :
>
> Are you planning to (acceptance level) test it from the outside as well?
>
> Or just for convergence to succeed?
>
> Am 22.04.2015 14:46 schrieb "Tensibai" < "> >:
>>
>> I forgot the link to terraform: http://www.terraform.io/intro/index.html
>>
>> Le 2015-04-22 14:41, Tensibai a écrit :
>>
>> I had a discussion with a colleague this morning about this and pointed to
>> Terraform[1].
>>
>> As we're already using vagrant with a custom provider (for vsphere) we're
>> planning to move toward this path. for now it's just an idea with no
>> feedback but the described way Terraform works sounds interesting to write
>> "plans" to ensure idem-potency for cookbooks (spin up a machine, converge it
>> once, re-converge it, change a parameter somewhere, converge again, ensure
>> expectation are there).
>>
>> That's what we're aiming to do, launched from jenkins on cookbook commit
>> after usual linting/chefspec test.
>>
>> I'll try to write something about it if we end up with something
>> functional at end.
>>
>> Le 2015-04-22 07:40, Torben Knerr a écrit :
>>
>> Hey everybody,
>>
>> @jtimberman's recent blog post about test-driven infrastructure with
>> Chef made me thinking about this again.
>>
>> All the tools we currently have are focused on cookbook testing, but
>> we have nothing established for acceptance level infrastructure
>> testing yet, do we?
>>
>> I mean specifically:
>>
>>  * testing from the outside vs from the inside
>>  * testing multi-vm setups
>>
>> See also my comment on the original post:
>>
>> https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/
>>
>>
>> What do we have in this space?
>>
>> Cheers,
>> Torben
>>
>>
>>
>>
>>
>>
>>
>>
>
>
>
>