[chef] Re: Re: Re: Re: Re: Re: Re: Testing Cookbooks vs Testing Infrastructure

[Zac Stevens < >]

On Wed, Apr 22, 2015 at 8:06 PM, Lamont Granquist < " target="_blank"> > wrote:

On 4/22/15 11:28 AM, Torben Knerr wrote:

A more concrete example: say I have a cookbook for setting up
JenkinsCI with Slaves on multiple nodes, with nginx in front, etc..
I'd like to have a test suite that:
- sets up the multi-node scenario
- then verifies that the front page is served...
- ...and that you can create a basic job that is executed on a slave
- ...and so on (some basic acceptance level smoke tests)

That's what I'm up to... did it get clearer?

My intuition is that the way that you're thinking about that violates promise theory a bit.  All of your tests should be "from the inside".  The VM that your webapp is being deployed on should really only test the world from its perspective.  It should test that its listening on its http port and that basic API calls can succeed and that it can make calls to a database.  Trying to think about it begin responsible for testing its own VIP violates promise theory since now its responsible for distant infrastructure and it doesn't have the information that it may need (it could be failing to register correctly in the VIP but another server in the cluster could be serving content so the VIP is up).  Its better to add tests to the loadbalancer "from the inside" so that the load balancer checks itself and validates that all its exepected clients are listed as avaiable and health checks are passing and the loadbalancer should be checking its own VIP.

Then what we need is a multi-node test-kitchen which used something like chef-provisioning to spin up all of these VMs and fire off the serverspec runs on all of them, and then report back.  For deployed infrastructure then audit mode could be running serverspec tests (and ideally hook busser up to audit mode so you could audit with bats tests and whatever) and you'd be getting reports back on every chef-client converge or audit mode run.

I really don't understand the conclusion that there _should not_ be any tests "from the outside".  (I also don't see anything in Torben's example prescribing where the outside-in tests should be run from)

All the tests described above are valuable, but testing that individual components work as expected is not a substitute for verifying that they work as expected when assembled.  Or put another way, unit tests are great but integration tests are also valuable - I'm of the opinion that this is true whether the units you're dealing with are classes or servers.

I'd love to see multi-node testing in test-kitchen.  I'd love to spin up a set of VMs that are configured to work together.  I'd love to run serverspec suites on each of them, to assert their behaviour at that level.  I'd love to spin up another VM to run additional outside-in tests making assertions about the behaviour of the whole.  This would be _slow_, and I wouldn't be running these integration tests unless everything looked good, but they would be _meaningful_.

Here's the kicker: I thought that this was obvious, something that everyone wanted to do, and it was simply a matter of time before Test Kitchen evolved to deliver it.  Now you've given me pause for thought.

Is this a philosophical question and I didn't realise I was on a side?

Is there no debate and I'm simply wrong?

Did I completely misunderstand your point, and now look a bit silly for writing this?

Zac