[chef] Re: Re: Re: Re: Re: Re: Re: Re: Testing Cookbooks vs Testing Infrastructure

[Lamont Granquist < >]

On 4/22/15 2:02 PM, Zac Stevens wrote:

" type="cite">

On Wed, Apr 22, 2015 at 8:06 PM, Lamont Granquist < " target="_blank"> > wrote:

On 4/22/15 11:28 AM, Torben Knerr wrote:

A more concrete example: say I have a cookbook for setting up
JenkinsCI with Slaves on multiple nodes, with nginx in front, etc..
I'd like to have a test suite that:
- sets up the multi-node scenario
- then verifies that the front page is served...
- ...and that you can create a basic job that is executed on a slave
- ...and so on (some basic acceptance level smoke tests)

That's what I'm up to... did it get clearer?

My intuition is that the way that you're thinking about that violates promise theory a bit.  All of your tests should be "from the inside".  The VM that your webapp is being deployed on should really only test the world from its perspective.  It should test that its listening on its http port and that basic API calls can succeed and that it can make calls to a database.  Trying to think about it begin responsible for testing its own VIP violates promise theory since now its responsible for distant infrastructure and it doesn't have the information that it may need (it could be failing to register correctly in the VIP but another server in the cluster could be serving content so the VIP is up).  Its better to add tests to the loadbalancer "from the inside" so that the load balancer checks itself and validates that all its exepected clients are listed as avaiable and health checks are passing and the loadbalancer should be checking its own VIP.

Then what we need is a multi-node test-kitchen which used something like chef-provisioning to spin up all of these VMs and fire off the serverspec runs on all of them, and then report back.  For deployed infrastructure then audit mode could be running serverspec tests (and ideally hook busser up to audit mode so you could audit with bats tests and whatever) and you'd be getting reports back on every chef-client converge or audit mode run.

I really don't understand the conclusion that there _should not_ be any tests "from the outside".  (I also don't see anything in Torben's example prescribing where the outside-in tests should be run from)

All the tests described above are valuable, but testing that individual components work as expected is not a substitute for verifying that they work as expected when assembled.  Or put another way, unit tests are great but integration tests are also valuable - I'm of the opinion that this is true whether the units you're dealing with are classes or servers.

I'd love to see multi-node testing in test-kitchen.  I'd love to spin up a set of VMs that are configured to work together.  I'd love to run serverspec suites on each of them, to assert their behaviour at that level.  I'd love to spin up another VM to run additional outside-in tests making assertions about the behaviour of the whole.  This would be _slow_, and I wouldn't be running these integration tests unless everything looked good, but they would be _meaningful_.

Here's the kicker: I thought that this was obvious, something that everyone wanted to do, and it was simply a matter of time before Test Kitchen evolved to deliver it.  Now you've given me pause for thought.

Is this a philosophical question and I didn't realise I was on a side?

Is there no debate and I'm simply wrong?

Did I completely misunderstand your point, and now look a bit silly for writing this?

Zac

This may be philosophical naval gazing.

The way that I'd test a cluster end-to-end would be to consider the load balancer part of the cluster and it should issue API calls to its own VIPs and should also assert that the health checks of all of its servers are healthy.  But the way that I express that is via the load balancer testing itself, but that API is the externally available interface to the cluster which is what you are going to test against for 'integration' or 'end-to-end' tests. 

So I suspect we get the same test coverage, but I think its important the way that I state who is responsible for testing what, and I don't see the need for 'external' testing and more pieces bolted onto the infrastructure to do 'new' kinds of tests.  I think the tools are already mostly there if you squint at it, and I agree that chef-provisioning and test-kitchen and serverspec/bats/etc seem to be the way forwards to deliver the whole thing and it will evolve to provide it.  All we need is to make test-kitchen more cluster-aware and then the process of simply being able to spin up the cluster, login and converge chef on everything and then run the tests on each individual node should be sufficient.  Even though there's no overall end-to-end testing framework being additionally added into test-kitchen, I don't think you'll miss it since by each node testing itself they also test their dependencies (it isn't like unit testing because you'll really be communicating with all your neighbors and not stubs or mocks of your neighbors -- if you fully test yourself, you should have also fully tested all of your adjacent dependencies and the whole should really be the sum of the parts).