[chef] Re: Re: Re: Re: Re: Re: Re: Testing Cookbooks vs Testing Infrastructure

[Nate DAmico < >]

We have been doing range of this type of testing for while..., serverspec rocks, everybody testing infrastructure/service should be using it.  It can be used locally or remote via ssh.., we primarily use locally.  Our style is in test mode run serverspec, and possibly other test scripts, on each node with agent.  We run an orchestrator to do mix of testing consisting of a couple modes:

-post deploy+config testing/assertion

-post stage assertion (ie: spin up nodes 1,2,3.., configure stages 1,2.., then assert things are correct, if not, error out)

To @lamont's point.., if you are automating nodes, configuration, network, etc.., as part of a service rollout you might want/need to run external tests to assert the things are configured correctly for connectivity and certain SLA's that need to be met.  We typically will coordinate a bastion to do some of this work, or some equivalent to test for instance connectivity between an app node and db node for example.

On Wed, Apr 22, 2015 at 1:05 PM, Tensibai Zhaoying < " target="_blank"> > wrote:

Oops, forgot to say we're doing this integration tests with gatling to test the service from Jenkins but targeting the app server directly and not through all the bricks a user will

Le 22 avr. 2015 20:28, Torben Knerr < "> > a écrit :
>
> Sorry, I was probably not specific enough :-)
>
> Serverspec is asserting state by ssh'ing into the VM, then checking
> for processes, files being present, services being started, etc... (I
> call that "from the inside")
>
> However, if all that works, it doesn't necessarily mean that it works
> for the end user (lets say the webapp is served on port 80 and
> accessible "from outside" of the VM. That's probably a grey zone where
> infrastructure testing ends and you the acceptance testing for a
> specific application starts.
>
> A more concrete example: say I have a cookbook for setting up
> JenkinsCI with Slaves on multiple nodes, with nginx in front, etc..
> I'd like to have a test suite that:
> - sets up the multi-node scenario
> - then verifies that the front page is served...
> - ...and that you can create a basic job that is executed on a slave
> - ...and so on (some basic acceptance level smoke tests)
>
> That's what I'm up to... did it get clearer?
>
> Cheers,
> Torben
>
>
>
>
>
> On Wed, Apr 22, 2015 at 3:54 PM, Tensibai < "> > wrote:
> > We wish to have integration tests launched from jenkins too, that's not
> > really outside, but the idea is to test the functional state in IC before
> > promotion to QA with human testers behind load balancer and reverse proxy.
> >
> > I may misunderstand what you think of under acceptance term, sorry if it's
> > the case :)
> >
> > Le 2015-04-22 15:37, Torben Knerr a écrit :
> >
> > Are you planning to (acceptance level) test it from the outside as well?
> >
> > Or just for convergence to succeed?
> >
> > Am 22.04.2015 14:46 schrieb "Tensibai" < "> >:
> >>
> >> I forgot the link to terraform: http://www.terraform.io/intro/index.html
> >>
> >> Le 2015-04-22 14:41, Tensibai a écrit :
> >>
> >> I had a discussion with a colleague this morning about this and pointed to
> >> Terraform[1].
> >>
> >> As we're already using vagrant with a custom provider (for vsphere) we're
> >> planning to move toward this path. for now it's just an idea with no
> >> feedback but the described way Terraform works sounds interesting to write
> >> "plans" to ensure idem-potency for cookbooks (spin up a machine, converge it
> >> once, re-converge it, change a parameter somewhere, converge again, ensure
> >> expectation are there).
> >>
> >> That's what we're aiming to do, launched from jenkins on cookbook commit
> >> after usual linting/chefspec test.
> >>
> >> I'll try to write something about it if we end up with something
> >> functional at end.
> >>
> >> Le 2015-04-22 07:40, Torben Knerr a écrit :
> >>
> >> Hey everybody,
> >>
> >> @jtimberman's recent blog post about test-driven infrastructure with
> >> Chef made me thinking about this again.
> >>
> >> All the tools we currently have are focused on cookbook testing, but
> >> we have nothing established for acceptance level infrastructure
> >> testing yet, do we?
> >>
> >> I mean specifically:
> >>
> >>  * testing from the outside vs from the inside
> >>  * testing multi-vm setups
> >>
> >> See also my comment on the original post:
> >>
> >> https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/
> >>
> >>
> >> What do we have in this space?
> >>
> >> Cheers,
> >> Torben
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >
> >
> >
> >