[chef] Re: Re: Re: Re: Re: Re: Re: Re: Continuous Integration

[Dreamcat4 < >]

On Thu, Jul 8, 2010 at 5:05 PM, Michael Hale 
<
 >
 wrote:
> I'm thinking that the useful thing to test is NOT did chef install
> some package or setup a user, but rather after chef has run can I
> interact with the system as I would expect from an external
> perspective. For example:
>
> * Is the website accessible?
> * Are unused ports blocked?
> * When I send an email thorough the website does it end up in my inbox?
>
> Capybara (http://github.com/jnicklas/capybara) enforces this external
> perspective for webapp testing:

If you think about it, kinnda makes sense to call those tools like
capybara and webrat from within cucumber to help do the heavy lifting
for these very high level integration tests. At least to get things
going without spending hours and hours over it all.

Then whats still missing is a couple of things:

1) Having some standard way of hooking into a chef run, so that once a
change is commited to your git cookbooks, then a probably git post
recieve hook can trigger the chef run, and the subsequent testing.

As far as i know integrity continuus integration server would be the
best tool for that job

2) Secondly, having some common library of test helpers. These would
be re-usable and perform the heavy lifting for several categories of
common testing. For example take a nondescript TCP service. It might
be a webserver. Or it could be anything running over TCP/UDP. Eg
Samba, NFS, LDAP etc. We probably would want some low-level network
service test, to assert that a given host is listening on TCP ports
A,B,C + udp ports X,Y,Z.

In cucumber you would have to get the hostnames etc from the chef
attributes of your test network. So that might require querying the
chef api, i guess.


> "Access to session, request and response from the test is not
> possible. Maybe we’ll do response headers at some point in the future,
> but the others really shouldn’t be touched in an integration test
> anyway. "
>
> They only let you interact with screen elements that a user could
> interact with. It makes sense because the things that users interact
> with are what provides the business value and if that interface
> doesn't work it doesn't really matter.
>
> On Thu, Jul 8, 2010 at 10:38 AM, mikehale 
> <
 >
>  wrote:
>> On Thu, Jul 8, 2010 at 12:36 AM, Chad Woolley 
>> <
 >
>>  wrote:
>>> On Wed, Jul 7, 2010 at 9:14 PM, AJ Christensen 
>>> <
 >
>>>  wrote:
>>>> It seems what we're all really talking about is some kind of BDD 
>>>> framework
>>>> for Chef cookbooks - unit testing (also related to --noop) has proven
>>>> unreliable for this kind of thing: you can't mock UNIX.
>>>> I know there was work done on this in the past - mikehale's chef-bdd
>>>> project http://github.com/mikehale/chef-bdd for example.
>>>
>>> I don't get what that is for.  To only test the filesystem aspects of
>>> cookbooks?  Does it somehow mock out other calls to run services and
>>> such?  What about stuff like the hostname reference:
>>>
>>> http://github.com/mikehale/chef-bdd/blob/master/features/step_definitions/custom_steps.rb#L17
>>>
>>> Too bad there isn't a README...
>>
>> This code is very much a work in progress, and I haven't made any progress 
>> in awhile. I started out trying to mock everything, but it was pointed out 
>> to me that mocking UNIX is silly. My current thinking is to combine some 
>> cucumber steps like those in the aforementioned file with vagrant to auto 
>> provision a test cloud that matches a description. So the cucumber might 
>> look something like this:
>>
>> Given I have cluster X
>> When I have run chef-client on all the nodes
>> Then I should see "my load balanced website" on "http://example.com";
>>
>>>
>>
>>
>