[chef] Re: Re: Re: ideas on testing a clustered application w/ Test-Kitchen

[Bryan Berry < >]

josh,

I think chef-workflow is neat but it isn't able to reuse
test-kitchen's yaml format nor its drivers. I also think that workflow
I have described isn't overly complicated to need custom rake tasks.

On Wed, May 8, 2013 at 7:35 AM, Joshua Timberman 
<
 >
 wrote:
> Have y'all looked into Erik Hollensbe's chef-workflow? I haven't used it yet
> myself but it shows a lot of promise for this kind of use case.
>
> https://github.com/chef-workflow
>
>
>
> --
> Joshua Timberman
>
> On Tuesday, May 7, 2013 at 2:31, John Dewey wrote:
>
> Would also be nice if the runner doesn't use chef-solo.  I like where
> Vegabond is
> going.  The ability to truly do integration testing by spinning up dependent
> systems/services,
> and test the cookbook in isolation is needed.
>
> I have abandoned test kitchen all together in favor of chefspec, since I can
> stub/mock
> dependencies that chef-solo cannot resolve.  Looking forward to you work, so
> we can
> add test kitchen to the openstack cookbooks, and get those much needed
> integration
> tests.
>
> For example, I have a use-cases were a cookbook depends on rabbit, mysql,
> memcached to be
> running, and those roles registered into the chef server.  At that point the
> prerequisites
> are met and the cookbook can install recipes, and perform assertions.  This
> is where I
> like the Vegabond approach.
>
> John
>
> On Tuesday, May 7, 2013 at 1:10 AM, Bryan Berry wrote:
>
> I have some ideas on extending test-kitchen to test clustered
> applications and I would love some feedback before I go coding off in
> a particular direction.
>
> Problem: I deal primarily with distributed applications and testing
> the related cookbooks can be a pain. I also have to make sure these
> cookbooks work across different linux distros. Test-Kitchen was not
> originally created with this use case in mind though at this time I
> don't see any reason it couldn't support this use case.
>
> Vagabond[1], written by Chris Roberts, extends .kitchen.yml to include
> a clusters component, among other things. I would love to see
> test-kitchen absorb some of that functionality or at least provide
> extension points to make this more easily pluggable.
>
> Testing a cluster works differently than testing an individual node. I
> want to interrogate the state of the cluster as a whole, not look
> inside each individual server. To do this I need to wait until all
> servers in a cluster converge or at least a quorum of them do. Once a
> quorum of nodes has converged, run a series of tests against the the
> cluster. These tests execute on the client executing `kitchen` rather
> than inside the nodes of the cluster.
>
> Here are the steps in brief:
> 1. Converge all nodes in a cluster
> 2. Wait for quorum of nodes to converge
> 3. execute tests against the cluster
>
> I would put the tests for a cluster in my_cluster/test/cluster/cluster_name
>
> Applications like Elasticsearch, Zookeeper, or Cassandra don't have a
> master node, so each node has an identical run_list and attribute set
>
> clusters:
> default:
> - member: zk1
> - member: zk2
> - member: zk3
>
> platforms:
> - name: ubuntu-12.04
> - name: centos-6.3
>
> suites:
> - name: default
> run_list: [ "recipe[zookeeper]" ]
>
> To test the default cluster on CentOS, `kitchen test --cluster
> default --platform centos-6.3 `
>
> Let's make this even DRYer
>
> clusters:
> default:
> node_count: 3
> quorum: 2
>
> platforms:
> - name: ubuntu-12.04
> - name: centos-6.3
>
> suites:
> - name: default
> run_list: [ "recipe[zookeeper]" ]
>
> The test for this example zookeeper cluster would connect to one
> zookeeper node, make sure it sees the other zookeeper nodes. I am
> paraphrasing the zookeeper code because I am not certain of what the
> actual call would be.
>
> my_cluster/test/cluster/default/check_members.rb
>
> require 'zk'
>
> describe "zookeeper cluster" do
> before(:all) do
> @zk = ZK.new(some_ip)
> end
>
> it "sees the other members" do
> peers = @zk.get("system/peers")
> peers.should == ACTUAL_PEERS
> end
>
> end
>
> The primary challenge here is resolving the names of the members of
> the cluster. One way to do this is to access the statefiles for the
> nodes in .kitchen/*.yml. Another would be to somehow access the
> @instances array for the current Kitchen config. Yet another option
> would be to stand up a chef-zero server.
>
> What about a distributed app where each node does not have an
> identical run_list? Here is how I would handle that for something like
> HBase that has a "hmaster" that stores metadata for the whole cluster.
>
> clusters:
> default:
> - member: head1
> run_list: [ "recipe[hbase::hmaster]" ]
> - member: store1
> run_list: [ "recipe[hbase::data_node]" ]
> - member: store2
> run_list: [ "recipe[hbase::data_node]" ]
>
> platforms:
> - name: ubuntu-12.04
> - name: centos-6.3
>
> suites:
> - name: default
> run_list: [ "recipe[zookeeper]" ]
>
>
>
> I know that test-kitchen is an unopinionated tool and attempts to be
> workflow agnostic but I feel that what I have shown here is a fairly
> simple workflow. The cluster definition I have presented is not
> suitable for modeling failover in a cluster or specific states. At
> that point one should use a custom workflow tool like chef-workflow or
> simply custom rake tasks.
>
> Depending on the feedback I get from this I plan to extend
> test-kitchen or Vagabond to handle the workflow I have described here.
> Thanks for reading!
>
>
> 1. https://github.com/chrisroberts/vagabond/tree/develop
>
>
>