[chef] Re: Re: Re: how to set up cluster that has dependencies?

[Kevin Nuckolls < >]

Doozer is probably the most mature middle ground between Noah and Zookeeper. It relies on logical time which implements a version of paxos and is similar in spirit to Google's Chubby and Apache Zookeeper. If you don't require the notion of ephemeral nodes (which you might), then it's a really nice and lightweight solution for distributed configuration and distributed locks. It also happens to have a persistent data structure style interface so you can look back and see what the entire state of the configuration was at a given point in time. I think the part that has the most allure to me is that and the clarity of the API. A major design decision of doozer was a clean api so that clients could be easily written. It's a major part of the heroku architecture and they open sourced it recently. Your use case is one of the primary things they use it for.

Glu is also meant to solve the "last-mile" problem of dependent configuration management. It's built around zookeeper. It's meant to calculate differences between what the data and configuration on your nodes should be and what it is. From speaking with some who have worked with it, I hear it's less than an optimal solution. But a solution none-the-less. This was recently open sourced by LinkedIn.

For simple purposes, a hybrid solution of fabric/capistrano/rundeck/azkaban with the chef search api may be powerful enough albiet complex. Beyond that you'll need to look into systems which provide you with distributed locks. Unfortunately there's not a great deal of "canned" solutions for integration with chef other than noah but it is still a non-distributed single point of failure.

Since I seem to have thoughts on the matter, I'll go ahead and outline my opinions on what chef is good at and what it is not (yet) good at. To me it appears that there are four kinds of configuration.

1. Base system packages, utilities, and applications

2. Data (database backups, read-only data, etc)

3. Configuration that expects data to be in the correct spot at the right time

4. Multi-Node configuration (routing, orchestration/locks/cache management, group membership, node states)

IMO in trivial use-cases chef is very good at 1, good enough at 2 and 3 and can at least achieve 4 if you're very careful. In larger scenarios I believe it's only very good at #1 and perhaps the wrong abstraction entirely for 2, 3, and 4. I think these are important things to tell beginners because the breadth of what could be needed by chef in different environments has very high variance. The tooling surrounding these problems are still rapidly growing and different teams will have different needs. Unfortunately, for moderately complex architectures I feel like a passing understanding of what the following tools can do is worthy knowledge to obtain while attempting to design a new system that you wish to (fully or partially) automate with configuration management.

Papers worth reading:

Time, Clocks, and the Ordering of Events in a Distributed  System (grandaddy paxos paper)

http://research.microsoft.com/en-us/um/people/lamport/pubs/time-clocks.pdf

The Chubby lock service for loosely coupled distributed systems

http://labs.google.com/papers/chubby.html

Open source implementations / approximations of paxos / chubby:

http://xph.us/2011/04/13/introducing-doozer.html

https://github.com/ha/doozerd

http://zookeeper.apache.org/

Fits the same mold as the others but is not distributed:

https://github.com/lusis/Noah

https://github.com/lusis-cookbooks/noah

Dependency management / workflow managers:

http://sna-projects.com/azkaban/

http://rundeck.org/

Deployment managers:

https://github.com/capistrano/capistrano

http://docs.fabfile.org/en/1.2.2/index.html

Other useful things:

https://github.com/linkedin/glu

http://sna-projects.com/norbert/

Hope that helps.

-Kevin

@kevinnuckolls

On Wed, Sep 21, 2011 at 9:28 PM, Noah Kantrowitz < "> > wrote:

On Sep 21, 2011, at 7:18 PM, Aaron Abramson wrote:

> The best thing to do would be to go through one of the getting-started tutorials: http://help.opscode.com/kb/otherhelp/build-a-lamp-stack
>
> Watch that, follow along and deploy it yourself (if you have access to EC2), or just read through it.  It will give you a good idea on how nodes can query and search within templates.
>
> Look through the php-quick-start repo, the haproxy cookbook searches chef for the apache nodes and updates the templates accordingly.
>
> ----- Original Message -----
> From: "jeff stroomer" < "> >
> To: ">
> Sent: Wednesday, September 21, 2011 7:00:17 PM GMT -06:00 US/Canada Central
> Subject: [chef] how to set up cluster that has dependencies?
>
> Chef folks,
>
> I have a question concerning the best way to use Chef to set up a cluster of
> nodes that have dependencies on one another. (Apologies in advance if this is a
> naïve question, but I’m new to Chef.)
>
> For concreteness, suppose I want node  V  to run varnish, node T to run tomcat,
> and node  M to run mongo.  And let’s say that T needs to know the IP address
> of M, and V needs to know the IP address of T.        I believe  that each node can
> register its IP address in a database maintained on the Chef server, and
> recipes run by each node can query this database.  For things to work properly,
> I ought to set up M first, then T, and finally V.
>
> My question is this: How I should plan to use Chef so that the setup of various
> nodes happens in the right order?  Do  I write a recipe that sets up M first,
> then T, and finally V?         If so, then what is that recipe associated to?  Or
> should I instead have a recipe for V that sets up T, and also  have the setup
> recipe for T begin by setting up M?  Or should I write recipes for T, M, and V
> that query the database, and don’t do anything unless they can find the IP
> addresses they need?

Chef doesn't (yet) address this kind of multi-node orchestration issue. search() does make the integration parts easy, but that isn't enough to handle a highly fluid environment sometimes. One option is just careful recipe construction, usually just making the Chef run abort early if a search for a needed component comes up empty can do the trick, as long as you run Chef in polling mode on a tight cycle. Another option is to use something like RunDeck or Fabric to execute chef-client in the correct order and not move on with the deployment until certain gate conditions are met. Beyond that you enter into the world of tools like Noah and ZooKeeper, which are built very specifically for this. Noah is somewhat new, but is also less encumbered by legacy Java craziness compared to ZooKeeper. ZooKeeper is probably your best bet though, as it has a powerful and flexible set of distributed locking and configuration primitives. We (Opscode) are also very interested in exploring this space as it is indeed a common problem and while we don't want to end up with a poor reimplementation of one or all of these something with tighter integration to Chef recipes would be awesome. Hope that helps!

--Noah

PS: Also a semi-related shootout to AJ's new pylon tool to do distributed master elections.


-----BEGIN PGP SIGNATURE-----

iEYEARECAAYFAk56nToACgkQA9fEp2eRsU+ACQCgiY94ToO+5GMMRbyrhqWfWNmy
q2oAoKrbnEB90ety+b9k9/4NYHaWnfO5
=88yU
-----END PGP SIGNATURE-----