[chef] Re: Re: Re: Re: High availability for chef

[Adam Jacob < >]

On Tue, Jul 5, 2011 at 11:51 AM, KC Braunschweig
<
 >
 wrote:
> I'd be interested in people's experience with the capacity of various
> Chef server configurations. I'm working on the HA setup as well.
> Active/passive failover seems relatively straightforward but the next
> question is when that won't be good enough. Anyone have numbers they'd
> be willing to post? Haven't started testing yet to see what the first
> bottleneck will be. Hoping that it'll scale well enough to handle our
> needs (only a couple thousand nodes) so that we won't have to do much
> more than split off the data components on separate nodes. Not
> terribly interested in mimicking the full multi-tier stack I imagine
> the platform runs on if I can avoid it.

For a couple of thousand nodes, assuming you have control over the
hardware, you can scale active/passive vertically.

You need to be aware of, and control:

1) The frequency of convergence on the chef-clients. (How often do
they check in, at what splay.) This will impact the number of API
workers you run (we recommend you do it on unicorn,) and the
configuration of the upstream proxy (we use nginx.) This is often
bound by memory, although CPU can be a pain point.

2) Make sure you are compacting the CouchDB database and the views on
a regular basis.

3) Solr may need tuning, specifically memory/heap/permgen space.

4) RabbitMQ really only does reliable HA by persisting to disk.

Best,
Adam

-- 
Opscode, Inc.
Adam Jacob, Chief Product Officer
T: (206) 619-7151 E: