[chef] Re: What kind of load can Chef Server support?

[Brian Akins < >]

On Jan 27, 2012, at 9:27 PM, 
<
 >
 
<
 >
 wrote:

> Anyone had any experiences with loading chef server with a large number of
> nodes?

It depends on what the recipes are doing.  We can have several dozen nodes of 
one type hitting chef-server at the same time and everything is fine. 
However, other types, we can kill the server with just a few nodes.  If you 
have a large number of nodes, full node searches -- search(:node, "*:*") -- 
is a good way to kill chef-server.  When we were running chef-server under 
unicorn, it took x nodes running that search to kill chef-serevr where x was 
the number of unicorn processes.  No other nodes could run chef.  Our problem 
was that we regularly triggered chef to run on those nodes at the same time. 
We tried larger splay, etc.  We switched to rainbows and when this situation 
happens, other nodes can still run chef, just very slowly.  Ultimately, we 
figured out a way to remove the full node search from those recipes.  Yes, I 
know trimming ohai data would have helped, but we want/need that data and it 
would have just delayed the eventual problem, I think.

We still have issues with chef-server getting slow.   It generally is caused 
by certain things happening in recipes (or via knife).  We run chef-server, 
solr, and couch on a fairly "wimpy" box - 4 cores, 8GB RAM.  This was our 
"proof-of-concept" hardware that just got rolled into production.  It's been 
on our list for weeks to migrate to a larger box. Cobbler's children and all 
that.

FWIW, with rainbows it seems each individual chef run is slightly slower (17 
seconds vs 15seconds, for example) but we can run several more concurrently 
before noticing any slowdown.We bumped up unicorn process count, but memory 
usage became an issue. We front chef-server with nginx.  

Looking at the chef-server api, it seems it would be fairly easy to write a 
version in Lua embedded in nginx.  But I guess that's not as cool as erlang ;)

--Brian