[chef] Re: Re: Re: Chef setup has become unstable

[KC Braunschweig < >]

On Sat, Jun 16, 2012 at 12:41 PM, Madhurranjan Mohaan
<
 >
 wrote:
> Do you think we should scale out ? If yes, what services do you think we
> should run on different servers? Also, on my end, I am trying to see if all

Regarding the instability, I can tell you I had issues on RHEL 5.7
because the versions of couchdb and erlang were old. Newer packages
probably would have fixed it, but I upgraded to RHEL 6.1 which also
had newer versions and things were happier. Doesn't sound exactly like
your instability, but worth considering.

Regarding the performance issues, I hope that Josh was joking. 160
nodes is nothing. Are they converging every 30 minutes? Do you have a
reasonable splay? Are your recipes very search heavy? It could be a
lot of things, but I'd start with considering the concurrency on the
server API. Are you running a single Thin process for the API server?
If so, consider running multiple processes with proxy balancer or some
such in front of them. Alternatively switch the server to run in
unicorn with nginx in front of it. I've been happy with unicorn so
far.

I don't think you should be there yet, but 4gb is probably not gonna
be enough forever. Eventually solr will want more heap and you'll need
memory as you add api server workers and couch will take whatever's
left. Which leads back to either adding memory or Josh's point of
splitting components on different servers. That's eventually though,
I'd hope you could get at least a couple hundred nodes with your
current VM and 1000+ with 8gb without too much trouble.

To give you an example, I have a preprod server with about 1000 nodes:
RHEL 6.1 VM
8gb
4 virtual cores
unicorn - 8 api workers, 2 webui workers
solr - 2gb heap
chef 0.10.4

KC

> On Sat, Jun 16, 2012 at 7:25 PM, Joshua Timberman 
> <
 >
> wrote:
>>
>> Are you running all the chef server services on one machine? What is the
>> hardware spec of it? 160 nodes is quite a few. Sounds like you may need to
>> start scaling out the server and run services on separate systems.