[chef] Re: Re: Re: Re: Chef setup has become unstable

[Sascha Bates < >]

Could it be the number of Windows servers and the astonishing amount of 
ohai data collected for Windows?  My understanding is that Windows ohai 
has an awful lot of data. I haven't worked with it in a few months so my 
memory is fading a bit and I was chef-solo anyway. 120 Windows nodes 
might produce a lot of data.

On 6/16/12 3:47 PM, KC Braunschweig wrote:
> On Sat, Jun 16, 2012 at 12:41 PM, Madhurranjan Mohaan
> <
>  >
>   wrote:
> > Do you think we should scale out ? If yes, what services do you think we
> > should run on different servers? Also, on my end, I am trying to see if all
> Regarding the instability, I can tell you I had issues on RHEL 5.7
> because the versions of couchdb and erlang were old. Newer packages
> probably would have fixed it, but I upgraded to RHEL 6.1 which also
> had newer versions and things were happier. Doesn't sound exactly like
> your instability, but worth considering.
>
> Regarding the performance issues, I hope that Josh was joking. 160
> nodes is nothing. Are they converging every 30 minutes? Do you have a
> reasonable splay? Are your recipes very search heavy? It could be a
> lot of things, but I'd start with considering the concurrency on the
> server API. Are you running a single Thin process for the API server?
> If so, consider running multiple processes with proxy balancer or some
> such in front of them. Alternatively switch the server to run in
> unicorn with nginx in front of it. I've been happy with unicorn so
> far.
>
> I don't think you should be there yet, but 4gb is probably not gonna
> be enough forever. Eventually solr will want more heap and you'll need
> memory as you add api server workers and couch will take whatever's
> left. Which leads back to either adding memory or Josh's point of
> splitting components on different servers. That's eventually though,
> I'd hope you could get at least a couple hundred nodes with your
> current VM and 1000+ with 8gb without too much trouble.
>
> To give you an example, I have a preprod server with about 1000 nodes:
> RHEL 6.1 VM
> 8gb
> 4 virtual cores
> unicorn - 8 api workers, 2 webui workers
> solr - 2gb heap
> chef 0.10.4
>
> KC
>
> > On Sat, Jun 16, 2012 at 7:25 PM, Joshua 
> > Timberman<
> >  >
> > wrote:
> > > Are you running all the chef server services on one machine? What is the
> > > hardware spec of it? 160 nodes is quite a few. Sounds like you may need to
> > > start scaling out the server and run services on separate systems.