[chef] Right sizing Chef11 Server

[< >]

Hi All,
I am wondering about any guidelines on 'right sizing' a Chef11 server.  I
understand things like your mileage may vary - but meanwhile usually with a
popular community supported product which also has a commercial edition there
are usually at least basic guidelines.

My situation is that we have approximately 2500 nodes distributed across four
data centers.  We have about 350ms round trip to the worst case data center.

What we did was turn up a single instance of Chef11 with 8-CPU and 32GB RAM. 
Guy before me went to all the Chef conferences, and I guess he must have drank
the kook-aid because we migrated from Chef10, added a few hundred nodes, and
Chef11 tipped over with '500'.

In all fairness, our original setup was set to have all nodes converge within 
a
5-minute splay time with a standard 30 minute cycle time.  Meanwhile, our
expectation was that Chef11 performs better.  We also moved everything (no 
more
Couch, etc) - onto a single server.

Workaround we did was to increase splay time to 30-minutes within 30-minute
schedule for now.

My impression is that we just installed Chef11 and did not spend any time
tuning the right knobs?  I have seen some posts where Postgres and such is
supposed to auto-size itself, but apparently that is based only on 
installation
and re-sizing does not work?

Sorry for lengthy post, sometimes context helps.   Questions are:



For a use case of:
5000 nodes
5 data centers
expected latency being around 300ms


Are there any knobs, dials, or other things that should be tuned to ensure 
that
a single Chef11 instance can handle that?   pg-sql, rabbit tuning jump to the
forefront for me.


Thanks in advance for help since this is my first post to the community and
generally like your product.

- Michael deMan