[chef] Chef server high cpu usage (not functional)

[Silviu Dicu < >]

Hello,

I have a server version 11.04 with 820 nodes. From time to time the server stops

responding. I see high cpu usauge (on 16GB with 4 cpu) each cpu is busy at least 60%.

The curl http://server/_status gives failed postgresql and sometime the search.

When I'm looking into postgress i don't see much happening

31.464393+00 | 2014-01-08 07:50:31.464876+00 | f       | idle   | SELECT 'pong' as ping LIMIT 1

 16384 | opscode_chef | 1165 |    16549 | opscode_chef  |                  | 127.0.0.1   |                 |       50467 | 2014-01-08 07:41:24.756074+00 |                              | 2014-01-08 07:50:

31.552503+00 | 2014-01-08 07:50:31.552891+00 | f       | idle   | SELECT 'pong' as ping LIMIT 1

 16384 | opscode_chef | 1442 |       10 | opscode-pgsql | psql             |             |                 |          -1 | 2014-01-08 07:45:22.293381+00 | 2014-01-08 07:50:32.44201+00 | 2014-01-08 07:50:

....

....

I increased the values for chef-server.rb as follow

postgresql['max_connections'] = 400

erchef['db_pool_size'] = 100

chef_expander['nodes'] = 4

chef_solr['heap_size'] = 8192

When the load will go around 4-5 the server will stop working.

Any suggestion will be welcome.

thanks!

-silviu