[chef] how to debug poor performance problems on Chef server?

["Julian C. Dunn" < >]

I have a Chef 10.12 server with about 100 client nodes & recently
started having performance problems with Chef.

The symptoms are that the chef-server process is constantly pegging
the CPU. I'm running the server on an Amazon m1.xlarge; this issue was
killing our m1.medium.

The Chef server logs are not that useful at figuring out what requests
are being issued from what clients, and what they're doing. All I can
see that's anomalous are constant node updates like this:

merb : chef-server (api) : worker (port 4000) ~ Started request
handling: Fri Aug 24 17:25:21 -0400 2012
merb : chef-server (api) : worker (port 4000) ~ Params:
{"inflated_object"=>#<Chef::Node:0x2b7a6331f778
@run_list=#<Chef::RunList:0x2b7a6331e288
@run_list_items=[#<Chef::RunList::RunListItem:0x2b7a632c6e70
@version=nil, @name="Mgmt", @type=:role>]>, ....
@couchdb=#<Chef::CouchDB:0x2b7a6331df40 @db="chef",
@rest=#<Chef::REST:0x2b7a6331ddb0 @redirects_followed=0,
@auth_credentials=#<Chef::REST::AuthCredentials:0x2b7a6331dcc0
@client_name=nil, @key_file=nil>, @cookies={}, @sign_request=true,
@default_headers={}, @disable_gzip=false,
@url="http://localhost:5984", @sign_on_redirect=true,
@redirect_limit=10>>>, "format"=>nil, "action"=>"update",
"id"=>"real-host-name-elided", "controller"=>"nodes"}

even though the chef-client on the node mentioned,
"real-host-name-elided", is running far less frequently than I see
these messages.

How can I start debugging the issue, short of putting a tcpdump on
port 4000? How come merb doesn't show what the requesting client is?

- Julian