[chef] 500 error on server when client has large passwd file?

[Jonathan Proulx < >]

Hi All,

this is my first crack at chef so I'm prepared to be wildy wrong, but
I think it cannot handle the size of my default password file.

the setup:

I'm using 0.10.8  opensource server running on Debian 6.0
(stable/squeeze) from opscode apt repository

workstaion is also Debian/Squeeze 0.10.8 from OpsCode repo

I have two nodes setup both installed with knife bootstrap from my
workstaion:

node1 is a bare install of ubuntu-server 11.04

node2 is our current standard Debian/Squeeze workstation install which
includes all kinds of packages and all of our user accouts (spoiler:
this node fails to run but reducing number of accounts makes it work):

wc -l /etc/passwd
2551 /etc/passwd

both have empty runlists.

node1 bootstrapped nicely

node2 got all the packages and configs to make chef-client go
registered client and node on the server but on first run:

 [Thu, 23 Feb 2012 16:52:05 -0500] INFO: *** Chef 0.10.8 ***
[Thu, 23 Feb 2012 16:52:09 -0500] INFO: Run List is []
[Thu, 23 Feb 2012 16:52:09 -0500] INFO: Run List expands to []
[Thu, 23 Feb 2012 16:52:09 -0500] INFO: Starting Chef Run for  
node2.csail.mit.edu
[Thu, 23 Feb 2012 16:52:09 -0500] INFO: Running start handlers
[Thu, 23 Feb 2012 16:52:09 -0500] INFO: Start handlers complete.
[Thu, 23 Feb 2012 16:52:10 -0500] INFO: Loading cookbooks []
[Thu, 23 Feb 2012 16:46:55 -0500] WARN: Node node2.csail.mit.edu has an empty 
run list.
[Thu, 23 Feb 2012 16:46:59 -0500] INFO: HTTP Request Returned 500 Internal 
Server Error: 400 "Bad Request"
[Thu, 23 Feb 2012 16:46:59 -0500] ERROR: Server returned error for
http://chef.csail.mit.edu:4000/nodes/lore.csail.mit.edu, retrying 1/5
in 3s

Looking on the server I notices an insanly long line logs with all the
ohai info (most notably user account info) and these errors after:

merb : chef-server (api) : worker (port 4000) ~ 400 "Bad Request" - 
(Net::HTTPServerException)
/usr/lib/ruby/1.8/net/http.rb:2105:in `error!'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:245:in `api_request'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:296:in `retriable_rest_request'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:226:in `api_request'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:130:in `put_rest'
/usr/lib/ruby/vendor_ruby/chef/couchdb.rb:114:in `store'
/usr/lib/ruby/vendor_ruby/chef/node.rb:618:in `cdb_save'
/usr/share/chef-server-api/app/controllers/nodes.rb:69:in `update'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in `send'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in 
`_call_action'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:289:in 
`_dispatch'
/usr/lib/ruby/1.8/merb-core/controller/merb_controller.rb:252:in `_dispatch'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:102:in `dispatch_action'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:74:in `handle'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:36:in `handle'
/usr/lib/ruby/1.8/merb-core/rack/application.rb:17:in `call'
/usr/lib/ruby/1.8/merb-core/rack/middleware/static.rb:28:in `call'
/usr/lib/ruby/1.8/rack/content_length.rb:13:in `call'
/usr/lib/ruby/1.8/thin/connection.rb:76:in `pre_process'
/usr/lib/ruby/1.8/thin/connection.rb:74:in `catch'
/usr/lib/ruby/1.8/thin/connection.rb:74:in `pre_process'
/usr/lib/ruby/1.8/thin/connection.rb:57:in `process'
/usr/lib/ruby/1.8/thin/connection.rb:42:in `receive_data'
/usr/lib/ruby/1.8/eventmachine.rb:257:in `run_machine'
/usr/lib/ruby/1.8/eventmachine.rb:257:in `run'
/usr/lib/ruby/1.8/thin/backends/base.rb:57:in `start'
/usr/lib/ruby/1.8/thin/server.rb:156:in `start'
/usr/lib/ruby/1.8/merb-core/rack/adapter/thin.rb:30:in `start_server'
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:298:in `start_at_port'
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:128:in `start'
/usr/lib/ruby/1.8/merb-core/server.rb:174:in `bootup'
/usr/lib/ruby/1.8/merb-core/server.rb:159:in `daemonize'
/usr/lib/ruby/1.8/merb-core/server.rb:143:in `fork'
/usr/lib/ruby/1.8/merb-core/server.rb:143:in `daemonize'
/usr/lib/ruby/1.8/merb-core/server.rb:35:in `start'
/usr/lib/ruby/1.8/merb-core.rb:170:in `start'
/usr/sbin/chef-server:86

Restricting /etc/passwd on node2 to just system accounts and myself:

wc -l /etc/passwd
26 /etc/passwd

chef-client run succeeds