Re: chef-client doesn't work "automatically"

[Albert Llop <mrsimo@gmail.com>]

After some sleep I decided that the "502 "Bad Gateway"" error looks too much like nginx, and I checked its erorr log (why didn't i do it before?

Whenever one of these nodes that try to contact the server and is in "zombie mode" like i like to call them (because chef-client is working... but not properly), nginx gives an error:

2009/06/19 11:11:38 [error] 27044#0: *40 upstream sent too big header while reading response header from upstream, client: 67.23.26.105, server: chef.kingsofmambo.net, request: "GET /openid/server?openid.assoc_handle=%7BHMAC-SHA1%7D%7B4a2f838d%7D%7Bhe1kgg%3D%3D%7D&openid.claimed_id=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fpotter_kingsofmambo_net&openid.identity=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fpotter_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete HTTP/1.1", upstream: "unix:/tmp/passenger.27019/master/helper_server.sock:", host: "chef.kingsofmambo.net"

The only information I can find googling is about problems with fastcgi, that I don't think have anything to do with chef (and passenger in general).

Any suggestions?

thanks again
-- 

Albert Llop
Tech Department
albert@kingsofmambo.com

2009/6/18 Arjuna Christensen <aj@opscode.com>

Hey!

On 18/06/2009, at 7:53 PM, Albert Llop wrote:

Hi again,
I set the log level to debug on one of the nodes, and restarted the client. After the restart it all worked fine for 14hours (until midnight more or less), and then stopped working.

The client log looks very similar with the debug option:

[Thu, 18 Jun 2009 00:33:28 +0200] INFO: Starting Chef Run
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Registering nouvel_kingsofmambo_net for an openid
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via GET to /registrations/nouvel_kingsofmambo_net
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Authenticating nouvel_kingsofmambo_net via openid
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via POST to /openid/consumer/start
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via GET to /openid/server?openid.assoc_handle=%7BHMAC-SHA1%7D%7B4a2f838d%7D%7Bhe1kgg%3D%3D%7D&openid.claimed_id=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.identity=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete
[Thu, 18 Jun 2009 00:33:29 +0200] ERROR: Net::HTTPFatalError
[Thu, 18 Jun 2009 00:33:29 +0200] FATAL: 502 "Bad Gateway"
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/1.8/net/http.rb:2097:in `error!'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:232:in `run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:230:in `run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:95:in `post_rest'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:223:in `authenticate'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:74:in `run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:163:in `run_application'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in `loop'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in `run_application'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application.rb:53:in `run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/bin/chef-client:26
/usr/bin/chef-client:19:in `load'
/usr/bin/chef-client:19
[Thu, 18 Jun 2009 00:33:29 +0200] FATAL: Sleeping for 1817 seconds before trying again

Doesn't seem to be anything indicative on the server side, so I'd suggest a network or temporal issue such as DNS. :\

On the server side:

 ~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
 ~ Params: {"format"=>nil, "action"=>"show", "id"=>"nouvel_kingsofmambo_net", "controller"=>"chef_server_slice/openid_register"}
 ~ {:before_filters_time=>0.0, :action_time=>0.0, :dispatch_time=>0.004, :after_filters_time=>0.0}
 ~

 ~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
 ~ Params: {"action"=>"node_page", "id"=>"nouvel_kingsofmambo_net", "controller"=>"chef_server_slice/openid_server"}
 ~ {:before_filters_time=>0.0, :action_time=>0.004, :dispatch_time=>0.004, :after_filters_time=>0.0}
 ~

 ~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
 ~ Params: {"submit"=>"Verify", "action"=>"start", "controller"=>"chef_server_slice/openid_consumer", "openid_identifier"=>"https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net"}
 ~ WARNING: making https request to https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net without verifying server certificate; no CA path was specified.
 ~ Generated checkid_setup request to https://chef.kingsofmambo.net/openid/server with assocication {HMAC-SHA1}{4a2f838d}{he1kgg==}
 ~ Generated checkid_setup request to https://chef.kingsofmambo.net/openid/server with assocication {HMAC-SHA1}{4a2f838d}{he1kgg==}
 ~ Redirecting to: https://chef.kingsofmambo.net/openid/server?openid.assoc_handle=%7BHMAC-SHA1%7D%7B4a2f838d%7D%7Bhe1kgg%3D%3D%7D&openid.claimed_id=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.identity=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete (302)
 ~ {:before_filters_time=>0.0, :action_time=>0.024001, :dispatch_time=>0.024001, :after_filters_time=>0.0}
 ~

 ~ Started request handling: Thu Jun 18 00:33:29 +0200 2009
 ~ Params: {"openid.claimed_id"=>"https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net", "openid.mode"=>"checkid_setup", "openid.return_to"=>"https://chef.kingsofmambo.net/openid/consumer/complete", "openid.ns"=>"http://specs.openid.net/auth/2.0", "action"=>"index", "controller"=>"chef_server_slice/openid_server", "openid.identity"=>"https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net", "openid.assoc_handle"=>"{HMAC-SHA1}{4a2f838d}{he1kgg==}", "openid.realm"=>"https://chef.kingsofmambo.net/openid/consumer"}
 ~ {:before_filters_time=>0.0, :action_time=>0.004, :dispatch_time=>0.004, :after_filters_time=>0.0}
 ~

Thanks beforehand!

--
Albert Llop
Tech Department
albert@kingsofmambo.com

2009/6/17 Arjuna Christensen <aj@opscode.com>

On 17/06/2009, at 7:41 PM, Albert Llop wrote:

I've been having some trouble with my chef-clients, I've my chef-server working properly, and my chef-clients can work just fine (with 0.7). I can login, and execute chef-client as root and it will work wonderfully.

But after that it just doesn't work. In the log you see this every 30mins:

[Wed, 17 Jun 2009 09:15:09 +0200] INFO: Starting Chef Run
[Wed, 17 Jun 2009 09:15:09 +0200] ERROR: Net::HTTPFatalError
[Wed, 17 Jun 2009 09:15:09 +0200] FATAL: 502 "Bad Gateway"
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/1.8/net/http.rb:2097:in `error!'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:232:in `run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:230:in `run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:95:in `post_rest'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:223:in `authenticate'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:74:in `run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:163:in `run_application'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in `loop'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in `run_application'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application.rb:53:in `run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/bin/chef-client:26
/usr/bin/chef-client:19:in `load'
/usr/bin/chef-client:19
[Wed, 17 Jun 2009 09:15:09 +0200] FATAL: Sleeping for 1815 seconds before trying again

Could you possibly set your log_level to debug on both the client and the server, and drop another email back with the increased log verbosity?

<snip>

-- 
AJ Christensen, Software Engineer

Opscode, Inc.
E: aj@opscode.com

-- 
AJ Christensen, Software Engineer

Opscode, Inc.
E: aj@opscode.com