[chef] Re: chef-client doesn't work "automatically"

[Albert Llop <mrsimo@gmail.com>]

Hi there again,

all this has been solved updating everything (server and clients) to the 0.7.4. Thanks a lot for the fixes!
--
Albert Llop
Tech Department
albert@kingsofmambo.com

2009/6/19 Arjuna Christensen <aj@opscode.com>

On 19/06/2009, at 10:05 PM, Albert Llop <mrsimo@gmail.com> wrote:

Thanks a lot, know that bit of info about the big cookies might be useful!

And yeah, it's been working under nginx for some time, obtained all the info I needed for that from this repo on github http://github.com/fujin/cookbooks/tree/nginx-chef-server

Oddly enough, that's one of my branches on my fork - I'm fujin!

Glad it's (kind of) working for you, was really just a hack of Josh's apache stuff.

:)
--
Albert Llop
Tech Department
albert@kingsofmambo.com

2009/6/19 Arjuna Christensen <aj@opscode.com>

On 19/06/2009, at 9:17 PM, Albert Llop wrote:

After some sleep I decided that the "502 "Bad Gateway"" error looks too much like nginx, and I checked its erorr log (why didn't i do it before?

Whenever one of these nodes that try to contact the server and is in "zombie mode" like i like to call them (because chef-client is working... but not properly), nginx gives an error:

2009/06/19 11:11:38 [error] 27044#0: *40 upstream sent too big header while reading response header from upstream, client: 67.23.26.105, server: chef.kingsofmambo.net, request: "GET /openid/server?openid.assoc_handle=%7BHMAC-SHA1%7D%7B4a2f838d%7D%7Bhe1kgg%3D%3D%7D&openid.claimed_id=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fpotter_kingsofmambo_net&openid.identity=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fpotter_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete HTTP/1.1", upstream: "unix:/tmp/passenger.27019/master/helper_server.sock:", host: "chef.kingsofmambo.net"

Very interesting. I wasn't aware you were running underneath another container (although this should definitely work correctly.)

We're currently experiencing some issues with chef clients receiving cookies bigger than 4kB from the Chef Server, which may be related to this. I wish Nginx would tell us what the header was that was too big.

I'm opening a ticket, and I'll scrub down the openid stuff while running under Nginx Passenger and see if I can get it to fold.

Thanks for the additional information!

2009/6/18 Arjuna Christensen <aj@opscode.com>

Hey!

On 18/06/2009, at 7:53 PM, Albert Llop wrote:

Hi again,
I set the log level to debug on one of the nodes, and restarted the client. After the restart it all worked fine for 14hours (until midnight more or less), and then stopped working.

The client log looks very similar with the debug option:

[Thu, 18 Jun 2009 00:33:28 +0200] INFO: Starting Chef Run
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Registering nouvel_kingsofmambo_net for an openid
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via GET to /registrations/nouvel_kingsofmambo_net
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Authenticating nouvel_kingsofmambo_net via openid
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via POST to /openid/consumer/start
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via GET to /openid/server?openid.assoc_handle=%7BHMAC-SHA1%7D%7B4a2f838d%7D%7Bhe1kgg%3D%3D%7D&openid.claimed_id=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.identity=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete
[Thu, 18 Jun 2009 00:33:29 +0200] ERROR: Net::HTTPFatalError
[Thu, 18 Jun 2009 00:33:29 +0200] FATAL: 502 "Bad Gateway"
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/1.8/net/http.rb:2097:in `error!'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:232:in `run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:230:in `run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:95:in `post_rest'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:223:in `authenticate'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:74:in `run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:163:in `run_application'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in `loop'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in `run_application'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application.rb:53:in `run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/bin/chef-client:26
/usr/bin/chef-client:19:in `load'
/usr/bin/chef-client:19
[Thu, 18 Jun 2009 00:33:29 +0200] FATAL: Sleeping for 1817 seconds before trying again

Doesn't seem to be anything indicative on the server side, so I'd suggest a network or temporal issue such as DNS. :\

On the server side:

 ~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
 ~ Params: {"format"=>nil, "action"=>"show", "id"=>"nouvel_kingsofmambo_net", "controller"=>"chef_server_slice/openid_register"}
 ~ {:before_filters_time=>0.0, :action_time=>0.0, :dispatch_time=>0.004, :after_filters_time=>0.0}
 ~

 ~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
 ~ Params: {"action"=>"node_page", "id"=>"nouvel_kingsofmambo_net", "controller"=>"chef_server_slice/openid_server"}
 ~ {:before_filters_time=>0.0, :action_time=>0.004, :dispatch_time=>0.004, :after_filters_time=>0.0}
 ~

 ~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
 ~ Params: {"submit"=>"Verify", "action"=>"start", "controller"=>"chef_server_slice/openid_consumer", "openid_identifier"=>"https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net"}
 ~ WARNING: making https request to https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net without verifying server certificate; no CA path was specified.
 ~ Generated checkid_setup request to https://chef.kingsofmambo.net/openid/server with assocication {HMAC-SHA1}{4a2f838d}{he1kgg==}
 ~ Generated checkid_setup request to https://chef.kingsofmambo.net/openid/server with assocication {HMAC-SHA1}{4a2f838d}{he1kgg==}
 ~ Redirecting to: https://chef.kingsofmambo.net/openid/server?openid.assoc_handle=%7BHMAC-SHA1%7D%7B4a2f838d%7D%7Bhe1kgg%3D%3D%7D&openid.claimed_id=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.identity=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete (302)
 ~ {:before_filters_time=>0.0, :action_time=>0.024001, :dispatch_time=>0.024001, :after_filters_time=>0.0}
 ~

 ~ Started request handling: Thu Jun 18 00:33:29 +0200 2009
 ~ Params: {"openid.claimed_id"=>"https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net", "openid.mode"=>"checkid_setup", "openid.return_to"=>"https://chef.kingsofmambo.net/openid/consumer/complete", "openid.ns"=>"http://specs.openid.net/auth/2.0", "action"=>"index", "controller"=>"chef_server_slice/openid_server", "openid.identity"=>"https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net", "openid.assoc_handle"=>"{HMAC-SHA1}{4a2f838d}{he1kgg==}", "openid.realm"=>"https://chef.kingsofmambo.net/openid/consumer"}
 ~ {:before_filters_time=>0.0, :action_time=>0.004, :dispatch_time=>0.004, :after_filters_time=>0.0}
 ~

Thanks beforehand!

--
Albert Llop
Tech Department
albert@kingsofmambo.com

2009/6/17 Arjuna Christensen <aj@opscode.com>

On 17/06/2009, at 7:41 PM, Albert Llop wrote:

I've been having some trouble with my chef-clients, I've my chef-server working properly, and my chef-clients can work just fine (with 0.7). I can login, and execute chef-client as root and it will work wonderfully.

But after that it just doesn't work. In the log you see this every 30mins:

[Wed, 17 Jun 2009 09:15:09 +0200] INFO: Starting Chef Run
[Wed, 17 Jun 2009 09:15:09 +0200] ERROR: Net::HTTPFatalError
[Wed, 17 Jun 2009 09:15:09 +0200] FATAL: 502 "Bad Gateway"
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/1.8/net/http.rb:2097:in `error!'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:232:in `run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:230:in `run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:95:in `post_rest'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:223:in `authenticate'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:74:in `run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:163:in `run_application'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in `loop'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in `run_application'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application.rb:53:in `run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/bin/chef-client:26
/usr/bin/chef-client:19:in `load'
/usr/bin/chef-client:19
[Wed, 17 Jun 2009 09:15:09 +0200] FATAL: Sleeping for 1815 seconds before trying again

Could you possibly set your log_level to debug on both the client and the server, and drop another email back with the increased log verbosity?

<snip>

-- 
AJ Christensen, Software Engineer

Opscode, Inc.
E: aj@opscode.com

-- 
AJ Christensen, Software Engineer

Opscode, Inc.
E: aj@opscode.com

-- 
AJ Christensen, Software Engineer

Opscode, Inc.
E: aj@opscode.com