[chef] Re: chef-server stops responding ever other request

[Mike Adolphs < >]

Hi,

time is a bit on the short side right now, but I had similar problems with 
chef-server at first and did a bit of debugging at that time. I've seen that 
each time an amqp message gets sent/delivered/processed (not sure when), the 
whole environment stopped working.
The reason behind all this: My rabbitmq-server package was way too new for 
chef-server. I've used the latest stable packages from rabbitmq's own apt 
repository. After downgrading to 2.6.1 everything went back to normal.

Within the rabbitmq changelog (http://www.rabbitmq.com/changelog.html) 
there're two interesting points which could be related:
* order preserved of messages re-queued for a consumer
* clients accept a new "amqp" URI scheme for connections


HTH,
Mike Adolphs

On Jun 5, 2012, at 2:54 PM, Stefan Wendler wrote:

> Hi List,
> 
> maybe you can help me wit this? I've set up a "chef playground" with a 
> workstation, a server (+UI) and two clients. While everything works so far 
> the 
> chef-server is making huge problems. I currently have to restart it after 
> every REST-call because it won't react to anything afterwards (until 
> restarted)
> 
> Example: I add a client (success) afterwards I try to add a cookbook (no 
> response). Even a simple client connect is enough to stop it
> 
> according to ps and netstat the merb process is still running and listening 
> to 
> port 4000. And I can telnet to the port.
> 
> chef-server just hangs and is not responding to anything. Not even the gui. 
> After restart it's working again for one request.
> 
> chef-server is started like this:
> 
>    /usr/local/bin/chef-server -d -e ${ENVIRONMENT}
> 
> where ENVIRONMENT is "production"
> 
> The "server.rb" is as follows
> ---------------- server.rb -------------------
> log_level          :debug
> log_location       "/var/log/chef/chef_server.log"
> ssl_verify_mode    :verify_none
> chef_server_url    "http://prodtest-slave:4000";
> 
> signing_ca_path    "/var/chef/ca"
> couchdb_database   'chef'
> 
> cookbook_path      [ "/var/chef/cookbooks", "/var/chef/site-cookbooks" ]
> 
> file_cache_path    "/var/chef/cache"
> node_path          "/var/chef/nodes"
> openid_store_path  "/var/chef/openid/store"
> openid_cstore_path "/var/chef/openid/cstore"
> search_index_path  "/var/chef/search_index"
> role_path          "/var/chef/roles"
> 
> validation_client_name "chef-validator"
> validation_key         "/etc/chef/validation.pem"
> client_key             "/etc/chef/client.pem"
> web_ui_key             "/etc/chef/webui.pem"
> web_ui_client_name     "chef-webui"
> 
> web_ui_admin_user_name "admin"
> web_ui_admin_default_password "<somepassword>"
> 
> supportdir = "/srv/chef/support"
> solr_jetty_path File.join(supportdir, "solr", "jetty")
> solr_data_path  File.join(supportdir, "solr", "data")
> solr_home_path  File.join(supportdir, "solr", "home")
> solr_heap_size  "256M"
> 
> user 'chef'
> group 'chef'
> 
> umask 0022
> 
> node_name 'chef_server'
> 
> Mixlib::Log::Formatter.show_time = true 
> ---------------- end of server.rb -------------------
> 
> When I start it undaemonized it logs just this when it hangs:
> 
> ------------------ logging ------------------
> merb : chef-server (api) : worker (port 4000) ~ Started request handling: 
> 2012-06-05 14:40:10 +0200
> merb : chef-server (api) : worker (port 4000) ~ Routed to: 
> {"controller"=>"nodes", "action"=>"show", "id"=>"prodtest-master", 
> "format"=>nil}
> merb : chef-server (api) : worker (port 4000) ~ Params: 
> {"controller"=>"nodes", "action"=>"show", "id"=>"prodtest-master", 
> "format"=>nil}
> merb : chef-server (api) : worker (port 4000) ~ 
> {:before_filters_time=>0.024005771, :after_filters_time=>1.1935e-05, 
> :action_time=>0.043984018, :dispatch_time=>0.044698021}
> merb : chef-server (api) : worker (port 4000) ~ 
> 
>      ConstrainVector: borrow[1] 0,     delta 0
> ConstrainV    ector: current[0] -1,     best_val -1
>      ConstrainVector: borrow[2] 0,     delta [0..10]
> ConstrainV    ector: current[1] [-10..0],       best_val -10
>      ConstrainVector: borrow[3] 0,     delta 0
> ConstrainV    ector: current[2] 0,      best_val 0
>      ConstrainVector: borrow[4] 0,     delta [0..1]
> ConstrainV    ector: current[3] [0..1], best_val 0
>      ConstrainVector: borrow[5] 0,     delta [0..1]
> ConstrainV    ector: current[4] [0..1], best_val 0
> dep_selector solve: SOLVED 2 packages, 1 constraints, Time: 1.764ms Stats: 
> 1 
> steps, 10970 bytes, 8 props, 3 nodes, 2 depth 
> merb : chef-server (api) : worker (port 4000) ~ Started request handling: 
> 2012-06-05 14:40:10 +0200
> merb : chef-server (api) : worker (port 4000) ~ Routed to: 
> {"run_list"=>["getting-started"], "controller"=>"environments", 
> "action"=>"cookbook_versions_for_run_list", "environment_id"=>"_default"}
> merb : chef-server (api) : worker (port 4000) ~ Params: 
> {"run_list"=>["getting-started"], "controller"=>"environments", 
> "action"=>"cookbook_versions_for_run_list", "environment_id"=>"_default"}
> merb : chef-server (api) : worker (port 4000) ~ 
> {:before_filters_time=>0.016346418, :after_filters_time=>9.857e-06, 
> :action_time=>0.077166217, :dispatch_time=>0.077611369}
> merb : chef-server (api) : worker (port 4000) ~ 
> 
> ~ Qrack::Queue#publish will be removed in Bunny 0.8. Use direct_exchange = 
> bunny.exchange(''); direct_exchange.publish('message', key: queue.name) if 
> you 
> want to publish directly to one given queue. For more informations see 
> https://github.com/ruby-amqp/bunny/issues/15 and for more theoretical ;
> explanation check http://bit.ly/nOF1CK
> 
> ------------------ end of logging ------------------
> 
> Version of chef is 0.10.10"
> 
> Thx alot for your help,
> Stefan



------
Mike Adolphs
Stitenstrasse 24
23554 Luebeck
- Germany -

Mail. 

 
Web. http://fooforge.com/
Github. https://github.com/fooforge/
XING. http://www.xing.com/profile/Mike_Adolphs