Hi,
We have a Problem since a server crash with chef.
The chef server is working but I cant rebuild the index /upload new data bags/modify data bags/upload cookbooks . I can't figure out where the problem is.
The Error Massage on the chef server (server.log) is:
merb : chef-server (api) : worker (port 4001) ~ end of file reached - (EOFError)
/usr/lib/ruby/1.8/net/protocol.rb:135:in `sysread'
/usr/lib/ruby/1.8/net/protocol.rb:135:in `rbuf_fill'
/usr/lib/ruby/1.8/timeout.rb:67:in `timeout'
/usr/lib/ruby/1.8/timeout.rb:101:in `timeout'
/usr/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'
/usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
/usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'
/usr/lib/ruby/1.8/net/http.rb:2028:in `read_status_line'
/usr/lib/ruby/1.8/net/http.rb:2017:in `read_new'
/usr/lib/ruby/1.8/net/http.rb:1051:in `request'
/usr/lib/ruby/1.8/net/http.rb:1037:in `request'
/usr/lib/ruby/1.8/net/http.rb:543:in `start'
/usr/lib/ruby/1.8/net/http.rb:1035:in `request'
/usr/lib/ruby/vendor_ruby/chef/rest/rest_request.rb:99:in `call'
/usr/lib/ruby/vendor_ruby/chef/rest/rest_request.rb:114:in `hide_net_http_bug'
/usr/lib/ruby/vendor_ruby/chef/rest/rest_request.rb:98:in `call'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:241:in `api_request'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:328:in `retriable_rest_request'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:240:in `api_request'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:144:in `put_rest'
/usr/lib/ruby/vendor_ruby/chef/couchdb.rb:114:in `store'
/usr/lib/ruby/vendor_ruby/chef/node.rb:619:in `cdb_save'
/usr/share/chef-server-api/app/controllers/nodes.rb:69:in `update'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in `send'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in `_call_action'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:289:in `_dispatch'
/usr/lib/ruby/1.8/merb-core/controller/merb_controller.rb:252:in `_dispatch'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:102:in `dispatch_action'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:74:in `handle'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:36:in `handle'
/usr/lib/ruby/1.8/merb-core/rack/application.rb:17:in `call'
/usr/lib/ruby/1.8/merb-core/rack/middleware/static.rb:28:in `call'
/usr/lib/ruby/1.8/rack/content_length.rb:13:in `call'
/usr/lib/ruby/1.8/thin/connection.rb:76:in `pre_process'
/usr/lib/ruby/1.8/thin/connection.rb:74:in `catch'
/usr/lib/ruby/1.8/thin/connection.rb:74:in `pre_process'
/usr/lib/ruby/1.8/thin/connection.rb:57:in `process'
/usr/lib/ruby/1.8/thin/connection.rb:42:in `receive_data'
/usr/lib/ruby/1.8/eventmachine.rb:257:in `run_machine'
/usr/lib/ruby/1.8/eventmachine.rb:257:in `run'
/usr/lib/ruby/1.8/thin/backends/base.rb:57:in `start'
/usr/lib/ruby/1.8/thin/server.rb:156:in `start'
/usr/lib/ruby/1.8/merb-core/rack/adapter/thin.rb:30:in `start_server'
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:298:in `start_at_port'
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:128:in `start'
/usr/lib/ruby/1.8/merb-core/server.rb:174:in `bootup'
/usr/lib/ruby/1.8/merb-core/server.rb:42:in `start'
/usr/lib/ruby/1.8/merb-core.rb:170:in `start'
/usr/sbin/chef-server:86
I increased the log level but still do not find anything.
The Full Log is:
chef-server -p 4001 -e production -a thin
Loading init file from /usr/share/chef-server-api/config/init.rb
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/_all_dbs
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/id_map
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/nodes
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/roles
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/data_bags
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/clients
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/users
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/cookbooks
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/sandboxes
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/checksums
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/environments
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/id_map/_view/name_to_id
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/id_map/_view/name_to_id
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/id_map/_view/name_to_id
[Mon, 11 Jun 2012 09:49:54 +0200] INFO: Loading roles
/usr/lib/ruby/1.8/mixlib/authentication/http_authentication_request.rb:25: warning: already initialized constant MANDATORY_HEADERS
/usr/lib/ruby/1.8/mixlib/authentication/signatureverification.rb:28: warning: already initialized constant SignatureResponse
~ Could not find resource model User
~ Could not find resource model Node
~ Could not find resource model Role
~ Could not find resource model Environment
~ Could not find resource model Datum
merb : chef-server (api) : worker (port 4001) ~ Starting Thin at port 4001
merb : chef-server (api) : worker (port 4001) ~ Using Thin adapter on host 0.0.0.0 and port 4001.
merb : chef-server (api) : worker (port 4001) ~ Successfully bound to port 4001
[Mon, 11 Jun 2012 09:50:29 +0200] INFO: Authenticating client j_langenbach
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/id_map/_view/name_to_id
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Authenticating Client:
#<Chef::ApiClient:0x7f573c1d54a8 @couchdb=#<Chef::CouchDB:0x7f573c1db498 @db="chef", @rest=#<Chef::REST:0x7f573c1db470 @sign_request=true, @default_headers={}, @url=""
http://localhost:5984", @sign_on_redirect=true, @redirect_limit=10, @redirects_followed=0, @auth_credentials=#<Chef::REST::AuthCredentials:0x7f573c1db218 @key_file=nil, @client_name=nil>, @cookies={}>>, @public_key="-----BEGIN RSA PUBLIC KEY-----\nMIIBCgKCAQEAskYoWJdzJF+jzGJuyJC9P4CRKUTIBdEthsRmitw2A+I5eoIVhJub\nkdWYGLjbzvy/WuVE8rfsAmJlolvRkcaXWYn+CxFopS3SUZ0EXouliTF2GLtE0jJO\n3L3OoEc+l2jIPfpcfvoLr01W/djk2Ap+4TWOiKbtphUf4ExEzr1eJ5pT2XMoPRT7\ncYkXSqpec3v1Zbusg+xzeEVK66JtBwfwKSW0NX0L96PdRbGrvrSe+RYEJyFzhyYJ\ny3NwkeCk97OyO8KfYoE8H5wkQ11jifhP38hvwnyHDFybW0XKyhEZC2CSP2GEwrik\nMCVzKX1MuHhFMXTncYlVwiWZ3X2VrDbicwIDAQAB\n-----END RSA PUBLIC KEY-----\n", @couchdb_id="6c2d5608-4c7e-4163-a57a-a2b3f795af57", @admin=true, @couchdb_rev="1-788bfa953a0d2c07f0dddaf7fbb12214", @private_key=nil, @name="j_langenbach", @index_id="6c2d5608-4c7e-4163-a57a-a2b3f795af57">
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Initializing header auth : #<Merb::Request:0x7f573c1dfbd8 @route=/search/reindex, @route_params={:action=">"reindex"," :controller=>"search"}, @multipart_params={}, @env={"HTTP_HOST"=>"
chef.kdz-ws.intra.net:4001", "HTTP_ACCEPT"=>"application/json", "SERVER_NAME"=>"
chef.kdz-ws.intra.net", "rack.url_scheme"=>"http", "CONTENT_LENGTH"=>"2", "HTTP_USER_AGENT"=>"Chef Knife/0.10.8 (ruby-1.9.3-p125; ohai-0.6.12; x86_64-linux; +
http://opscode.com)", "REQUEST_PATH"=>"//search/reindex", "SERVER_PROTOCOL"=>"HTTP/1.1", "CONTENT_TYPE"=>"application/json", "HTTP_X_CHEF_VERSION"=>"0.10.8", "HTTP_X_OPS_CONTENT_HASH"=>"vyGp6PvFo4RvsFtPoIWeCReyIC8=", "HTTP_X_OPS_TIMESTAMP"=>"2012-06-11T07:50:30Z", "rack.errors"=>#<IO:0x7f57437c0aa0>, "async.callback"=>#<Method: Thin::Connection#post_process>, "REMOTE_ADDR"=>"172.16.81.180", "PATH_INFO"=>"//search/reindex", "rack.run_once"=>false, "rack.version"=>[1, 0], "SERVER_SOFTWARE"=>"thin 1.2.4 codename Flaming Astroboy", "SCRIPT_NAME"=>"", "HTTP_X_OPS_AUTHORIZATION_1"=>"JnNPHNDCMxAxNRebYy5RFuTEvaT5a9W+xHNi1Za+EqH93lPm8TSNgpQr0pfk", "HTTP_X_OPS_AUTHORIZATION_2"=>"1mjiotOfWfBbz6dAclzGrUc55gv/J35vfQydABF2YYXc2oUIlvEDmHguPm6Q", "HTTP_VERSION"=>"HTTP/1.1", "rack.multithread"=>false, "HTTP_X_OPS_AUTHORIZATION_3"=>"z0bdvrgFSnXJ9JJLqjfBdIgYfZX5VkCqSP9SlUdKnWQ3GEPV3n11z3PItltH", "REQUEST_URI"=>"//search/reindex", "rack.multiprocess"=>false, "SERVER_PORT"=>"4001", "HTTP_X_OPS_AUTHORIZATION_4"=>"6WqNv5GD7dZICyMIpTyU/NnivtpHyb50bgl+ZvnKMppDUignv2QuTQUn46oY", "async.close"=>#<EventMachine::DefaultDeferrable:0x7f573c1dfed0>, "HTTP_X_OPS_AUTHORIZATION_5"=>"cMy06RqkqBHUHD94T3jYJCZRM4kSUig617Z9YisMMSpm42Jf7nVokn2Mp7dJ", "HTTP_X_OPS_USERID"=>"j_langenbach", "REQUEST_METHOD"=>"POST", "HTTP_X_OPS_AUTHORIZATION_6"=>"DQRs4OXc/OUhTr4y/iZ+HJQyhCaXNAu26c9a7wwkXQ==", "QUERY_STRING"=>"", "GATEWAY_INTERFACE"=>"CGI/1.2", "HTTP_CONNECTION"=>"close", "HTTP_X_OPS_SIGN"=>"version=1.0", "rack.input"=>#<StringIO:0x7f573c1e75b8>}, @json_params={}, @raw_post="{}", @params={"action"=>"reindex", "controller"=>"search"}, @body_params=nil, @method=:post, @query_params={}, @xml_params=nil, @body_and_query_params={}, @body=#<StringIO:0x7f573c1e75b8>>
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Parsed signing description: {:version=>"1.0"}
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Digesting body: '{}'
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Reconstituted (user-supplied) request signature: JnNPHNDCMxAxNRebYy5RFuTEvaT5a9W+xHNi1Za+EqH93lPm8TSNgpQr0pfk
1mjiotOfWfBbz6dAclzGrUc55gv/J35vfQydABF2YYXc2oUIlvEDmHguPm6Q
z0bdvrgFSnXJ9JJLqjfBdIgYfZX5VkCqSP9SlUdKnWQ3GEPV3n11z3PItltH
6WqNv5GD7dZICyMIpTyU/NnivtpHyb50bgl+ZvnKMppDUignv2QuTQUn46oY
cMy06RqkqBHUHD94T3jYJCZRM4kSUig617Z9YisMMSpm42Jf7nVokn2Mp7dJ
DQRs4OXc/OUhTr4y/iZ+HJQyhCaXNAu26c9a7wwkXQ==
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Verifying request signature:
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Expected Block is: 'Method:POST
Hashed Path:ZKpVFjmsFMEHqnjeztcib+NRIgo=
X-Ops-Content-Hash:vyGp6PvFo4RvsFtPoIWeCReyIC8=
X-Ops-Timestamp:2012-06-11T07:50:30Z
X-Ops-UserId:j_langenbach'
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Decrypted block is: 'Method:POST
Hashed Path:ZKpVFjmsFMEHqnjeztcib+NRIgo=
X-Ops-Content-Hash:vyGp6PvFo4RvsFtPoIWeCReyIC8=
X-Ops-Timestamp:2012-06-11T07:50:30Z
X-Ops-UserId:j_langenbach'
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Signatures match? : 'true'
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Request time difference: 0.155941, within 900 seconds? : true
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Expected content hash is: 'vyGp6PvFo4RvsFtPoIWeCReyIC8='
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Request Content Hash is: 'vyGp6PvFo4RvsFtPoIWeCReyIC8='
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Hashes match?: true
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: POSTing document to SOLR:
<?xml version="1.0" encoding="UTF-8"?>
<delete><query>X_CHEF_database_CHEF_X:chef</query></delete>
[Mon, 11 Jun 2012 09:50:30 +0200] DEBUG: POSTing document to SOLR:
<?xml version="1.0" encoding="UTF-8"?>
<commit/>
[Mon, 11 Jun 2012 09:50:30 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/clients/_view/all
[Mon, 11 Jun 2012 09:50:30 +0200] FATAL: Chef encountered an error while attempting to load Chef::ApiClient objects back into the index
merb : chef-server (api) : worker (port 4001) ~ Started request handling: Mon Jun 11 09:50:29 +0200 2012
merb : chef-server (api) : worker (port 4001) ~ Routed to: {"action"=>"reindex", "controller"=>"search"}
merb : chef-server (api) : worker (port 4001) ~ Params: {"action"=>"reindex", "controller"=>"search"}
merb : chef-server (api) : worker (port 4001) ~ end of file reached - (EOFError)
/usr/lib/ruby/1.8/net/protocol.rb:135:in `sysread'chef-server -p 4001 -e production -a thin
Loading init file from /usr/share/chef-server-api/config/init.rb
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/_all_dbs
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/id_map
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/nodes
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/roles
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/data_bags
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/clients
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/users
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/cookbooks
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/sandboxes
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/checksums
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/environments
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/id_map/_view/name_to_id
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/id_map/_view/name_to_id
[Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/id_map/_view/name_to_id
[Mon, 11 Jun 2012 09:49:54 +0200] INFO: Loading roles
/usr/lib/ruby/1.8/mixlib/authentication/http_authentication_request.rb:25: warning: already initialized constant MANDATORY_HEADERS
/usr/lib/ruby/1.8/mixlib/authentication/signatureverification.rb:28: warning: already initialized constant SignatureResponse
~ Could not find resource model User
~ Could not find resource model Node
~ Could not find resource model Role
~ Could not find resource model Environment
~ Could not find resource model Datum
merb : chef-server (api) : worker (port 4001) ~ Starting Thin at port 4001
merb : chef-server (api) : worker (port 4001) ~ Using Thin adapter on host 0.0.0.0 and port 4001.
merb : chef-server (api) : worker (port 4001) ~ Successfully bound to port 4001
[Mon, 11 Jun 2012 09:50:29 +0200] INFO: Authenticating client j_langenbach
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/id_map/_view/name_to_id
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Authenticating Client:
#<Chef::ApiClient:0x7f573c1d54a8 @couchdb=#<Chef::CouchDB:0x7f573c1db498 @db="chef", @rest=#<Chef::REST:0x7f573c1db470 @sign_request=true, @default_headers={}, @url=""
http://localhost:5984", @sign_on_redirect=true, @redirect_limit=10, @redirects_followed=0, @auth_credentials=#<Chef::REST::AuthCredentials:0x7f573c1db218 @key_file=nil, @client_name=nil>, @cookies={}>>, @public_key="-----BEGIN RSA PUBLIC KEY-----\nMIIBCgKCAQEAskYoWJdzJF+jzGJuyJC9P4CRKUTIBdEthsRmitw2A+I5eoIVhJub\nkdWYGLjbzvy/WuVE8rfsAmJlolvRkcaXWYn+CxFopS3SUZ0EXouliTF2GLtE0jJO\n3L3OoEc+l2jIPfpcfvoLr01W/djk2Ap+4TWOiKbtphUf4ExEzr1eJ5pT2XMoPRT7\ncYkXSqpec3v1Zbusg+xzeEVK66JtBwfwKSW0NX0L96PdRbGrvrSe+RYEJyFzhyYJ\ny3NwkeCk97OyO8KfYoE8H5wkQ11jifhP38hvwnyHDFybW0XKyhEZC2CSP2GEwrik\nMCVzKX1MuHhFMXTncYlVwiWZ3X2VrDbicwIDAQAB\n-----END RSA PUBLIC KEY-----\n", @couchdb_id="6c2d5608-4c7e-4163-a57a-a2b3f795af57", @admin=true, @couchdb_rev="1-788bfa953a0d2c07f0dddaf7fbb12214", @private_key=nil, @name="j_langenbach", @index_id="6c2d5608-4c7e-4163-a57a-a2b3f795af57">
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Initializing header auth : #<Merb::Request:0x7f573c1dfbd8 @route=/search/reindex, @route_params={:action=">"reindex"," :controller=>"search"}, @multipart_params={}, @env={"HTTP_HOST"=>"
chef.kdz-ws.intra.net:4001", "HTTP_ACCEPT"=>"application/json", "SERVER_NAME"=>"
chef.kdz-ws.intra.net", "rack.url_scheme"=>"http", "CONTENT_LENGTH"=>"2", "HTTP_USER_AGENT"=>"Chef Knife/0.10.8 (ruby-1.9.3-p125; ohai-0.6.12; x86_64-linux; +
http://opscode.com)", "REQUEST_PATH"=>"//search/reindex", "SERVER_PROTOCOL"=>"HTTP/1.1", "CONTENT_TYPE"=>"application/json", "HTTP_X_CHEF_VERSION"=>"0.10.8", "HTTP_X_OPS_CONTENT_HASH"=>"vyGp6PvFo4RvsFtPoIWeCReyIC8=", "HTTP_X_OPS_TIMESTAMP"=>"2012-06-11T07:50:30Z", "rack.errors"=>#<IO:0x7f57437c0aa0>, "async.callback"=>#<Method: Thin::Connection#post_process>, "REMOTE_ADDR"=>"172.16.81.180", "PATH_INFO"=>"//search/reindex", "rack.run_once"=>false, "rack.version"=>[1, 0], "SERVER_SOFTWARE"=>"thin 1.2.4 codename Flaming Astroboy", "SCRIPT_NAME"=>"", "HTTP_X_OPS_AUTHORIZATION_1"=>"JnNPHNDCMxAxNRebYy5RFuTEvaT5a9W+xHNi1Za+EqH93lPm8TSNgpQr0pfk", "HTTP_X_OPS_AUTHORIZATION_2"=>"1mjiotOfWfBbz6dAclzGrUc55gv/J35vfQydABF2YYXc2oUIlvEDmHguPm6Q", "HTTP_VERSION"=>"HTTP/1.1", "rack.multithread"=>false, "HTTP_X_OPS_AUTHORIZATION_3"=>"z0bdvrgFSnXJ9JJLqjfBdIgYfZX5VkCqSP9SlUdKnWQ3GEPV3n11z3PItltH", "REQUEST_URI"=>"//search/reindex", "rack.multiprocess"=>false, "SERVER_PORT"=>"4001", "HTTP_X_OPS_AUTHORIZATION_4"=>"6WqNv5GD7dZICyMIpTyU/NnivtpHyb50bgl+ZvnKMppDUignv2QuTQUn46oY", "async.close"=>#<EventMachine::DefaultDeferrable:0x7f573c1dfed0>, "HTTP_X_OPS_AUTHORIZATION_5"=>"cMy06RqkqBHUHD94T3jYJCZRM4kSUig617Z9YisMMSpm42Jf7nVokn2Mp7dJ", "HTTP_X_OPS_USERID"=>"j_langenbach", "REQUEST_METHOD"=>"POST", "HTTP_X_OPS_AUTHORIZATION_6"=>"DQRs4OXc/OUhTr4y/iZ+HJQyhCaXNAu26c9a7wwkXQ==", "QUERY_STRING"=>"", "GATEWAY_INTERFACE"=>"CGI/1.2", "HTTP_CONNECTION"=>"close", "HTTP_X_OPS_SIGN"=>"version=1.0", "rack.input"=>#<StringIO:0x7f573c1e75b8>}, @json_params={}, @raw_post="{}", @params={"action"=>"reindex", "controller"=>"search"}, @body_params=nil, @method=:post, @query_params={}, @xml_params=nil, @body_and_query_params={}, @body=#<StringIO:0x7f573c1e75b8>>
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Parsed signing description: {:version=>"1.0"}
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Digesting body: '{}'
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Reconstituted (user-supplied) request signature: JnNPHNDCMxAxNRebYy5RFuTEvaT5a9W+xHNi1Za+EqH93lPm8TSNgpQr0pfk
1mjiotOfWfBbz6dAclzGrUc55gv/J35vfQydABF2YYXc2oUIlvEDmHguPm6Q
z0bdvrgFSnXJ9JJLqjfBdIgYfZX5VkCqSP9SlUdKnWQ3GEPV3n11z3PItltH
6WqNv5GD7dZICyMIpTyU/NnivtpHyb50bgl+ZvnKMppDUignv2QuTQUn46oY
cMy06RqkqBHUHD94T3jYJCZRM4kSUig617Z9YisMMSpm42Jf7nVokn2Mp7dJ
DQRs4OXc/OUhTr4y/iZ+HJQyhCaXNAu26c9a7wwkXQ==
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Verifying request signature:
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Expected Block is: 'Method:POST
Hashed Path:ZKpVFjmsFMEHqnjeztcib+NRIgo=
X-Ops-Content-Hash:vyGp6PvFo4RvsFtPoIWeCReyIC8=
X-Ops-Timestamp:2012-06-11T07:50:30Z
X-Ops-UserId:j_langenbach'
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Decrypted block is: 'Method:POST
Hashed Path:ZKpVFjmsFMEHqnjeztcib+NRIgo=
X-Ops-Content-Hash:vyGp6PvFo4RvsFtPoIWeCReyIC8=
X-Ops-Timestamp:2012-06-11T07:50:30Z
X-Ops-UserId:j_langenbach'
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Signatures match? : 'true'
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Request time difference: 0.155941, within 900 seconds? : true
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Expected content hash is: 'vyGp6PvFo4RvsFtPoIWeCReyIC8='
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Request Content Hash is: 'vyGp6PvFo4RvsFtPoIWeCReyIC8='
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Hashes match?: true
[Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: POSTing document to SOLR:
<?xml version="1.0" encoding="UTF-8"?>
<delete><query>X_CHEF_database_CHEF_X:chef</query></delete>
[Mon, 11 Jun 2012 09:50:30 +0200] DEBUG: POSTing document to SOLR:
<?xml version="1.0" encoding="UTF-8"?>
<commit/>
[Mon, 11 Jun 2012 09:50:30 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/clients/_view/all
[Mon, 11 Jun 2012 09:50:30 +0200] FATAL: Chef encountered an error while attempting to load Chef::ApiClient objects back into the index
merb : chef-server (api) : worker (port 4001) ~ Started request handling: Mon Jun 11 09:50:29 +0200 2012
merb : chef-server (api) : worker (port 4001) ~ Routed to: {"action"=>"reindex", "controller"=>"search"}
merb : chef-server (api) : worker (port 4001) ~ Params: {"action"=>"reindex", "controller"=>"search"}
merb : chef-server (api) : worker (port 4001) ~ end of file reached - (EOFError)
/usr/lib/ruby/1.8/net/protocol.rb:135:in `sysread'
/usr/lib/ruby/1.8/net/protocol.rb:135:in `rbuf_fill'
/usr/lib/ruby/1.8/timeout.rb:67:in `timeout'
/usr/lib/ruby/1.8/timeout.rb:101:in `timeout'
/usr/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'
/usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
/usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'
/usr/lib/ruby/1.8/net/http.rb:2028:in `read_status_line'
/usr/lib/ruby/1.8/net/http.rb:2017:in `read_new'
/usr/lib/ruby/1.8/net/http.rb:1051:in `request'
/usr/lib/ruby/1.8/net/http.rb:1037:in `request'
/usr/lib/ruby/1.8/net/http.rb:543:in `start'
/usr/lib/ruby/1.8/net/http.rb:1035:in `request'
/usr/lib/ruby/vendor_ruby/chef/rest/rest_request.rb:99:in `call'
/usr/lib/ruby/vendor_ruby/chef/rest/rest_request.rb:114:in `hide_net_http_bug'
/usr/lib/ruby/vendor_ruby/chef/rest/rest_request.rb:98:in `call'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:241:in `api_request'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:328:in `retriable_rest_request'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:240:in `api_request'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:128:in `get_rest'
/usr/lib/ruby/vendor_ruby/chef/couchdb.rb:181:in `list'
/usr/lib/ruby/vendor_ruby/chef/api_client.rb:178:in `cdb_list'
/usr/lib/ruby/vendor_ruby/chef/solr_query.rb:167:in `reindex_all'
/usr/lib/ruby/vendor_ruby/chef/solr_query.rb:156:in `rebuild_index'
/usr/lib/ruby/vendor_ruby/chef/solr_query.rb:155:in `each'
/usr/lib/ruby/vendor_ruby/chef/solr_query.rb:155:in `rebuild_index'
/usr/share/chef-server-api/app/controllers/search.rb:55:in `reindex'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in `send'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in `_call_action'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:289:in `_dispatch'
/usr/lib/ruby/1.8/merb-core/controller/merb_controller.rb:252:in `_dispatch'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:102:in `dispatch_action'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:74:in `handle'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:36:in `handle'
/usr/lib/ruby/1.8/merb-core/rack/application.rb:17:in `call'
/usr/lib/ruby/1.8/merb-core/rack/middleware/static.rb:28:in `call'
/usr/lib/ruby/1.8/rack/content_length.rb:13:in `call'
/usr/lib/ruby/1.8/thin/connection.rb:76:in `pre_process'
/usr/lib/ruby/1.8/thin/connection.rb:74:in `catch'
/usr/lib/ruby/1.8/thin/connection.rb:74:in `pre_process'
/usr/lib/ruby/1.8/thin/connection.rb:57:in `process'
/usr/lib/ruby/1.8/thin/connection.rb:42:in `receive_data'
/usr/lib/ruby/1.8/eventmachine.rb:257:in `run_machine'
/usr/lib/ruby/1.8/eventmachine.rb:257:in `run'
/usr/lib/ruby/1.8/thin/backends/base.rb:57:in `start'
/usr/lib/ruby/1.8/thin/server.rb:156:in `start'
/usr/lib/ruby/1.8/merb-core/rack/adapter/thin.rb:30:in `start_server'
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:298:in `start_at_port'
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:128:in `start'
/usr/lib/ruby/1.8/merb-core/server.rb:174:in `bootup'
/usr/lib/ruby/1.8/merb-core/server.rb:42:in `start'
/usr/lib/ruby/1.8/merb-core.rb:170:in `start'
/usr/sbin/chef-server:86
merb : chef-server (api) : worker (port 4001) ~ Params: {"action"=>"reindex", "controller"=>"search"}
merb : chef-server (api) : worker (port 4001) ~
/usr/lib/ruby/1.8/net/protocol.rb:135:in `rbuf_fill'
/usr/lib/ruby/1.8/timeout.rb:67:in `timeout'
/usr/lib/ruby/1.8/timeout.rb:101:in `timeout'
/usr/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'
/usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
/usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'
/usr/lib/ruby/1.8/net/http.rb:2028:in `read_status_line'
/usr/lib/ruby/1.8/net/http.rb:2017:in `read_new'
/usr/lib/ruby/1.8/net/http.rb:1051:in `request'
/usr/lib/ruby/1.8/net/http.rb:1037:in `request'
/usr/lib/ruby/1.8/net/http.rb:543:in `start'
/usr/lib/ruby/1.8/net/http.rb:1035:in `request'
/usr/lib/ruby/vendor_ruby/chef/rest/rest_request.rb:99:in `call'
/usr/lib/ruby/vendor_ruby/chef/rest/rest_request.rb:114:in `hide_net_http_bug'
/usr/lib/ruby/vendor_ruby/chef/rest/rest_request.rb:98:in `call'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:241:in `api_request'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:328:in `retriable_rest_request'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:240:in `api_request'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:128:in `get_rest'
/usr/lib/ruby/vendor_ruby/chef/couchdb.rb:181:in `list'
/usr/lib/ruby/vendor_ruby/chef/api_client.rb:178:in `cdb_list'
/usr/lib/ruby/vendor_ruby/chef/solr_query.rb:167:in `reindex_all'
/usr/lib/ruby/vendor_ruby/chef/solr_query.rb:156:in `rebuild_index'
/usr/lib/ruby/vendor_ruby/chef/solr_query.rb:155:in `each'
/usr/lib/ruby/vendor_ruby/chef/solr_query.rb:155:in `rebuild_index'
/usr/share/chef-server-api/app/controllers/search.rb:55:in `reindex'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in `send'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in `_call_action'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:289:in `_dispatch'
/usr/lib/ruby/1.8/merb-core/controller/merb_controller.rb:252:in `_dispatch'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:102:in `dispatch_action'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:74:in `handle'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:36:in `handle'
/usr/lib/ruby/1.8/merb-core/rack/application.rb:17:in `call'
/usr/lib/ruby/1.8/merb-core/rack/middleware/static.rb:28:in `call'
/usr/lib/ruby/1.8/rack/content_length.rb:13:in `call'
/usr/lib/ruby/1.8/thin/connection.rb:76:in `pre_process'
/usr/lib/ruby/1.8/thin/connection.rb:74:in `catch'
/usr/lib/ruby/1.8/thin/connection.rb:74:in `pre_process'
/usr/lib/ruby/1.8/thin/connection.rb:57:in `process'
/usr/lib/ruby/1.8/thin/connection.rb:42:in `receive_data'
/usr/lib/ruby/1.8/eventmachine.rb:257:in `run_machine'
/usr/lib/ruby/1.8/eventmachine.rb:257:in `run'
/usr/lib/ruby/1.8/thin/backends/base.rb:57:in `start'
/usr/lib/ruby/1.8/thin/server.rb:156:in `start'
/usr/lib/ruby/1.8/merb-core/rack/adapter/thin.rb:30:in `start_server'
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:298:in `start_at_port'
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:128:in `start'
/usr/lib/ruby/1.8/merb-core/server.rb:174:in `bootup'
/usr/lib/ruby/1.8/merb-core/server.rb:42:in `start'
/usr/lib/ruby/1.8/merb-core.rb:170:in `start'
/usr/sbin/chef-server:86
merb : chef-server (api) : worker (port 4001) ~ Params: {"action"=>"reindex", "controller"=>"search"}
merb : chef-server (api) : worker (port 4001) ~
I also started chef-server with strace to see if it is a real file or the tcp stream, I couldn't find any relevant file which chef tries to access before the error occurs.
Any Ideas?
Thank you