General discussion about Chef

[chef] Re: Re: Error after crash of Chef server


Chronological Thread 
  • From: Nick Willever < >
  • To:
  • Subject: [chef] Re: Re: Error after crash of Chef server
  • Date: Mon, 11 Jun 2012 16:05:23 -0700
Whenever I saw an EOF message couchdb was working well, but it couldn't find the cookbook_index checksum files.  Check those files, make sure you still have them.. and/or send us a cookbook upload with a -VV switch

I just ran into the EOF message while moving to another chef server and not grabing the cookbook_index files, one fix for it would be to remove all couchdb cookbook documents and re-upload all the cookbooks.

Thanks,




On Mon, Jun 11, 2012 at 1:36 PM, Daniel DeLeo < " target="_blank"> > wrote:


On Monday, June 11, 2012 at 2:07 AM, Jesai Langenbach wrote:

>
> Hi,
>
> We have a Problem since a server crash with chef.
>
> The chef server is working but I cant rebuild the index /upload new data bags/modify data bags/upload cookbooks . I can't figure out where the problem is.
>
> The Error Massage on the chef server (server.log) is:
> merb : chef-server (api) : worker (port 4001) ~ end of file reached - (EOFError)
> /usr/lib/ruby/1.8/net/protocol.rb:135:in `sysread'
> /usr/lib/ruby/1.8/net/protocol.rb:135:in `rbuf_fill'
> /usr/lib/ruby/1.8/timeout.rb:67:in `timeout'
> /usr/lib/ruby/1.8/timeout.rb:101:in `timeout'
> /usr/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'
> /usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
> /usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'
> /usr/lib/ruby/1.8/net/http.rb:2028:in `read_status_line'
> /usr/lib/ruby/1.8/net/http.rb:2017:in `read_new'
> /usr/lib/ruby/1.8/net/http.rb:1051:in `request'
> /usr/lib/ruby/1.8/net/http.rb:1037:in `request'
> /usr/lib/ruby/1.8/net/http.rb:543:in `start'
> /usr/lib/ruby/1.8/net/http.rb:1035:in `request'
> /usr/lib/ruby/vendor_ruby/chef/rest/rest_request.rb:99:in `call'
> /usr/lib/ruby/vendor_ruby/chef/rest/rest_request.rb:114:in `hide_net_http_bug'
> /usr/lib/ruby/vendor_ruby/chef/rest/rest_request.rb:98:in `call'
> /usr/lib/ruby/vendor_ruby/chef/rest.rb:241:in `api_request'
> /usr/lib/ruby/vendor_ruby/chef/rest.rb:328:in `retriable_rest_request'
> /usr/lib/ruby/vendor_ruby/chef/rest.rb:240:in `api_request'
> /usr/lib/ruby/vendor_ruby/chef/rest.rb:144:in `put_rest'
> /usr/lib/ruby/vendor_ruby/chef/couchdb.rb:114:in `store'
> /usr/lib/ruby/vendor_ruby/chef/node.rb:619:in `cdb_save'
> /usr/share/chef-server-api/app/controllers/nodes.rb:69:in `update'
> /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in `send'
> /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in `_call_action'
> /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:289:in `_dispatch'
> /usr/lib/ruby/1.8/merb-core/controller/merb_controller.rb:252:in `_dispatch'
> /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:102:in `dispatch_action'
> /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:74:in `handle'
> /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:36:in `handle'
> /usr/lib/ruby/1.8/merb-core/rack/application.rb:17:in `call'
> /usr/lib/ruby/1.8/merb-core/rack/middleware/static.rb:28:in `call'
> /usr/lib/ruby/1.8/rack/content_length.rb:13:in `call'
> /usr/lib/ruby/1.8/thin/connection.rb:76:in `pre_process'
> /usr/lib/ruby/1.8/thin/connection.rb:74:in `catch'
> /usr/lib/ruby/1.8/thin/connection.rb:74:in `pre_process'
> /usr/lib/ruby/1.8/thin/connection.rb:57:in `process'
> /usr/lib/ruby/1.8/thin/connection.rb:42:in `receive_data'
> /usr/lib/ruby/1.8/eventmachine.rb:257:in `run_machine'
> /usr/lib/ruby/1.8/eventmachine.rb:257:in `run'
> /usr/lib/ruby/1.8/thin/backends/base.rb:57:in `start'
> /usr/lib/ruby/1.8/thin/server.rb:156:in `start'
> /usr/lib/ruby/1.8/merb-core/rack/adapter/thin.rb:30:in `start_server'
> /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:298:in `start_at_port'
> /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:128:in `start'
> /usr/lib/ruby/1.8/merb-core/server.rb:174:in `bootup'
> /usr/lib/ruby/1.8/merb-core/server.rb:42:in `start'
> /usr/lib/ruby/1.8/merb-core.rb:170:in `start'
> /usr/sbin/chef-server:86
>
>
>
> I increased the log level but still do not find anything.
>
> The Full Log is:
> chef-server -p 4001 -e production -a thin
> Loading init file from /usr/share/chef-server-api/config/init.rb
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/_all_dbs
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/id_map
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/nodes
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/roles
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/data_bags
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/clients
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/users
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/cookbooks
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/sandboxes
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/checksums
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/environments
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/id_map/_view/name_to_id
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/id_map/_view/name_to_id
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/id_map/_view/name_to_id
> [Mon, 11 Jun 2012 09:49:54 +0200] INFO: Loading roles
> /usr/lib/ruby/1.8/mixlib/authentication/http_authentication_request.rb:25: warning: already initialized constant MANDATORY_HEADERS
> /usr/lib/ruby/1.8/mixlib/authentication/signatureverification.rb:28: warning: already initialized constant SignatureResponse
> ~ Could not find resource model User
> ~ Could not find resource model Node
> ~ Could not find resource model Role
> ~ Could not find resource model Environment
> ~ Could not find resource model Datum
> merb : chef-server (api) : worker (port 4001) ~ Starting Thin at port 4001
> merb : chef-server (api) : worker (port 4001) ~ Using Thin adapter on host 0.0.0.0 and port 4001.
> merb : chef-server (api) : worker (port 4001) ~ Successfully bound to port 4001
> [Mon, 11 Jun 2012 09:50:29 +0200] INFO: Authenticating client j_langenbach
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/id_map/_view/name_to_id
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Authenticating Client:
> #<Chef::ApiClient:0x7f573c1d54a8 @couchdb=#<Chef::CouchDB:0x7f573c1db498 @db="chef", @rest=#<Chef::REST:0x7f573c1db470 @sign_request=true, @default_headers={}, @url=""http://localhost:5984", @sign_on_redirect=true, @redirect_limit=10, @redirects_followed=0, @auth_credentials=#<Chef::REST::AuthCredentials:0x7f573c1db218 @key_file=nil, @client_name=nil>, @cookies={}>>, @public_key="-----BEGIN RSA PUBLIC KEY-----\nMIIBCgKCAQEAskYoWJdzJF+jzGJuyJC9P4CRKUTIBdEthsRmitw2A+I5eoIVhJub\nkdWYGLjbzvy/WuVE8rfsAmJlolvRkcaXWYn+CxFopS3SUZ0EXouliTF2GLtE0jJO\n3L3OoEc+l2jIPfpcfvoLr01W/djk2Ap+4TWOiKbtphUf4ExEzr1eJ5pT2XMoPRT7\ncYkXSqpec3v1Zbusg+xzeEVK66JtBwfwKSW0NX0L96PdRbGrvrSe+RYEJyFzhyYJ\ny3NwkeCk97OyO8KfYoE8H5wkQ11jifhP38hvwnyHDFybW0XKyhEZC2CSP2GEwrik\nMCVzKX1MuHhFMXTncYlVwiWZ3X2VrDbicwIDAQAB\n-----END RSA PUBLIC KEY-----\n", @couchdb_id="6c2d5608-4c7e-4163-a57a-a2b3f795af57", @admin=true, @couchdb_rev="1-788bfa953a0d2c07f0dddaf7fbb12214", @private_key=nil, @name="j_langenbach", @index_id="6c
 2d5608-4
c7e-4163-a57a-a2b3f795af57">
>
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Initializing header auth : #<Merb::Request:0x7f573c1dfbd8 @route=/search/reindex, @route_params={:action=">"reindex"," :controller=>"search"}, @multipart_params={}, @env={"HTTP_HOST"=>"chef.kdz-ws.intra.net:4001 (http://chef.kdz-ws.intra.net:4001)", "HTTP_ACCEPT"=>"application/json", "SERVER_NAME"=>"chef.kdz-ws.intra.net (http://chef.kdz-ws.intra.net)", "rack.url_scheme"=>"http", "CONTENT_LENGTH"=>"2", "HTTP_USER_AGENT"=>"Chef Knife/0.10.8 (ruby-1.9.3-p125; ohai-0.6.12; x86_64-linux; +http://opscode.com)", "REQUEST_PATH"=>"//search/reindex", "SERVER_PROTOCOL"=>"HTTP/1.1", "CONTENT_TYPE"=>"application/json", "HTTP_X_CHEF_VERSION"=>"0.10.8", "HTTP_X_OPS_CONTENT_HASH"=>"vyGp6PvFo4RvsFtPoIWeCReyIC8=", "HTTP_X_OPS_TIMESTAMP"=>"2012-06-11T07:50:30Z", "rack.errors"=>#<IO:0x7f57437c0aa0>, "async.callback"=>#<Method: Thin::Connection#post_process>, "REMOTE_ADDR"=>"172.16.81.180", "PATH_INFO"=>"//search/reindex", "rack.run_once"=>false, "rack.ve
 rsion"=>
[1, 0], "SERVER_SOFTWARE"=>"thin 1.2.4 codename Flaming Astroboy", "SCRIPT_NAME"=>"", "HTTP_X_OPS_AUTHORIZATION_1"=>"JnNPHNDCMxAxNRebYy5RFuTEvaT5a9W+xHNi1Za+EqH93lPm8TSNgpQr0pfk", "HTTP_X_OPS_AUTHORIZATION_2"=>"1mjiotOfWfBbz6dAclzGrUc55gv/J35vfQydABF2YYXc2oUIlvEDmHguPm6Q", "HTTP_VERSION"=>"HTTP/1.1", "rack.multithread"=>false, "HTTP_X_OPS_AUTHORIZATION_3"=>"z0bdvrgFSnXJ9JJLqjfBdIgYfZX5VkCqSP9SlUdKnWQ3GEPV3n11z3PItltH", "REQUEST_URI"=>"//search/reindex", "rack.multiprocess"=>false, "SERVER_PORT"=>"4001", "HTTP_X_OPS_AUTHORIZATION_4"=>"6WqNv5GD7dZICyMIpTyU/NnivtpHyb50bgl+ZvnKMppDUignv2QuTQUn46oY", "async.close"=>#<EventMachine::DefaultDeferrable:0x7f573c1dfed0>, "HTTP_X_OPS_AUTHORIZATION_5"=>"cMy06RqkqBHUHD94T3jYJCZRM4kSUig617Z9YisMMSpm42Jf7nVokn2Mp7dJ", "HTTP_X_OPS_USERID"=>"j_langenbach", "REQUEST_METHOD"=>"POST", "HTTP_X_OPS_AUTHORIZATION_6"=>"DQRs4OXc/OUhTr4y/iZ+HJQyhCaXNAu26c9a7wwkXQ==", "QUERY_STRING"=>"", "GATEWAY_INTERFACE"=>"CGI/1.2", "HTTP_CONNECTION"=>"close", "HTTP_
 X_OPS_SI
GN"=>"version=1.0", "rack.input"=>#<StringIO:0x7f573c1e75b8>}, @json_params={}, @raw_post="{}", @params={"action"=>"reindex", "controller"=>"search"}, @body_params=nil, @method=:post, @query_params={}, @xml_params=nil, @body_and_query_params={}, @body=#<StringIO:0x7f573c1e75b8>>
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Parsed signing description: {:version=>"1.0"}
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Digesting body: '{}'
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Reconstituted (user-supplied) request signature: JnNPHNDCMxAxNRebYy5RFuTEvaT5a9W+xHNi1Za+EqH93lPm8TSNgpQr0pfk
> 1mjiotOfWfBbz6dAclzGrUc55gv/J35vfQydABF2YYXc2oUIlvEDmHguPm6Q
> z0bdvrgFSnXJ9JJLqjfBdIgYfZX5VkCqSP9SlUdKnWQ3GEPV3n11z3PItltH
> 6WqNv5GD7dZICyMIpTyU/NnivtpHyb50bgl+ZvnKMppDUignv2QuTQUn46oY
> cMy06RqkqBHUHD94T3jYJCZRM4kSUig617Z9YisMMSpm42Jf7nVokn2Mp7dJ
> DQRs4OXc/OUhTr4y/iZ+HJQyhCaXNAu26c9a7wwkXQ==
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Verifying request signature:
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Expected Block is: 'Method:POST
> Hashed Path:ZKpVFjmsFMEHqnjeztcib+NRIgo=
> X-Ops-Content-Hash:vyGp6PvFo4RvsFtPoIWeCReyIC8=
> X-Ops-Timestamp:2012-06-11T07:50:30Z
> X-Ops-UserId:j_langenbach'
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Decrypted block is: 'Method:POST
> Hashed Path:ZKpVFjmsFMEHqnjeztcib+NRIgo=
> X-Ops-Content-Hash:vyGp6PvFo4RvsFtPoIWeCReyIC8=
> X-Ops-Timestamp:2012-06-11T07:50:30Z
> X-Ops-UserId:j_langenbach'
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Signatures match? : 'true'
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Request time difference: 0.155941, within 900 seconds? : true
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Expected content hash is: 'vyGp6PvFo4RvsFtPoIWeCReyIC8='
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Request Content Hash is: 'vyGp6PvFo4RvsFtPoIWeCReyIC8='
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Hashes match?: true
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: POSTing document to SOLR:
> <?xml version="1.0" encoding="UTF-8"?>
> <delete><query>X_CHEF_database_CHEF_X:chef</query></delete>
>
> [Mon, 11 Jun 2012 09:50:30 +0200] DEBUG: POSTing document to SOLR:
> <?xml version="1.0" encoding="UTF-8"?>
> <commit/>
>
> [Mon, 11 Jun 2012 09:50:30 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/clients/_view/all
> [Mon, 11 Jun 2012 09:50:30 +0200] FATAL: Chef encountered an error while attempting to load Chef::ApiClient objects back into the index
> merb : chef-server (api) : worker (port 4001) ~ Started request handling: Mon Jun 11 09:50:29 +0200 2012
> merb : chef-server (api) : worker (port 4001) ~ Routed to: {"action"=>"reindex", "controller"=>"search"}
> merb : chef-server (api) : worker (port 4001) ~ Params: {"action"=>"reindex", "controller"=>"search"}
> merb : chef-server (api) : worker (port 4001) ~ end of file reached - (EOFError)
> /usr/lib/ruby/1.8/net/protocol.rb:135:in `sysread'chef-server -p 4001 -e production -a thin
> Loading init file from /usr/share/chef-server-api/config/init.rb
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/_all_dbs
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/id_map
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/nodes
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/roles
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/data_bags
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/clients
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/users
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/cookbooks
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/sandboxes
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/checksums
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/environments
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/id_map/_view/name_to_id
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/id_map/_view/name_to_id
> [Mon, 11 Jun 2012 09:49:54 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/id_map/_view/name_to_id
> [Mon, 11 Jun 2012 09:49:54 +0200] INFO: Loading roles
> /usr/lib/ruby/1.8/mixlib/authentication/http_authentication_request.rb:25: warning: already initialized constant MANDATORY_HEADERS
> /usr/lib/ruby/1.8/mixlib/authentication/signatureverification.rb:28: warning: already initialized constant SignatureResponse
> ~ Could not find resource model User
> ~ Could not find resource model Node
> ~ Could not find resource model Role
> ~ Could not find resource model Environment
> ~ Could not find resource model Datum
> merb : chef-server (api) : worker (port 4001) ~ Starting Thin at port 4001
> merb : chef-server (api) : worker (port 4001) ~ Using Thin adapter on host 0.0.0.0 and port 4001.
> merb : chef-server (api) : worker (port 4001) ~ Successfully bound to port 4001
> [Mon, 11 Jun 2012 09:50:29 +0200] INFO: Authenticating client j_langenbach
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/id_map/_view/name_to_id
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Authenticating Client:
> #<Chef::ApiClient:0x7f573c1d54a8 @couchdb=#<Chef::CouchDB:0x7f573c1db498 @db="chef", @rest=#<Chef::REST:0x7f573c1db470 @sign_request=true, @default_headers={}, @url=""http://localhost:5984", @sign_on_redirect=true, @redirect_limit=10, @redirects_followed=0, @auth_credentials=#<Chef::REST::AuthCredentials:0x7f573c1db218 @key_file=nil, @client_name=nil>, @cookies={}>>, @public_key="-----BEGIN RSA PUBLIC KEY-----\nMIIBCgKCAQEAskYoWJdzJF+jzGJuyJC9P4CRKUTIBdEthsRmitw2A+I5eoIVhJub\nkdWYGLjbzvy/WuVE8rfsAmJlolvRkcaXWYn+CxFopS3SUZ0EXouliTF2GLtE0jJO\n3L3OoEc+l2jIPfpcfvoLr01W/djk2Ap+4TWOiKbtphUf4ExEzr1eJ5pT2XMoPRT7\ncYkXSqpec3v1Zbusg+xzeEVK66JtBwfwKSW0NX0L96PdRbGrvrSe+RYEJyFzhyYJ\ny3NwkeCk97OyO8KfYoE8H5wkQ11jifhP38hvwnyHDFybW0XKyhEZC2CSP2GEwrik\nMCVzKX1MuHhFMXTncYlVwiWZ3X2VrDbicwIDAQAB\n-----END RSA PUBLIC KEY-----\n", @couchdb_id="6c2d5608-4c7e-4163-a57a-a2b3f795af57", @admin=true, @couchdb_rev="1-788bfa953a0d2c07f0dddaf7fbb12214", @private_key=nil, @name="j_langenbach", @index_id="6c
 2d5608-4
c7e-4163-a57a-a2b3f795af57">
>
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Initializing header auth : #<Merb::Request:0x7f573c1dfbd8 @route=/search/reindex, @route_params={:action=">"reindex"," :controller=>"search"}, @multipart_params={}, @env={"HTTP_HOST"=>"chef.kdz-ws.intra.net:4001 (http://chef.kdz-ws.intra.net:4001)", "HTTP_ACCEPT"=>"application/json", "SERVER_NAME"=>"chef.kdz-ws.intra.net (http://chef.kdz-ws.intra.net)", "rack.url_scheme"=>"http", "CONTENT_LENGTH"=>"2", "HTTP_USER_AGENT"=>"Chef Knife/0.10.8 (ruby-1.9.3-p125; ohai-0.6.12; x86_64-linux; +http://opscode.com)", "REQUEST_PATH"=>"//search/reindex", "SERVER_PROTOCOL"=>"HTTP/1.1", "CONTENT_TYPE"=>"application/json", "HTTP_X_CHEF_VERSION"=>"0.10.8", "HTTP_X_OPS_CONTENT_HASH"=>"vyGp6PvFo4RvsFtPoIWeCReyIC8=", "HTTP_X_OPS_TIMESTAMP"=>"2012-06-11T07:50:30Z", "rack.errors"=>#<IO:0x7f57437c0aa0>, "async.callback"=>#<Method: Thin::Connection#post_process>, "REMOTE_ADDR"=>"172.16.81.180", "PATH_INFO"=>"//search/reindex", "rack.run_once"=>false, "rack.ve
 rsion"=>
[1, 0], "SERVER_SOFTWARE"=>"thin 1.2.4 codename Flaming Astroboy", "SCRIPT_NAME"=>"", "HTTP_X_OPS_AUTHORIZATION_1"=>"JnNPHNDCMxAxNRebYy5RFuTEvaT5a9W+xHNi1Za+EqH93lPm8TSNgpQr0pfk", "HTTP_X_OPS_AUTHORIZATION_2"=>"1mjiotOfWfBbz6dAclzGrUc55gv/J35vfQydABF2YYXc2oUIlvEDmHguPm6Q", "HTTP_VERSION"=>"HTTP/1.1", "rack.multithread"=>false, "HTTP_X_OPS_AUTHORIZATION_3"=>"z0bdvrgFSnXJ9JJLqjfBdIgYfZX5VkCqSP9SlUdKnWQ3GEPV3n11z3PItltH", "REQUEST_URI"=>"//search/reindex", "rack.multiprocess"=>false, "SERVER_PORT"=>"4001", "HTTP_X_OPS_AUTHORIZATION_4"=>"6WqNv5GD7dZICyMIpTyU/NnivtpHyb50bgl+ZvnKMppDUignv2QuTQUn46oY", "async.close"=>#<EventMachine::DefaultDeferrable:0x7f573c1dfed0>, "HTTP_X_OPS_AUTHORIZATION_5"=>"cMy06RqkqBHUHD94T3jYJCZRM4kSUig617Z9YisMMSpm42Jf7nVokn2Mp7dJ", "HTTP_X_OPS_USERID"=>"j_langenbach", "REQUEST_METHOD"=>"POST", "HTTP_X_OPS_AUTHORIZATION_6"=>"DQRs4OXc/OUhTr4y/iZ+HJQyhCaXNAu26c9a7wwkXQ==", "QUERY_STRING"=>"", "GATEWAY_INTERFACE"=>"CGI/1.2", "HTTP_CONNECTION"=>"close", "HTTP_
 X_OPS_SI
GN"=>"version=1.0", "rack.input"=>#<StringIO:0x7f573c1e75b8>}, @json_params={}, @raw_post="{}", @params={"action"=>"reindex", "controller"=>"search"}, @body_params=nil, @method=:post, @query_params={}, @xml_params=nil, @body_and_query_params={}, @body=#<StringIO:0x7f573c1e75b8>>
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Parsed signing description: {:version=>"1.0"}
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Digesting body: '{}'
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Reconstituted (user-supplied) request signature: JnNPHNDCMxAxNRebYy5RFuTEvaT5a9W+xHNi1Za+EqH93lPm8TSNgpQr0pfk
> 1mjiotOfWfBbz6dAclzGrUc55gv/J35vfQydABF2YYXc2oUIlvEDmHguPm6Q
> z0bdvrgFSnXJ9JJLqjfBdIgYfZX5VkCqSP9SlUdKnWQ3GEPV3n11z3PItltH
> 6WqNv5GD7dZICyMIpTyU/NnivtpHyb50bgl+ZvnKMppDUignv2QuTQUn46oY
> cMy06RqkqBHUHD94T3jYJCZRM4kSUig617Z9YisMMSpm42Jf7nVokn2Mp7dJ
> DQRs4OXc/OUhTr4y/iZ+HJQyhCaXNAu26c9a7wwkXQ==
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Verifying request signature:
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Expected Block is: 'Method:POST
> Hashed Path:ZKpVFjmsFMEHqnjeztcib+NRIgo=
> X-Ops-Content-Hash:vyGp6PvFo4RvsFtPoIWeCReyIC8=
> X-Ops-Timestamp:2012-06-11T07:50:30Z
> X-Ops-UserId:j_langenbach'
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Decrypted block is: 'Method:POST
> Hashed Path:ZKpVFjmsFMEHqnjeztcib+NRIgo=
> X-Ops-Content-Hash:vyGp6PvFo4RvsFtPoIWeCReyIC8=
> X-Ops-Timestamp:2012-06-11T07:50:30Z
> X-Ops-UserId:j_langenbach'
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Signatures match? : 'true'
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Request time difference: 0.155941, within 900 seconds? : true
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Expected content hash is: 'vyGp6PvFo4RvsFtPoIWeCReyIC8='
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Request Content Hash is: 'vyGp6PvFo4RvsFtPoIWeCReyIC8='
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: Hashes match?: true
> [Mon, 11 Jun 2012 09:50:29 +0200] DEBUG: POSTing document to SOLR:
> <?xml version="1.0" encoding="UTF-8"?>
> <delete><query>X_CHEF_database_CHEF_X:chef</query></delete>
>
> [Mon, 11 Jun 2012 09:50:30 +0200] DEBUG: POSTing document to SOLR:
> <?xml version="1.0" encoding="UTF-8"?>
> <commit/>
>
> [Mon, 11 Jun 2012 09:50:30 +0200] DEBUG: Sending HTTP Request via GET to localhost:5984/chef/_design/clients/_view/all
> [Mon, 11 Jun 2012 09:50:30 +0200] FATAL: Chef encountered an error while attempting to load Chef::ApiClient objects back into the index
> merb : chef-server (api) : worker (port 4001) ~ Started request handling: Mon Jun 11 09:50:29 +0200 2012
> merb : chef-server (api) : worker (port 4001) ~ Routed to: {"action"=>"reindex", "controller"=>"search"}
> merb : chef-server (api) : worker (port 4001) ~ Params: {"action"=>"reindex", "controller"=>"search"}
> merb : chef-server (api) : worker (port 4001) ~ end of file reached - (EOFError)
> /usr/lib/ruby/1.8/net/protocol.rb:135:in `sysread'
> /usr/lib/ruby/1.8/net/protocol.rb:135:in `rbuf_fill'
> /usr/lib/ruby/1.8/timeout.rb:67:in `timeout'
> /usr/lib/ruby/1.8/timeout.rb:101:in `timeout'
> /usr/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'
> /usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
> /usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'
> /usr/lib/ruby/1.8/net/http.rb:2028:in `read_status_line'
> /usr/lib/ruby/1.8/net/http.rb:2017:in `read_new'
> /usr/lib/ruby/1.8/net/http.rb:1051:in `request'
> /usr/lib/ruby/1.8/net/http.rb:1037:in `request'
> /usr/lib/ruby/1.8/net/http.rb:543:in `start'
> /usr/lib/ruby/1.8/net/http.rb:1035:in `request'
> /usr/lib/ruby/vendor_ruby/chef/rest/rest_request.rb:99:in `call'
> /usr/lib/ruby/vendor_ruby/chef/rest/rest_request.rb:114:in `hide_net_http_bug'
> /usr/lib/ruby/vendor_ruby/chef/rest/rest_request.rb:98:in `call'
> /usr/lib/ruby/vendor_ruby/chef/rest.rb:241:in `api_request'
> /usr/lib/ruby/vendor_ruby/chef/rest.rb:328:in `retriable_rest_request'
> /usr/lib/ruby/vendor_ruby/chef/rest.rb:240:in `api_request'
> /usr/lib/ruby/vendor_ruby/chef/rest.rb:128:in `get_rest'
> /usr/lib/ruby/vendor_ruby/chef/couchdb.rb:181:in `list'
> /usr/lib/ruby/vendor_ruby/chef/api_client.rb:178:in `cdb_list'
> /usr/lib/ruby/vendor_ruby/chef/solr_query.rb:167:in `reindex_all'
> /usr/lib/ruby/vendor_ruby/chef/solr_query.rb:156:in `rebuild_index'
> /usr/lib/ruby/vendor_ruby/chef/solr_query.rb:155:in `each'
> /usr/lib/ruby/vendor_ruby/chef/solr_query.rb:155:in `rebuild_index'
> /usr/share/chef-server-api/app/controllers/search.rb:55:in `reindex'
> /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in `send'
> /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in `_call_action'
> /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:289:in `_dispatch'
> /usr/lib/ruby/1.8/merb-core/controller/merb_controller.rb:252:in `_dispatch'
> /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:102:in `dispatch_action'
> /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:74:in `handle'
> /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:36:in `handle'
> /usr/lib/ruby/1.8/merb-core/rack/application.rb:17:in `call'
> /usr/lib/ruby/1.8/merb-core/rack/middleware/static.rb:28:in `call'
> /usr/lib/ruby/1.8/rack/content_length.rb:13:in `call'
> /usr/lib/ruby/1.8/thin/connection.rb:76:in `pre_process'
> /usr/lib/ruby/1.8/thin/connection.rb:74:in `catch'
> /usr/lib/ruby/1.8/thin/connection.rb:74:in `pre_process'
> /usr/lib/ruby/1.8/thin/connection.rb:57:in `process'
> /usr/lib/ruby/1.8/thin/connection.rb:42:in `receive_data'
> /usr/lib/ruby/1.8/eventmachine.rb:257:in `run_machine'
> /usr/lib/ruby/1.8/eventmachine.rb:257:in `run'
> /usr/lib/ruby/1.8/thin/backends/base.rb:57:in `start'
> /usr/lib/ruby/1.8/thin/server.rb:156:in `start'
> /usr/lib/ruby/1.8/merb-core/rack/adapter/thin.rb:30:in `start_server'
> /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:298:in `start_at_port'
> /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:128:in `start'
> /usr/lib/ruby/1.8/merb-core/server.rb:174:in `bootup'
> /usr/lib/ruby/1.8/merb-core/server.rb:42:in `start'
> /usr/lib/ruby/1.8/merb-core.rb:170:in `start'
> /usr/sbin/chef-server:86
> merb : chef-server (api) : worker (port 4001) ~ Params: {"action"=>"reindex", "controller"=>"search"}
> merb : chef-server (api) : worker (port 4001) ~
> /usr/lib/ruby/1.8/net/protocol.rb:135:in `rbuf_fill'
> /usr/lib/ruby/1.8/timeout.rb:67:in `timeout'
> /usr/lib/ruby/1.8/timeout.rb:101:in `timeout'
> /usr/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'
> /usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
> /usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'
> /usr/lib/ruby/1.8/net/http.rb:2028:in `read_status_line'
> /usr/lib/ruby/1.8/net/http.rb:2017:in `read_new'
> /usr/lib/ruby/1.8/net/http.rb:1051:in `request'
> /usr/lib/ruby/1.8/net/http.rb:1037:in `request'
> /usr/lib/ruby/1.8/net/http.rb:543:in `start'
> /usr/lib/ruby/1.8/net/http.rb:1035:in `request'
> /usr/lib/ruby/vendor_ruby/chef/rest/rest_request.rb:99:in `call'
> /usr/lib/ruby/vendor_ruby/chef/rest/rest_request.rb:114:in `hide_net_http_bug'
> /usr/lib/ruby/vendor_ruby/chef/rest/rest_request.rb:98:in `call'
> /usr/lib/ruby/vendor_ruby/chef/rest.rb:241:in `api_request'
> /usr/lib/ruby/vendor_ruby/chef/rest.rb:328:in `retriable_rest_request'
> /usr/lib/ruby/vendor_ruby/chef/rest.rb:240:in `api_request'
> /usr/lib/ruby/vendor_ruby/chef/rest.rb:128:in `get_rest'
> /usr/lib/ruby/vendor_ruby/chef/couchdb.rb:181:in `list'
> /usr/lib/ruby/vendor_ruby/chef/api_client.rb:178:in `cdb_list'
> /usr/lib/ruby/vendor_ruby/chef/solr_query.rb:167:in `reindex_all'
> /usr/lib/ruby/vendor_ruby/chef/solr_query.rb:156:in `rebuild_index'
> /usr/lib/ruby/vendor_ruby/chef/solr_query.rb:155:in `each'
> /usr/lib/ruby/vendor_ruby/chef/solr_query.rb:155:in `rebuild_index'
> /usr/share/chef-server-api/app/controllers/search.rb:55:in `reindex'
> /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in `send'
> /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in `_call_action'
> /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:289:in `_dispatch'
> /usr/lib/ruby/1.8/merb-core/controller/merb_controller.rb:252:in `_dispatch'
> /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:102:in `dispatch_action'
> /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:74:in `handle'
> /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:36:in `handle'
> /usr/lib/ruby/1.8/merb-core/rack/application.rb:17:in `call'
> /usr/lib/ruby/1.8/merb-core/rack/middleware/static.rb:28:in `call'
> /usr/lib/ruby/1.8/rack/content_length.rb:13:in `call'
> /usr/lib/ruby/1.8/thin/connection.rb:76:in `pre_process'
> /usr/lib/ruby/1.8/thin/connection.rb:74:in `catch'
> /usr/lib/ruby/1.8/thin/connection.rb:74:in `pre_process'
> /usr/lib/ruby/1.8/thin/connection.rb:57:in `process'
> /usr/lib/ruby/1.8/thin/connection.rb:42:in `receive_data'
> /usr/lib/ruby/1.8/eventmachine.rb:257:in `run_machine'
> /usr/lib/ruby/1.8/eventmachine.rb:257:in `run'
> /usr/lib/ruby/1.8/thin/backends/base.rb:57:in `start'
> /usr/lib/ruby/1.8/thin/server.rb:156:in `start'
> /usr/lib/ruby/1.8/merb-core/rack/adapter/thin.rb:30:in `start_server'
> /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:298:in `start_at_port'
> /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:128:in `start'
> /usr/lib/ruby/1.8/merb-core/server.rb:174:in `bootup'
> /usr/lib/ruby/1.8/merb-core/server.rb:42:in `start'
> /usr/lib/ruby/1.8/merb-core.rb:170:in `start'
> /usr/sbin/chef-server:86
> merb : chef-server (api) : worker (port 4001) ~ Params: {"action"=>"reindex", "controller"=>"search"}
> merb : chef-server (api) : worker (port 4001) ~
>
>
> I also started chef-server with strace to see if it is a real file or the tcp stream, I couldn't find any relevant file which chef tries to access before the error occurs.
>
> Any Ideas?
>
> Thank you
Both of those stack traces point to issues fetching documents from CouchDB. What version of CouchDB are you using, and is there any relevant info in the logs?

--
Dan DeLeo





Archive powered by MHonArc 2.6.16.

§