chef - [chef] Re: Chef Server Crashes on Cookbook Upload

Subscribers: 1946
Owners
Bryan McLellan
Joshua Timberman
Nathen Harvey
Seth Chisamore
Serdar Sutay

Subscribe
Unsubscribe
Info
Archive

Post

RSS
Shared documents

General discussion about Chef

[chef] Re: Chef Server Crashes on Cookbook Upload

From: Daniel DeLeo < >
To:
Subject: [chef] Re: Chef Server Crashes on Cookbook Upload
Date: Wed, 3 Oct 2012 11:19:47 -0700

Looks like the Tempfile code that chef-server uses to temporarily store uploaded files is failing to create tempfiles, and this is causing a secondary error when it attempts to clean up after itself.

Is /tmp full on your chef-server box? Are you able to create files and directories there by hand?

Daniel DeLeo

On Wednesday, October 3, 2012 at 10:38 AM, wrote:

Hi Chefs,

I had recently provisioned a new chef-server using 10.14.4 and was in the
process of moving all of our cookbooks onto the new chef-server when I get this
error client side,

ERROR: Failed to upload : 500 Internal Server Error
{"error":["No such file or directory -
/tmp/sandbox20121003-5631-1ofw8am.lock"]}
ERROR: Failed to upload : 500 Internal Server Error
{"error":["No such file or directory -
/tmp/sandbox20121003-5631-1li831d.lock"]}
ERROR: Failed to upload : 500 Internal Server Error
{"error":["No such file or directory - /tmp/sandbox20121003-5631-le9ona.lock"]}
ERROR: Failed to upload : Server broke connection
ERROR: Failed to upload : Server broke connection
/opt/chef/embedded/lib/ruby/1.9.1/net/protocol.rb:199:in `write': Broken pipe
(Errno::EPIPE)
from /opt/chef/embedded/lib/ruby/1.9.1/net/protocol.rb:199:in `write0'
from /opt/chef/embedded/lib/ruby/1.9.1/net/protocol.rb:173:in `block in
write'
from /opt/chef/embedded/lib/ruby/1.9.1/net/protocol.rb:190:in `writing'
from /opt/chef/embedded/lib/ruby/1.9.1/net/protocol.rb:172:in `write'
from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1937:in
`send_request_with_body'
from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1919:in `exec'
from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1317:in `block in
transport_request'
from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1316:in `catch'
from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1316:in
`transport_request'
from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1293:in `request'
from
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/net_http_ext.rb:51:in
`request'
from
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:176:in
`block in transmit'
from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:745:in `start'
from
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:172:in
`transmit'
from
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in
`execute'
from
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in
`execute'
from
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/resource.rb:76:in
`put'
from
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.14.4/lib/chef/cookbook_uploader.rb:142:in
`block in uploader_function_for'
from
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.14.4/lib/chef/cookbook_uploader.rb:25:in
`call'
from
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.14.4/lib/chef/cookbook_uploader.rb:25:in
`block (3 levels) in setup_worker_threads'
from
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.14.4/lib/chef/cookbook_uploader.rb:24:in
`loop'
from
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.14.4/lib/chef/cookbook_uploader.rb:24:in
`block (2 levels) in setup_worker_threads'

Chef-Server had died and needed to be restarted. Most of the cookbooks made it
but a few did not, re-running the upload process kills it again.

Looking at the server logs,

merb : chef-server (api) : worker (port 4000) ~ Started request handling:
2012-10-03 12:50:17 -0500
merb : chef-server (api) : worker (port 4000) ~ Routed to:
{"controller"=>"sandboxes", "action"=>"upload_checksum",
"sandbox_id"=>"e0f51964514641e2979656e67b1ff4d1",
"checksum"=>"97070fbb7de37914603425fff17e9bde"}
merb : chef-server (api) : worker (port 4000) ~ Params:
{"controller"=>"sandboxes", "action"=>"upload_checksum",
"sandbox_id"=>"e0f51964514641e2979656e67b1ff4d1",
"checksum"=>"97070fbb7de37914603425fff17e9bde"}
merb : chef-server (api) : worker (port 4000) ~ No such file or directory -
/tmp/sandbox20121003-5631-1ulbjbm.lock - (Errno::ENOENT)
/opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:346:in `rmdir'
/opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:346:in `rmdir'
/opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:338:in `ensure in locking'
/opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:338:in `locking'
/opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:144:in `block in initialize'
/opt/chef/embedded/lib/ruby/1.9.1/tmpdir.rb:133:in `create'
/opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:134:in `initialize'
/opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:316:in `new'
/opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:316:in `open'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-server-api-10.14.4/app/models/sandbox_file.rb:101:in
`commit_stringio_to'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-server-api-10.14.4/app/models/sandbox_file.rb:53:in
`commit_to'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-server-api-10.14.4/app/controllers/sandboxes.rb:96:in
`upload_checksum'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/abstract_controller.rb:321:in
`_call_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/abstract_controller.rb:289:in
`block in _dispatch'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/abstract_controller.rb:284:in
`catch'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/abstract_controller.rb:284:in
`_dispatch'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/merb_controller.rb:285:in
`_dispatch'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/merb_controller.rb:262:in
`_call'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/merb_controller.rb:252:in
`call'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/dispatch/dispatcher.rb:91:in
`dispatch_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/dispatch/dispatcher.rb:69:in
`handle'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/dispatch/dispatcher.rb:29:in
`handle'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/rack/application.rb:17:in
`call'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/rack/middleware/static.rb:28:in
`call'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rack-1.4.1/lib/rack/content_length.rb:14:in
`call'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:81:in
`block in pre_process'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:79:in
`catch'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:79:in
`pre_process'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:54:in
`process'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:39:in
`receive_data'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/eventmachine-1.0.0/lib/eventmachine.rb:187:in
`run_machine'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/eventmachine-1.0.0/lib/eventmachine.rb:187:in
`run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/backends/base.rb:63:in
`start'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/server.rb:159:in
`start'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/rack/adapter/thin.rb:30:in
`start_server'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/rack/adapter/abstract.rb:305:in
`start_at_port'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/rack/adapter/abstract.rb:138:in
`start'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/server.rb:174:in
`bootup'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/server.rb:159:in
`block in daemonize'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/server.rb:143:in
`fork'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/server.rb:143:in
`daemonize'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/server.rb:35:in
`start'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core.rb:165:in
`start'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-server-api-10.14.4/bin/chef-server:84:in
`<top (required)>'
/usr/sbin/chef-server:23:in `load'
/usr/sbin/chef-server:23:in `<main>'

[chef] Chef Server Crashes on Cookbook Upload, bjbq4d, 10/03/2012
- [chef] Re: Chef Server Crashes on Cookbook Upload, Daniel DeLeo, 10/03/2012
  - [chef] Re: Re: Chef Server Crashes on Cookbook Upload, Bryan Baugher, 10/03/2012