- From:
- To:
- Subject: [chef] Re: Re: Re: Re: cookbook upload: HTTP 500: Too many links
- Date: Tue, 7 Feb 2012 22:33:10 -0800
On Wed, 08 Feb 2012, AJ Christensen wrote:
>
No, I meant the directories such as:
>
>
/var/cache/chef/sandboxes/26f2b236b39b403ca272034f5fa95599
ahh. well...
getting no such file or dir for the paths mentioned in the errors below.
but:
DR chef server having the problem:
find /var/cache/chef/sandboxes -maxdepth 1 -type d | wc -l
31999
prod chef server that's thankfully not having problem:
find /var/cache/chef/sandboxes -maxdepth 1 -type d | wc -l
1162
yeah that's a lotta dirs in that dir.
how'd that happen? i wonder what dirs therein i can prune, how to
prevent it from happening again. or if i'll need to take a heavy hammer
and wipe and recreate this DR chef server.
>
On 8 February 2012 19:13,
>
<
>
>
wrote:
>
> On Wed, 08 Feb 2012, AJ Christensen wrote:
>
>
>
>> what file system are you using on the chef server? what is the
>
>> contents of the directory the error comes from? have you got some
>
>> crazy symlink stuff going on in a cookbook?
>
>
>
> chef server filesystem is ext3.
>
>
>
> no crazy symlink stuff is going on in any cookbooks (that i know of).
>
> nothing written by me anyway.
>
>
>
> contents of the directory the error comes from .. you mean cwd?
>
>
>
> chef:~/ops/chef-repo$ ls -l
>
> total 48
>
> drwxr-xr-x 2 wopr wopr 4096 2012-01-13 00:32 certificates
>
> -rw-r--r-- 1 wopr wopr 156 2012-01-13 00:32 chefignore
>
> drwxr-xr-x 2 wopr wopr 4096 2012-01-13 00:32 config
>
> drwxr-xr-x 49 wopr wopr 4096 2012-02-03 23:55 cookbooks
>
> drwxr-xr-x 11 wopr wopr 4096 2012-02-07 08:20 data_bags
>
> drwxr-xr-x 2 wopr wopr 4096 2012-01-13 00:32 environments
>
> drwxr-xr-x 2 wopr wopr 4096 2012-02-07 08:20 nodes
>
> -rw-r--r-- 1 wopr wopr 2484 2012-01-13 00:32 Rakefile
>
> -rw-r--r-- 1 wopr wopr 3521 2012-01-13 00:32 README.md
>
> drwxr-xr-x 2 wopr wopr 4096 2012-02-07 08:20 roles
>
> drwxr-xr-x 23 wopr wopr 4096 2012-01-13 00:32 site-cookbooks
>
> drwxr-xr-x 2 wopr wopr 4096 2012-01-13 00:32 tasks
>
>
>
> chef:~/ops/chef-repo$ ls -1 cookbooks/ | wc -l
>
> 48
>
>
>
>
>
>
>
>
>
>
>
>> On 8 February 2012 19:06,
>
>> <
>
>
>> wrote:
>
>> >
>
>> > hiya. seeing this error on one of my chef servers. anyone run into
>
>> > this?
>
>> > know what to do?
>
>> >
>
>> > chef:~/ops/chef-repo$ knife cookbook upload -a -o ./cookbooks/ -V
>
>> > Uploading activemq [1.0.0]
>
>> > INFO: Saving activemq
>
>> > INFO: Validating ruby files
>
>> > INFO: Validating templates
>
>> > INFO: Syntax OK
>
>> > INFO: HTTP Request Returned 500 Internal Server Error: Too many links
>
>> > - /var/cache/chef/sandboxes/26f2b236b39b403ca272034f5fa95599
>
>> > ERROR: Server returned error for
>
>> > https://chef.dr.spergtastic.com/sandboxes, retrying 1/5 in 4s
>
>> > INFO: HTTP Request Returned 500 Internal Server Error: Too many links
>
>> > - /var/cache/chef/sandboxes/242d0dbbe1a84a98b11a6ab776fb5ddd
>
>> > ERROR: Server returned error for
>
>> > https://chef.dr.spergtastic.com/sandboxes, retrying 2/5 in 7s
>
>> > INFO: HTTP Request Returned 500 Internal Server Error: Too many links
>
>> > - /var/cache/chef/sandboxes/848dbce5df314dfcaaed178c1e50856f
>
>> > ERROR: Server returned error for
>
>> > https://chef.dr.spergtastic.com/sandboxes, retrying 3/5 in 11s
>
>> > INFO: HTTP Request Returned 500 Internal Server Error: Too many links
>
>> > - /var/cache/chef/sandboxes/5e975b05a06347c0bb949cbe1c6f549e
>
>> >
>
>> >
>
>> > i try uploading just one cookbook, and the error is different:
>
>> >
>
>> > chef:~/ops/chef-repo$ knife cookbook upload s3cmd -o ./cookbooks/
>
>> > Uploading s3cmd [0.0.1]
>
>> > ERROR: Server returned error for
>
>> > https://chef.dr.spergtastic.com/sandboxes, retrying 1/5 in 3s
>
>> > ERROR: Server returned error for
>
>> > https://chef.dr.spergtastic.com/sandboxes, retrying 2/5 in 7s
>
>> > ERROR: Server returned error for
>
>> > https://chef.dr.spergtastic.com/sandboxes, retrying 3/5 in 10s
>
>> >
>
>> >
>
>> > i tried restarting couchdb and chef-server, still no joy.
>
>> >
>
>> > chef-server is 0.10.8-1, ubuntu natty.
>
>> > this is a secondary chef-server in my infrastructure serving a DR
>
>> > datacenter. it loads the same sets of cookbooks, data bags, etc from
>
>> > chef-repo in git.
>
>> >
>
>> > fwiw, i realized i hadn't been compacting couchdb here. the chef
>
>> > couchdb
>
>> > on the fs was 1.9G. i ran compact and it's now 45Mb. but i'm still
>
>> > seeing
>
>> > the above error.
>
>> >
>
>> > also fwiw, above compact was done like so, not via chef-server
>
>> > cookbook:
>
>> >
>
>> > /usr/bin/curl -H "Content-Type: application/json" -X POST
>
>> > http://localhost:5984/chef/_compact
>
>> > /usr/bin/curl -H "Content-Type: application/json" -X POST
>
>> > http://localhost:5984/chef/_view_cleanup
>
>> > for view in registrations sandboxes id_map clients checksums cookbooks
>
>> > data_bags nodes roles users; do
>
>> > /usr/bin/curl -H "Content-Type: application/json" -X POST
>
>> > http://localhost:5984/chef/_compact/$view
>
>> > done
>
>> >
>
>> > thanks
>
>> > kallen
>
>> >
>
>> >
>
>> >
>
>> >
>
>> > and one more bit of log in case it's helpful:
>
>> >
>
>> > merb : chef-server (api) : worker (port 4000) ~ Started request
>
>> > handling: Wed Feb 08 05:47:27 +0000 2012
>
>> > merb : chef-server (api) : worker (port 4000) ~ Params:
>
>> > {"action"=>"create", "checksums"=>{"83580e30b45b9a8e6b3359d47
>
>> > d14208e"=>nil, "d217102ad8b64da1aa17218621820d49"=>nil,
>
>> > "099b134c847994fc237be5593604fad7"=>nil, "cd33cd00aad2e9f89a6
>
>> > 74baa9c49b24c"=>nil, "cd357558feae778ce719e03dedd4f074"=>nil,
>
>> > "bfccb2c807ce7d9fdb9c6a2cf6f49133"=>nil}, "controller"=
>
>> >>"sandboxes"}
>
>> > merb : chef-server (api) : worker (port 4000) ~ Too many links -
>
>> > /var/cache/chef/sandboxes/26f2b236b39b403ca272034f5fa95599 -
>
>> > (Errno::EMLINK)
>
>> > /usr/lib/ruby/1.8/fileutils.rb:243:in `mkdir'
>
>> > /usr/lib/ruby/1.8/fileutils.rb:243:in `fu_mkdir'
>
>> > /usr/lib/ruby/1.8/fileutils.rb:217:in `mkdir_p'
>
>> > /usr/lib/ruby/1.8/fileutils.rb:215:in `reverse_each'
>
>> > /usr/lib/ruby/1.8/fileutils.rb:215:in `mkdir_p'
>
>> > /usr/lib/ruby/1.8/fileutils.rb:201:in `each'
>
>> > /usr/lib/ruby/1.8/fileutils.rb:201:in `mkdir_p'
>
>> > /usr/share/chef-server-api/app/controllers/sandboxes.rb:76:in `create'
>
>> > /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in
>
>> > `send'
>
>> > /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in
>
>> > `_call_action'
>
>> > /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:289:in
>
>> > `_dispatch'
>
>> > /usr/lib/ruby/1.8/merb-core/controller/merb_controller.rb:252:in
>
>> > `_dispatch'
>
>> > /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:102:in
>
>> > `dispatch_action'
>
>> > /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:74:in `handle'
>
>> > /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:36:in `handle'
>
>> > /usr/lib/ruby/1.8/merb-core/rack/application.rb:17:in `call'
>
>> > /usr/lib/ruby/1.8/merb-core/rack/middleware/static.rb:28:in `call'
>
>> > /usr/lib/ruby/1.8/rack/content_length.rb:13:in `call'
>
>> > /usr/lib/ruby/1.8/thin/connection.rb:76:in `pre_process'
>
>> > /usr/lib/ruby/1.8/thin/connection.rb:74:in `catch'
>
>> > /usr/lib/ruby/1.8/thin/connection.rb:74:in `pre_process'
>
>> > /usr/lib/ruby/1.8/thin/connection.rb:57:in `process'
>
>> > /usr/lib/ruby/1.8/thin/connection.rb:42:in `receive_data'
>
>> > /usr/lib/ruby/1.8/eventmachine.rb:257:in `run_machine'
>
>> > /usr/lib/ruby/1.8/eventmachine.rb:257:in `run'
>
>> > /usr/lib/ruby/1.8/thin/backends/base.rb:57:in `start'
>
>> > /usr/lib/ruby/1.8/thin/server.rb:156:in `start'
>
>> > /usr/lib/ruby/1.8/merb-core/rack/adapter/thin.rb:30:in `start_server'
>
>> > /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:298:in
>
>> > `start_at_port'
>
>> > /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:128:in `start'
>
>> > /usr/lib/ruby/1.8/merb-core/server.rb:174:in `bootup'
>
>> > /usr/lib/ruby/1.8/merb-core/server.rb:159:in `daemonize'
>
>> > /usr/lib/ruby/1.8/merb-core/server.rb:143:in `fork'
>
>> > /usr/lib/ruby/1.8/merb-core/server.rb:143:in `daemonize'
>
>> > /usr/lib/ruby/1.8/merb-core/server.rb:35:in `start'
>
>> > /usr/lib/ruby/1.8/merb-core.rb:170:in `start'
>
>> > /usr/sbin/chef-server:86
>
>> >
Archive powered by MHonArc 2.6.16.