[chef] Re: Re: Re: Re: Re: cookbook upload: HTTP 500: Too many links

[AJ Christensen < >]

You can probably smash the sandboxes directory and re-upload all of
the cookbooks, do backups or whatever.

What are all the files? some bad symlinking? you might actually have
to look at them, not just count them :)

--AJ

On 8 February 2012 19:33,  
<
 >
 wrote:
> On Wed, 08 Feb 2012, AJ Christensen wrote:
>
>> No, I meant the directories such as:
>>
>> /var/cache/chef/sandboxes/26f2b236b39b403ca272034f5fa95599
>
> ahh. well...
>
> getting no such file or dir for the paths mentioned in the errors below.
> but:
>
> DR chef server having the problem:
> find /var/cache/chef/sandboxes -maxdepth 1 -type d | wc -l
> 31999
>
> prod chef server that's thankfully not having problem:
> find /var/cache/chef/sandboxes -maxdepth 1 -type d | wc -l
> 1162
>
> yeah that's a lotta dirs in that dir.
>
> how'd that happen? i wonder what dirs therein i can prune, how to
> prevent it from happening again. or if i'll need to take a heavy hammer
> and wipe and recreate this DR chef server.
>
>
>> On 8 February 2012 19:13,  
>> <
 >
>>  wrote:
>> > On Wed, 08 Feb 2012, AJ Christensen wrote:
>> >
>> >> what file system are you using on the chef server? what is the
>> >> contents of the directory the error comes from? have you got some
>> >> crazy symlink stuff going on in a cookbook?
>> >
>> > chef server filesystem is ext3.
>> >
>> > no crazy symlink stuff is going on in any cookbooks (that i know of).
>> > nothing written by me anyway.
>> >
>> > contents of the directory the error comes from .. you mean cwd?
>> >
>> > chef:~/ops/chef-repo$ ls -l
>> > total 48
>> > drwxr-xr-x  2 wopr wopr 4096 2012-01-13 00:32 certificates
>> > -rw-r--r--  1 wopr wopr  156 2012-01-13 00:32 chefignore
>> > drwxr-xr-x  2 wopr wopr 4096 2012-01-13 00:32 config
>> > drwxr-xr-x 49 wopr wopr 4096 2012-02-03 23:55 cookbooks
>> > drwxr-xr-x 11 wopr wopr 4096 2012-02-07 08:20 data_bags
>> > drwxr-xr-x  2 wopr wopr 4096 2012-01-13 00:32 environments
>> > drwxr-xr-x  2 wopr wopr 4096 2012-02-07 08:20 nodes
>> > -rw-r--r--  1 wopr wopr 2484 2012-01-13 00:32 Rakefile
>> > -rw-r--r--  1 wopr wopr 3521 2012-01-13 00:32 README.md
>> > drwxr-xr-x  2 wopr wopr 4096 2012-02-07 08:20 roles
>> > drwxr-xr-x 23 wopr wopr 4096 2012-01-13 00:32 site-cookbooks
>> > drwxr-xr-x  2 wopr wopr 4096 2012-01-13 00:32 tasks
>> >
>> > chef:~/ops/chef-repo$ ls -1 cookbooks/ | wc -l
>> > 48
>> >
>> >
>> >
>> >
>> >
>> >> On 8 February 2012 19:06,  
>> >> <
 >
>> >>  wrote:
>> >> >
>> >> > hiya. seeing this error on one of my chef servers. anyone run into 
>> >> > this?
>> >> > know what to do?
>> >> >
>> >> > chef:~/ops/chef-repo$ knife cookbook upload -a -o ./cookbooks/ -V
>> >> > Uploading activemq                    [1.0.0]
>> >> > INFO: Saving activemq
>> >> > INFO: Validating ruby files
>> >> > INFO: Validating templates
>> >> > INFO: Syntax OK
>> >> > INFO: HTTP Request Returned 500 Internal Server Error: Too many links 
>> >> > - /var/cache/chef/sandboxes/26f2b236b39b403ca272034f5fa95599
>> >> > ERROR: Server returned error for 
>> >> > https://chef.dr.spergtastic.com/sandboxes, retrying 1/5 in 4s
>> >> > INFO: HTTP Request Returned 500 Internal Server Error: Too many links 
>> >> > - /var/cache/chef/sandboxes/242d0dbbe1a84a98b11a6ab776fb5ddd
>> >> > ERROR: Server returned error for 
>> >> > https://chef.dr.spergtastic.com/sandboxes, retrying 2/5 in 7s
>> >> > INFO: HTTP Request Returned 500 Internal Server Error: Too many links 
>> >> > - /var/cache/chef/sandboxes/848dbce5df314dfcaaed178c1e50856f
>> >> > ERROR: Server returned error for 
>> >> > https://chef.dr.spergtastic.com/sandboxes, retrying 3/5 in 11s
>> >> > INFO: HTTP Request Returned 500 Internal Server Error: Too many links 
>> >> > - /var/cache/chef/sandboxes/5e975b05a06347c0bb949cbe1c6f549e
>> >> >
>> >> >
>> >> > i try uploading just one cookbook, and the error is different:
>> >> >
>> >> > chef:~/ops/chef-repo$ knife cookbook upload s3cmd -o ./cookbooks/
>> >> > Uploading s3cmd             [0.0.1]
>> >> > ERROR: Server returned error for 
>> >> > https://chef.dr.spergtastic.com/sandboxes, retrying 1/5 in 3s
>> >> > ERROR: Server returned error for 
>> >> > https://chef.dr.spergtastic.com/sandboxes, retrying 2/5 in 7s
>> >> > ERROR: Server returned error for 
>> >> > https://chef.dr.spergtastic.com/sandboxes, retrying 3/5 in 10s
>> >> >
>> >> >
>> >> > i tried restarting couchdb and chef-server, still no joy.
>> >> >
>> >> > chef-server is 0.10.8-1, ubuntu natty.
>> >> > this is a secondary chef-server in my infrastructure serving a DR
>> >> > datacenter. it loads the same sets of cookbooks, data bags, etc from
>> >> > chef-repo in git.
>> >> >
>> >> > fwiw, i realized i hadn't been compacting couchdb here. the chef 
>> >> > couchdb
>> >> > on the fs was 1.9G. i ran compact and it's now 45Mb. but i'm still 
>> >> > seeing
>> >> > the above error.
>> >> >
>> >> > also fwiw, above compact was done like so, not via chef-server 
>> >> > cookbook:
>> >> >
>> >> > /usr/bin/curl -H "Content-Type: application/json" -X POST 
>> >> > http://localhost:5984/chef/_compact
>> >> > /usr/bin/curl -H "Content-Type: application/json" -X POST 
>> >> > http://localhost:5984/chef/_view_cleanup
>> >> > for view in registrations sandboxes id_map clients checksums 
>> >> > cookbooks data_bags nodes roles users; do
>> >> >    /usr/bin/curl -H "Content-Type: application/json" -X POST 
>> >> > http://localhost:5984/chef/_compact/$view
>> >> >    done
>> >> >
>> >> > thanks
>> >> > kallen
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > and one more bit of log in case it's helpful:
>> >> >
>> >> > merb : chef-server (api) : worker (port 4000) ~ Started request 
>> >> > handling: Wed Feb 08 05:47:27 +0000 2012
>> >> > merb : chef-server (api) : worker (port 4000) ~ Params: 
>> >> > {"action"=>"create", "checksums"=>{"83580e30b45b9a8e6b3359d47
>> >> > d14208e"=>nil, "d217102ad8b64da1aa17218621820d49"=>nil, 
>> >> > "099b134c847994fc237be5593604fad7"=>nil, "cd33cd00aad2e9f89a6
>> >> > 74baa9c49b24c"=>nil, "cd357558feae778ce719e03dedd4f074"=>nil, 
>> >> > "bfccb2c807ce7d9fdb9c6a2cf6f49133"=>nil}, "controller"=
>> >> >>"sandboxes"}
>> >> > merb : chef-server (api) : worker (port 4000) ~ Too many links - 
>> >> > /var/cache/chef/sandboxes/26f2b236b39b403ca272034f5fa95599 - 
>> >> > (Errno::EMLINK)
>> >> > /usr/lib/ruby/1.8/fileutils.rb:243:in `mkdir'
>> >> > /usr/lib/ruby/1.8/fileutils.rb:243:in `fu_mkdir'
>> >> > /usr/lib/ruby/1.8/fileutils.rb:217:in `mkdir_p'
>> >> > /usr/lib/ruby/1.8/fileutils.rb:215:in `reverse_each'
>> >> > /usr/lib/ruby/1.8/fileutils.rb:215:in `mkdir_p'
>> >> > /usr/lib/ruby/1.8/fileutils.rb:201:in `each'
>> >> > /usr/lib/ruby/1.8/fileutils.rb:201:in `mkdir_p'
>> >> > /usr/share/chef-server-api/app/controllers/sandboxes.rb:76:in `create'
>> >> > /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in 
>> >> > `send'
>> >> > /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in 
>> >> > `_call_action'
>> >> > /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:289:in 
>> >> > `_dispatch'
>> >> > /usr/lib/ruby/1.8/merb-core/controller/merb_controller.rb:252:in 
>> >> > `_dispatch'
>> >> > /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:102:in 
>> >> > `dispatch_action'
>> >> > /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:74:in `handle'
>> >> > /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:36:in `handle'
>> >> > /usr/lib/ruby/1.8/merb-core/rack/application.rb:17:in `call'
>> >> > /usr/lib/ruby/1.8/merb-core/rack/middleware/static.rb:28:in `call'
>> >> > /usr/lib/ruby/1.8/rack/content_length.rb:13:in `call'
>> >> > /usr/lib/ruby/1.8/thin/connection.rb:76:in `pre_process'
>> >> > /usr/lib/ruby/1.8/thin/connection.rb:74:in `catch'
>> >> > /usr/lib/ruby/1.8/thin/connection.rb:74:in `pre_process'
>> >> > /usr/lib/ruby/1.8/thin/connection.rb:57:in `process'
>> >> > /usr/lib/ruby/1.8/thin/connection.rb:42:in `receive_data'
>> >> > /usr/lib/ruby/1.8/eventmachine.rb:257:in `run_machine'
>> >> > /usr/lib/ruby/1.8/eventmachine.rb:257:in `run'
>> >> > /usr/lib/ruby/1.8/thin/backends/base.rb:57:in `start'
>> >> > /usr/lib/ruby/1.8/thin/server.rb:156:in `start'
>> >> > /usr/lib/ruby/1.8/merb-core/rack/adapter/thin.rb:30:in `start_server'
>> >> > /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:298:in 
>> >> > `start_at_port'
>> >> > /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:128:in `start'
>> >> > /usr/lib/ruby/1.8/merb-core/server.rb:174:in `bootup'
>> >> > /usr/lib/ruby/1.8/merb-core/server.rb:159:in `daemonize'
>> >> > /usr/lib/ruby/1.8/merb-core/server.rb:143:in `fork'
>> >> > /usr/lib/ruby/1.8/merb-core/server.rb:143:in `daemonize'
>> >> > /usr/lib/ruby/1.8/merb-core/server.rb:35:in `start'
>> >> > /usr/lib/ruby/1.8/merb-core.rb:170:in `start'
>> >> > /usr/sbin/chef-server:86
>> >> >