[chef] Re: Re: Re: Re: odd errors after restoring couchdb onto new chef server

[Daniel DeLeo < >]

On Friday, February 24, 2012 at 6:18 PM, 

 
 wrote:

> On Fri, 24 Feb 2012, 
> 
 
>  
> (mailto:
 )
>  wrote:
> 
> > On Sat, 25 Feb 2012, AJ Christensen wrote:
> > 
> > > > 2) wierd errors running a chef-client against new server, like:
> > > > * ERROR: Server returned error for 
> > > > https://chef.venus.spergacula.com/cookbooks/users/1.0.0/files/[snip]
> > > > * cookbook_file EOFError: end of file reached
> > > 
> > 
> > 
> > 
> > > did you copy the upload/copy the cookbooks to the new server? are they
> > > in the file cache? Kind of looks like couchdb has information about
> > > the files, but the files aren't on the chef-server file system (hazard
> > > guess)
> > 
> > 
> > 
> > 
> > i hadn't done an upload/copy of the cookbooks to the new server because
> > i thought the cookbooks were loaded in the couchdb-load. when i run
> > knife cookbook list, it looks like all my cookbooks are already there.
> > 
> > regardless, i just reuploaded all cookbooks from my chef-repo to the new
> > server. i saw no upload errors. now, running chef-client on a
> > non-chef-server node, pointing chef_server_url directly at port 4000
> > rather than hitting nginx proxy over https, is erroring out on
> > "EOFError - end of file reached", which is similar to error shown in gist.
> > 
> > 
> > [Fri, 24 Feb 2012 23:13:08 +0000] DEBUG: Sending HTTP Request via GET to 
> > chef.venus.spergacula.com:4000/cookbooks/mrepo/0.0.1/files/c0cffe50c70191353d8bd9a8bd568ce5
> >  
> > (http://chef.venus.spergacula.com:4000/cookbooks/mrepo/0.0.1/files/c0cffe50c70191353d8bd9a8bd568ce5)
> > [Fri, 24 Feb 2012 23:13:08 +0000] ERROR: Running exception handlers
> > [Fri, 24 Feb 2012 23:13:08 +0000] FATAL: Saving node information to 
> > /var/cache/chef/failed-run-data.json
> > [Fri, 24 Feb 2012 23:13:08 +0000] ERROR: Exception handlers complete
> > [Fri, 24 Feb 2012 23:13:08 +0000] DEBUG: Re-raising exception: EOFError - 
> > end of file reached
> > /usr/lib/ruby/1.8/net/protocol.rb:135:in `sysread'
> > /usr/lib/ruby/1.8/net/protocol.rb:135:in `rbuf_fill'
> > /usr/lib/ruby/1.8/timeout.rb:67:in `timeout'
> > /usr/lib/ruby/1.8/timeout.rb:101:in `timeout'
> > 
> > 
> > i'm looking at chef-server debug log and couchdb log. but nothing is
> > jumping out at me as relevant. but i may not recognize relevancy..
> > 
> > 
> > 
> > last 3 lines of couchdb log fwiw:
> > 
> > [Fri, 24 Feb 2012 23:18:13 GMT] [info] [<0.17029.0>] 127.0.0.1 - - 'POST' 
> > /chef/_all_docs?include_docs=true 200
> > [Fri, 24 Feb 2012 23:18:14 GMT] [info] [<0.17030.0>] 127.0.0.1 - - 'GET' 
> > /chef/_design/id_map/_view/name_to_id?key=[%22client%22,%22admin2.venus.spergacula.com
> >  (http://22admin2.venus.spergacula.com)%22]&include_docs=true 200
> > [Fri, 24 Feb 2012 23:18:14 GMT] [info] [<0.17031.0>] 127.0.0.1 - - 'GET' 
> > /chef/_design/id_map/_view/name_to_id?key=[%22cookbook_version%22,%22mrepo-0.0.1%22]&include_docs=true
> >  200
> > 
> > 
> > thoughts?
> 
> 
> trying various things, flailing. for the heck of it:
> 
> * deleted possible problematic cookbook (mrepo) and reuploaded. same errors.
> * deleted client node which i'm testing on (admin2), reran, same errors.
> * reloaded couchdb dump again
> 
> and now a different cookbook download claims "EOFError - end of file 
> reached".
> 
> 
> [Sat, 25 Feb 2012 02:14:22 +0000] DEBUG: Sending HTTP Request via GET to 
> cheftain04.venus.spergacula.com:4000/cookbooks/begin/0.0.1/files/e447bc0fcbef6dafe72007fe70507767
>  
> (http://cheftain04.venus.spergacula.com:4000/cookbooks/begin/0.0.1/files/e447bc0fcbef6dafe72007fe70507767)
> [Sat, 25 Feb 2012 02:14:22 +0000] ERROR: Running exception handlers
> [Sat, 25 Feb 2012 02:14:22 +0000] FATAL: Saving node information to 
> /var/cache/chef/failed-run-data.json
> [Sat, 25 Feb 2012 02:14:22 +0000] ERROR: Exception handlers complete
> [Sat, 25 Feb 2012 02:14:22 +0000] DEBUG: Re-raising exception: EOFError - 
> end of file reached
> /usr/lib/ruby/1.8/net/protocol.rb:135:in `sysread'
> /usr/lib/ruby/1.8/net/protocol.rb:135:in `rbuf_fill'
> 
> 
> dunno what to do here.. 
The files that comprise your cookbooks aren't stored in couch, they're stored 
according to their MD5 under whatever you've configured for `checksum_path` 
in your server.rb. When you backup/restore your chef server, you need to 
restore these files also.

The error you're seeing occurs because the file transfer happens outside of 
the normal request cycle, so server side errors aren't converted to 500s[1].

To fix it, you can either bulk delete all cookbooks using the `--purge` 
option, or copy the files.

HTH,

-- 
Dan DeLeo


1. Merb is trying to hand off the work to another thread (in the case of 
threaded servers) or the event loop (in case of evented servers like thin) so 
that the transfer can happen while other requests are processed. I'm not sure 
if this even works any more. In any case, I believe a user has contributed a 
fix for this problem.