[chef] RE: Re: odd errors after restoring couchdb onto new chef server

["Van Fossan,Randy" < >]

Kallen,

Do you have the steps you used to backup and then restore the couchdb to
a newly built server?   If you could post them, I would be extremely
grateful..

Thanks

-----Original Message-----
From: 

 
 
[mailto:
 
 
Sent: Tuesday, February 28, 2012 2:09 PM
To: 

 
Subject: [chef] Re: Re: Re: Re: Re: odd errors after restoring couchdb
onto new chef server


just to close out this thread:

to solve this on the newly built server, i purged all cookbooks, then
reuploaded all from my chef-repo in git. i didn't attempt to backup and
restore the checksum_path dir.

the new restored server seems to be working ok.

thanks for the help!

kallen


On Fri, 24 Feb 2012, Daniel DeLeo wrote:

> 
> 
> On Friday, February 24, 2012 at 6:18 PM, 
> 
 
>  wrote:
> 
> > On Fri, 24 Feb 2012, 
> > 
 
(mailto:
 )
 wrote:
> > 
> > > On Sat, 25 Feb 2012, AJ Christensen wrote:
> > > 
> > > > > 2) wierd errors running a chef-client against new server,
like:
> > > > > * ERROR: Server returned error for 
> > > > > https://chef.venus.spergacula.com/cookbooks/users/1.0.0/files/
> > > > > [snip]
> > > > > * cookbook_file EOFError: end of file reached
> > > > 
> > > 
> > > 
> > > 
> > > > did you copy the upload/copy the cookbooks to the new server? 
> > > > are they in the file cache? Kind of looks like couchdb has 
> > > > information about the files, but the files aren't on the 
> > > > chef-server file system (hazard
> > > > guess)
> > > 
> > > 
> > > 
> > > 
> > > i hadn't done an upload/copy of the cookbooks to the new server 
> > > because i thought the cookbooks were loaded in the couchdb-load. 
> > > when i run knife cookbook list, it looks like all my cookbooks are
already there.
> > > 
> > > regardless, i just reuploaded all cookbooks from my chef-repo to 
> > > the new server. i saw no upload errors. now, running chef-client 
> > > on a non-chef-server node, pointing chef_server_url directly at 
> > > port 4000 rather than hitting nginx proxy over https, is erroring 
> > > out on "EOFError - end of file reached", which is similar to error
shown in gist.
> > > 
> > > 
> > > [Fri, 24 Feb 2012 23:13:08 +0000] DEBUG: Sending HTTP Request via 
> > > GET to 
> > > chef.venus.spergacula.com:4000/cookbooks/mrepo/0.0.1/files/c0cffe5
> > > 0c70191353d8bd9a8bd568ce5 
> > > (http://chef.venus.spergacula.com:4000/cookbooks/mrepo/0.0.1/files
> > > /c0cffe50c70191353d8bd9a8bd568ce5)
> > > [Fri, 24 Feb 2012 23:13:08 +0000] ERROR: Running exception 
> > > handlers [Fri, 24 Feb 2012 23:13:08 +0000] FATAL: Saving node 
> > > information to /var/cache/chef/failed-run-data.json
> > > [Fri, 24 Feb 2012 23:13:08 +0000] ERROR: Exception handlers 
> > > complete [Fri, 24 Feb 2012 23:13:08 +0000] DEBUG: Re-raising 
> > > exception: EOFError - end of file reached
/usr/lib/ruby/1.8/net/protocol.rb:135:in `sysread'
> > > /usr/lib/ruby/1.8/net/protocol.rb:135:in `rbuf_fill'
> > > /usr/lib/ruby/1.8/timeout.rb:67:in `timeout'
> > > /usr/lib/ruby/1.8/timeout.rb:101:in `timeout'
> > > 
> > > 
> > > i'm looking at chef-server debug log and couchdb log. but nothing 
> > > is jumping out at me as relevant. but i may not recognize
relevancy..
> > > 
> > > 
> > > 
> > > last 3 lines of couchdb log fwiw:
> > > 
> > > [Fri, 24 Feb 2012 23:18:13 GMT] [info] [<0.17029.0>] 127.0.0.1 - -

> > > 'POST' /chef/_all_docs?include_docs=true 200 [Fri, 24 Feb 2012 
> > > 23:18:14 GMT] [info] [<0.17030.0>] 127.0.0.1 - - 'GET' 
> > > /chef/_design/id_map/_view/name_to_id?key=[%22client%22,%22admin2.
> > > venus.spergacula.com 
> > > (http://22admin2.venus.spergacula.com)%22]&include_docs=true 200 
> > > [Fri, 24 Feb 2012 23:18:14 GMT] [info] [<0.17031.0>] 127.0.0.1 - -

> > > 'GET' 
> > > /chef/_design/id_map/_view/name_to_id?key=[%22cookbook_version%22,
> > > %22mrepo-0.0.1%22]&include_docs=true 200
> > > 
> > > 
> > > thoughts?
> > 
> > 
> > trying various things, flailing. for the heck of it:
> > 
> > * deleted possible problematic cookbook (mrepo) and reuploaded. same
errors.
> > * deleted client node which i'm testing on (admin2), reran, same
errors.
> > * reloaded couchdb dump again
> > 
> > and now a different cookbook download claims "EOFError - end of file
reached".
> > 
> > 
> > [Sat, 25 Feb 2012 02:14:22 +0000] DEBUG: Sending HTTP Request via 
> > GET to 
> > cheftain04.venus.spergacula.com:4000/cookbooks/begin/0.0.1/files/e44
> > 7bc0fcbef6dafe72007fe70507767 
> > (http://cheftain04.venus.spergacula.com:4000/cookbooks/begin/0.0.1/f
> > iles/e447bc0fcbef6dafe72007fe70507767)
> > [Sat, 25 Feb 2012 02:14:22 +0000] ERROR: Running exception handlers 
> > [Sat, 25 Feb 2012 02:14:22 +0000] FATAL: Saving node information to 
> > /var/cache/chef/failed-run-data.json
> > [Sat, 25 Feb 2012 02:14:22 +0000] ERROR: Exception handlers complete

> > [Sat, 25 Feb 2012 02:14:22 +0000] DEBUG: Re-raising exception: 
> > EOFError - end of file reached
/usr/lib/ruby/1.8/net/protocol.rb:135:in `sysread'
> > /usr/lib/ruby/1.8/net/protocol.rb:135:in `rbuf_fill'
> > 
> > 
> > dunno what to do here.. 
> The files that comprise your cookbooks aren't stored in couch, they're
stored according to their MD5 under whatever you've configured for
`checksum_path` in your server.rb. When you backup/restore your chef
server, you need to restore these files also.
> 
> The error you're seeing occurs because the file transfer happens
outside of the normal request cycle, so server side errors aren't
converted to 500s[1].
> 
> To fix it, you can either bulk delete all cookbooks using the
`--purge` option, or copy the files.
> 
> HTH,
> 
> --
> Dan DeLeo
> 
> 
> 1. Merb is trying to hand off the work to another thread (in the case
of threaded servers) or the event loop (in case of evented servers like
thin) so that the transfer can happen while other requests are
processed. I'm not sure if this even works any more. In any case, I
believe a user has contributed a fix for this problem.