[chef] Re: Re: odd errors after restoring couchdb onto new chef server

On Sat, 25 Feb 2012, AJ Christensen wrote:

> On 25 February 2012 11:28,  
> <
 >
>  wrote:
> >
> > hi. i'm trying to retire one chef server, and create a new one by loading
> > couchdb backups from the old one. i'm taking the restored chef server
> > for a spin and seeing odd errors. i'd like to ask about two things here:
> >
> > 1) previous knife search results are no longer present on new server
> 
> you have to rebuild the index (solr), knife index rebuild
 
aha. makes sense. done.

> > 2) wierd errors running a chef-client against new server, like:
> > * ERROR: Server returned error for 
> > https://chef.venus.spergacula.com/cookbooks/users/1.0.0/files/[snip]
> > * cookbook_file EOFError: end of file reached
 
> did you copy the upload/copy the cookbooks to the new server? are they
> in the file cache? Kind of looks like couchdb has information about
> the files, but the files aren't on the chef-server file system (hazard
> guess)


i hadn't done an upload/copy of the cookbooks to the new server because
i thought the cookbooks were loaded in the couchdb-load. when i run
knife cookbook list, it looks like all my cookbooks are already there.

regardless, i just reuploaded all cookbooks from my chef-repo to the new
server. i saw no upload errors. now, running chef-client on a
non-chef-server node, pointing chef_server_url directly at port 4000
rather than hitting nginx proxy over https, is erroring out on
"EOFError - end of file reached", which is similar to error shown in gist.


[Fri, 24 Feb 2012 23:13:08 +0000] DEBUG: Sending HTTP Request via GET to 
chef.venus.spergacula.com:4000/cookbooks/mrepo/0.0.1/files/c0cffe50c70191353d8bd9a8bd568ce5
[Fri, 24 Feb 2012 23:13:08 +0000] ERROR: Running exception handlers
[Fri, 24 Feb 2012 23:13:08 +0000] FATAL: Saving node information to 
/var/cache/chef/failed-run-data.json
[Fri, 24 Feb 2012 23:13:08 +0000] ERROR: Exception handlers complete
[Fri, 24 Feb 2012 23:13:08 +0000] DEBUG: Re-raising exception: EOFError - end 
of file reached
/usr/lib/ruby/1.8/net/protocol.rb:135:in `sysread'
  /usr/lib/ruby/1.8/net/protocol.rb:135:in `rbuf_fill'
  /usr/lib/ruby/1.8/timeout.rb:67:in `timeout'
  /usr/lib/ruby/1.8/timeout.rb:101:in `timeout'


i'm looking at chef-server debug log and couchdb log. but nothing is
jumping out at me as relevant. but i may not recognize relevancy..



last 3 lines of couchdb log fwiw:

[Fri, 24 Feb 2012 23:18:13 GMT] [info] [<0.17029.0>] 127.0.0.1 - - 'POST' 
/chef/_all_docs?include_docs=true 200
[Fri, 24 Feb 2012 23:18:14 GMT] [info] [<0.17030.0>] 127.0.0.1 - - 'GET' 
/chef/_design/id_map/_view/name_to_id?key=[%22client%22,%22admin2.venus.spergacula.com%22]&include_docs=true
 200
[Fri, 24 Feb 2012 23:18:14 GMT] [info] [<0.17031.0>] 127.0.0.1 - - 'GET' 
/chef/_design/id_map/_view/name_to_id?key=[%22cookbook_version%22,%22mrepo-0.0.1%22]&include_docs=true
 200


thoughts?


> > it makes me wonder if my couchdb load is borked. i do pass to couchdb-load
> > "--ignore-errors" because it won't load without that .. :>
> >
> >
> > the first question:
> >
> > here's the old chef server i'm attempting to retire. i do indeed get
> > results from this query:
> >
> > [cheftain02-auw2p chef-repo]$ knife search node 'role:postfix-server' | 
> > grep ^Node
> > Node Name:   admin1.venus.spergacula.com
> > Node Name:   admin2.venus.spergacula.com
> >
> >
> > new chef server which was restored from the old one's couchdb:
> >
> > [cheftain04-auw2p chef-repo]$ knife search node 'role:postfix-server'
> > 0 items found
> >
> > .. nothing. why?
> >
> >
> >
> >
> > the second question involves lots of debugging output, seen here:
> > https://gist.github.com/1904175
> >
> > this gist shows me trying to run cheftain04 as a chef-client against
> > itself. doing this worked for me on the old server, cheftain02.
> >
> > any idea why i'm seeing the 2 kinds of errors therein?
> >
> > ERROR: Server returned error for 
> > https://chef.venus.spergacula.com/cookbooks/users/1.0.0/files/[snip]
> >
> > and
> >
> > [Fri, 24 Feb 2012 21:55:03 +0000] DEBUG: Re-raising exception: EOFError - 
> > cookbook_file[/home/billeh/.ssh/known_hosts] (users::user-file-dist line 
> > 49) had an error: EOFError: end of file reached
> >
> >
> > i note that i was able to successfully run a different chef-client
> > against the new chef server successfully (after i disabled something
> > in a recipe that relied on search results that are currently missing).