[chef] chef-server 0.9.14 not updating cookbook_index on upload

[Jon Wood < >]

Hi there,

I upgraded a Chef server and it's nodes to 0.9.14 on Ubuntu 10.04 from
the Opscode packages yesterday, and I'm now seeing intermittent errors
when uploading new cookbooks, although it's frustratingly only on
happening on some nodes.

As I understand it one of the changes that took place was for unused
checksums to be deleted at the end of a Chef run, which requires the
path for the Chef server to store its checksums in to be changed.

My server now uses /var/lib/chef/cookbook_index as the store for the
server, while leaving the client's checksums in
/var/cache/chef/checksums, however I'm getting increasingly frequent
errors when running chef-client on nodes. The server will return a 500
response code, with the following in the log file:

merb : chef-server (api) : worker (port 4000) ~ Started request
handling: Thu Mar 17 16:22:29 +0000 2011
merb : chef-server (api) : worker (port 4000) ~ Params:
{"cookbook_version"=>"0.21.4", "action"=>"show_file",
"cookbook_name"=>"mysql",
"checksum"=>"2ddc3a4aa15c27989a74dd69a72e13ea",
"controller"=>"cookbooks"}
merb : chef-server (api) : worker (port 4000) ~ File with checksum
2ddc3a4aa15c27989a74dd69a72e13ea not found in the repository (this
should not happen) - (Merb::ControllerExceptions::InternalServerError)
/usr/share/chef-server-api/app/controllers/cookbooks.rb:88:in `show_file'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in `send'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in
`_call_action'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:289:in 
`_dispatch'
/usr/lib/ruby/1.8/merb-core/controller/merb_controller.rb:252:in `_dispatch'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:102:in `dispatch_action'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:74:in `handle'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:36:in `handle'
/usr/lib/ruby/1.8/merb-core/rack/application.rb:17:in `call'
/usr/lib/ruby/1.8/rack/content_length.rb:13:in `call'
/usr/lib/ruby/1.8/thin/connection.rb:76:in `pre_process'
/usr/lib/ruby/1.8/thin/connection.rb:74:in `catch'
/usr/lib/ruby/1.8/thin/connection.rb:74:in `pre_process'
/usr/lib/ruby/1.8/thin/connection.rb:57:in `process'
/usr/lib/ruby/1.8/thin/connection.rb:42:in `receive_data'
/usr/lib/ruby/1.8/eventmachine.rb:240:in `run_machine'
/usr/lib/ruby/1.8/eventmachine.rb:240:in `run'
/usr/lib/ruby/1.8/thin/backends/base.rb:57:in `start'
/usr/lib/ruby/1.8/thin/server.rb:156:in `start'
/usr/lib/ruby/1.8/merb-core/rack/adapter/thin.rb:30:in `start_server'
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:298:in `start_at_port'
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:128:in `start'
/usr/lib/ruby/1.8/merb-core/server.rb:174:in `bootup'
/usr/lib/ruby/1.8/merb-core/server.rb:159:in `daemonize'
/usr/lib/ruby/1.8/merb-core/server.rb:143:in `fork'
/usr/lib/ruby/1.8/merb-core/server.rb:143:in `daemonize'
/usr/lib/ruby/1.8/merb-core/server.rb:35:in `start'
/usr/lib/ruby/1.8/merb-core.rb:170:in `start'
/usr/sbin/chef-server:85

When checking the filesystem that checksum does not exist in
/var/lib/chef/cookbook_index/2d/2ddc3a4aa15c27989a74dd69a72e13ea, but
does exist in the original path of
/var/cache/chef/checksums/2d/2ddc3a4aa15c27989a74dd69a72e13ea.
Deleting the contents of /var/cache/chef/ didn't help either.

I'm fairly stumped with this now, so any pointers in the right
direction would be very much appreciated.

Cheers,
Jon
-- 
Jon Wood
Blank Pad Development

07827 888143