[chef] First run on Ubuntu 12 node: /var/cache/chef/cookbooks hierarchy contains no files

[Kevin Yank < >]

Running chef-client 10.12.0 for the first time on a fresh Ubuntu 12.04 LTS 
node, the run fails after downloading the cookbooks from Chef Server, because 
all the files it supposedly downloaded into the /var/cache/chef/cookbooks 
directory hierarchy are missing!

Taking a look at the debug output, I can see, for example, this portion 
showing a couple of cookbook files that get downloaded successfully:

[2012-09-07T06:57:57+00:00] DEBUG: Signing the request as 
ip-10-171-90-254.us-west-1.compute.internal
[2012-09-07T06:57:57+00:00] DEBUG: String to sign: 'Method:GET
Hashed Path:Bsx3zOo6YGnjwbdCT2G3Fb1FewI=
X-Ops-Content-Hash:2jmj7l5rSw0yVb/vlWAYkK/YBwk=
X-Ops-Timestamp:2012-09-07T06:57:57Z
X-Ops-UserId:ip-10-171-90-254.us-west-1.compute.internal'
Header hash: {"X-Ops-Timestamp"=>"2012-09-07T06:57:57Z", 
"X-Ops-Content-Hash"=>"2jmj7l5rSw0yVb/vlWAYkK/YBwk=", 
"X-Ops-Userid"=>"ip-10-171-90-254.us-west-1.compute.internal", 
"X-Ops-Authorization-6"=>"PBvVh0poHpqoG6aqbKXPqoutjHEc5Zgd7XS9fHFhgg==", 
"X-Ops-Authorization-5"=>"MnUQDK/FwsbwJW4P6nzDIN3IcCWIX9BWsvEdq4zKQ7VX1X2XH9+eo259GJmB",
 
"X-Ops-Authorization-4"=>"dbAAW/ulGTgn5wpdO7QsPz1MWOwlEUXb2RVA/mBnwOoz/Y50/mE6UiJos4f8",
 
"X-Ops-Authorization-3"=>"SgPapXlqI0GhepeyxXNy2GjpaORcvwyWVfj+YHbqcD9V639MNY2T5G2Gj04t",
 
"X-Ops-Authorization-2"=>"BkwrC1hZ/1naKNasziUTfHgVR5L7N63jVIqbmpaiAs9xrf3yVLNztHILohfu",
 
"X-Ops-Authorization-1"=>"re6e46EOyrcFCpVU1IHcjsfpFDqXZNKBIIJ10ZMpldx77d8hLLogal+fTZGe",
 "X-Ops-Sign"=>"algorithm=sha1;version=1.0;"}
[2012-09-07T06:57:57+00:00] DEBUG: Sending HTTP Request via GET to 
ec2-184-169-187-137.us-west-1.compute.amazonaws.com:4000/cookbooks/mysql/1.2.6/files/76c6ff4ec2e7ca2f46e9407ed305f6dd
[2012-09-07T06:57:57+00:00] DEBUG: Streaming download from 
http://ec2-184-169-187-137.us-west-1.compute.amazonaws.com:4000/cookbooks/mysql/1.2.6/files/76c6ff4ec2e7ca2f46e9407ed305f6dd
 to tempfile /tmp/chef-rest20120907-9045-1a50gu1-0
[2012-09-07T06:57:57+00:00] INFO: Storing updated 
cookbooks/mysql/libraries/helpers.rb in the cache.
[2012-09-07T06:57:57+00:00] DEBUG: Creating directory 
/var/cache/chef/cookbooks/mysql/libraries
[2012-09-07T06:57:57+00:00] DEBUG: Signing the request as 
ip-10-171-90-254.us-west-1.compute.internal
[2012-09-07T06:57:57+00:00] DEBUG: String to sign: 'Method:GET
Hashed Path:QdRxdiZUMx/eQd93YXhPtNmk5oI=
X-Ops-Content-Hash:2jmj7l5rSw0yVb/vlWAYkK/YBwk=
X-Ops-Timestamp:2012-09-07T06:57:57Z
X-Ops-UserId:ip-10-171-90-254.us-west-1.compute.internal'
Header hash: {"X-Ops-Timestamp"=>"2012-09-07T06:57:57Z", 
"X-Ops-Content-Hash"=>"2jmj7l5rSw0yVb/vlWAYkK/YBwk=", 
"X-Ops-Userid"=>"ip-10-171-90-254.us-west-1.compute.internal", 
"X-Ops-Authorization-6"=>"0gRheLKQvX5syxKDzurFIAd4HvsQ/e3X49ewpeSU/A==", 
"X-Ops-Authorization-5"=>"LOjc8OQGznPh8pniJz4NPajQjZtHPxqg2/Dd3k5v3enGaGWuGFSp3Tfa8gec",
 
"X-Ops-Authorization-4"=>"nWDybANYZLRKLHD6NNAGBOxFvl0Dwgl323j5I/5SDAi3pDMwjbDZtvw6aDjD",
 
"X-Ops-Authorization-3"=>"aI1BefpY9vs4IHB7dmzj0fU4vfsDH7aWXH+Hocyktmdypk+Ab9aIQAfhtyCO",
 
"X-Ops-Authorization-2"=>"KfLMPzH1OjyidJNpGruNF37MIBD7j7GnxAksI3SQaW8YBBGBIN/+E87tOapl",
 
"X-Ops-Authorization-1"=>"4zLvH8UEN5PMmNfUVTIQ051Mu8IIb6Qw63lUE6TXwRTc6bMpytyD+e/9E/3o",
 "X-Ops-Sign"=>"algorithm=sha1;version=1.0;"}
[2012-09-07T06:57:57+00:00] DEBUG: Sending HTTP Request via GET to 
ec2-184-169-187-137.us-west-1.compute.amazonaws.com:4000/cookbooks/mysql/1.2.6/files/d05e075954b208e1a15f6ddcd1ab83c0
[2012-09-07T06:57:57+00:00] DEBUG: Streaming download from 
http://ec2-184-169-187-137.us-west-1.compute.amazonaws.com:4000/cookbooks/mysql/1.2.6/files/d05e075954b208e1a15f6ddcd1ab83c0
 to tempfile /tmp/chef-rest20120907-9045-109g3we-0
[2012-09-07T06:57:57+00:00] INFO: Storing updated 
cookbooks/mysql/libraries/database.rb in the cache.

Given this, you would expect to find helpers.rb and database.rb in 
/var/cache/chef/cookbooks/mysql/libraries. But they aren’t there. The run 
fails as soon as it attempts to use one of the cookbook files it has 
downloaded:

[2012-09-07T06:58:10+00:00] DEBUG: Loading cookbook mysql's library file: 
/var/cache/chef/cookbooks/mysql/libraries/helpers.rb
[2012-09-07T06:58:10+00:00] ERROR: Running exception handlers
[2012-09-07T06:58:10+00:00] FATAL: Saving node information to 
/var/cache/chef/failed-run-data.json
[2012-09-07T06:58:10+00:00] ERROR: Exception handlers complete
[2012-09-07T06:58:10+00:00] DEBUG: Re-raising exception: LoadError - no such 
file to load -- /var/cache/chef/cookbooks/mysql/libraries/helpers.rb
/var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/run_context.rb:82:in 
`load'
  /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/run_context.rb:82:in 
`load_libraries'
  /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/run_context.rb:120:in 
`call'
  /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/run_context.rb:120:in 
`foreach_cookbook_load_segment'
  /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/run_context.rb:119:in 
`each'
  /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/run_context.rb:119:in 
`foreach_cookbook_load_segment'
  /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/run_context.rb:117:in 
`each'
  /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/run_context.rb:117:in 
`foreach_cookbook_load_segment'
  /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/run_context.rb:80:in 
`load_libraries'
  /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/run_context.rb:57:in 
`load'
  /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/client.rb:199:in 
`setup_run_context'
  /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/client.rb:162:in `run'
  
/var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/application/client.rb:254:in
 `run_application'
  
/var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/application/client.rb:241:in
 `loop'
  
/var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/application/client.rb:241:in
 `run_application'
  /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/application.rb:70:in 
`run'
  /var/lib/gems/1.8/gems/chef-10.12.0/bin/chef-client:26
  /usr/local/bin/chef-client:19:in `load'
  /usr/local/bin/chef-client:19
[2012-09-07T06:58:10+00:00] FATAL: Stacktrace dumped to 
/var/cache/chef/chef-stacktrace.out

Investigating /var/cache/chef/cookbooks after this failed run, the full 
directory hierarchy is there, but it doesn’t contain a single file. 
/var/cache/chef/cookbooks/mysql/libraries exists, for example, but it’s empty.

Any ideas what could be wrong?

This same bootstrapping process is working perfectly for us on Ubuntu 11.04. 
I’m working on moving to Ubuntu 12.04, however, and this problem is where I’m 
stuck.

--
Kevin Yank
Chief Technology Officer, Avalanche Technology Group / AVG (AU/NZ) Pty Ltd

2/101 Tulip Street, Sandringham Victoria 3191 Australia
ph: +61 3 9581 0806