[chef] Re: Re: EC2 Knife Bootstrap -> Ubuntu Package Errors

[Alex Araujo < >]

Hey Jesse, thanks for chiming in.  This is happening on m1.small and m1.large instances.  The timeout issue seems plausible if xfs packages are on a different (default) mirror than other packages as it appears to effect xfs libs almost exclusively.  I'm using chef 0.9.12.  Here's an example error for xfsdump, but it happens regularly with xfsprogs and xfslibs-dev:

ec2-184-72-152-3.compute-1.amazonaws.com [Tue, 25 Jan 2011 03:28:16 +0000] INFO: Installing package[xfsdump] version 3.0.4
ec2-184-72-152-3.compute-1.amazonaws.com [Tue, 25 Jan 2011 03:28:16 +0000] ERROR: package[xfsdump] (/var/chef/cache/cookbooks/xfs/recipes/default.rb:21:in `from_file') had an error:
ec2-184-72-152-3.compute-1.amazonaws.com apt-get -q -y install xfsdump=3.0.4 returned 100, expected 0
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/mixin/command.rb:184:in `handle_command_failures'
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/mixin/command.rb:131:in `run_command'
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/mixin/command.rb:199:in `run_command_with_systems_locale'
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/provider/package/apt.rb:68:in `install_package'
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/provider/package.rb:60:in `action_install'
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/resource.rb:395:in `send'
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/resource.rb:395:in `run_action'
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/runner.rb:53:in `run_action'
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/runner.rb:89:in `converge'
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/runner.rb:89:in `each'
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/runner.rb:89:in `converge'
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/resource_collection.rb:94
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/resource_collection/stepable_iterator.rb:116:in `call'
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/resource_collection/stepable_iterator.rb:116:in `call_iterator_block'
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/resource_collection.rb:92:in `execute_each_resource'
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/runner.rb:84:in `converge'
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/client.rb:268:in `converge'
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/client.rb:171:in `run'
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/application/client.rb:222:in `run_application'
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/application/client.rb:212:in `loop'
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/application/client.rb:212:in `run_application'
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/application.rb:62:in `run'
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/chef-client:26
ec2-184-72-152-3.compute-1.amazonaws.com /usr/bin/chef-client:19:in `load'
ec2-184-72-152-3.compute-1.amazonaws.com /usr/bin/chef-client:19
ec2-184-72-152-3.compute-1.amazonaws.com [Tue, 25 Jan 2011 03:28:16 +0000] ERROR: Running exception handlers
ec2-184-72-152-3.compute-1.amazonaws.com [Tue, 25 Jan 2011 03:28:16 +0000] ERROR: Exception handlers complete
ec2-184-72-152-3.compute-1.amazonaws.com /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/mixin/command.rb:184:in `handle_command_failures': apt-get -q -y install xfsdump=3.0.4 returned 100, expected 0 (Chef::Exceptions::Exec)
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/mixin/command.rb:131:in `run_command'
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/mixin/command.rb:199:in `run_command_with_systems_locale'
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/provider/package/apt.rb:68:in `install_package'
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/provider/package.rb:60:in `action_install'
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/resource.rb:395:in `send'
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/resource.rb:395:in `run_action'
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/runner.rb:53:in `run_action'
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/runner.rb:89:in `converge'
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/runner.rb:89:in `each'
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/runner.rb:89:in `converge'
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/resource_collection.rb:94
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/resource_collection/stepable_iterator.rb:116:in `call'
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/resource_collection/stepable_iterator.rb:116:in `call_iterator_block'
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/resource_collection.rb:92:in `execute_each_resource'
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/runner.rb:84:in `converge'
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/client.rb:268:in `converge'
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/client.rb:171:in `run'
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/application/client.rb:222:in `run_application'
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/application/client.rb:212:in `loop'
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/application/client.rb:212:in `run_application'
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/application.rb:62:in `run'
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/chef-client:26
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/bin/chef-client:19:in `load'
ec2-184-72-152-3.compute-1.amazonaws.com     from /usr/bin/chef-client:19

On Tue, Jan 25, 2011 at 10:26 AM, Jesse Callaway < "> > wrote:

On Mon, Jan 24, 2011 at 3:42 PM, Alex Araujo < " target="_blank"> > wrote:

Has anyone ever seen apt-get install errors when bootstrapping an ec2 node using knife, only to find the package installed successfully when logging in?  Prior to today this would happen very infrequently and now I'm seeing it happen regularly.  Same recipe, different package each time.

Any troubleshooting advice would be greatly appreciated.

Could be a timeout on the first attempt, followed up by a successful attempt the second time. You'll get slowdowns (throttling) on micro instances pretty regularly. I'm just guessing here. Throw up some output for us to see.

-jesse