[chef] Re: unattended chef-client runs fail

[Andrew Gross < >]

Looks like it is a timeout connecting to a package repository to check to make sure it has the correct version of ntp.  These are fairly common in my experience.  The simple solution that catches most issues is to add retries and retry_delay to all of your package install blocks to get around minor hiccups.

Old:

package "ntp" do

  action :install

end

New:

package "ntp" do

  action :install

  retry_delay 5

  retries 2

end

Additionally, I would recommend setting up a failure handler than emails you(or dumps somewhere off machine) the Chef run log so you don't have to do so much digging.  The error handler Opscode has is a decent start: http://docs.opscode.com/essentials_handlers.html

On Fri, Nov 15, 2013 at 9:52 AM, Phil Cryer < " target="_blank"> > wrote:

Randomly a chef run will fail - right now I have 4 nodes, all built by chef, all working like champs, but usually overnight there will be 1 or 2 that fail a chef run, so when I check in the morning they haven't been chef'd for 10-12 hours. It says it's trying again in 1800 seconds, but if it's running once an hour it should have succeeded by now... when I login and run chef-client manually, it just works without me having to restart chef or kill any processes. Looking at the stacktrace it looks like it stopped at yum-dump.py - could this be what's holding it up? And if so, why doesn't it clear itself on the next run, since it did when I manually ran it. Thanks

[...LOGS...]

[2013-11-15T08:05:32-06:00] INFO: Processing yum_key[RPM-GPG-KEY-EPEL-6] action add (yum::epel line 22)

[2013-11-15T08:05:32-06:00] INFO: Processing yum_repository[epel] action add (yum::epel line 27)

[2013-11-15T08:05:32-06:00] INFO: Processing package[ntp] action install (ntp::default line 25)

[2013-11-15T08:20:43-06:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py exceeded timeout 900

[2013-11-15T08:20:43-06:00] INFO: Running queued delayed notifications before re-raising exception

[2013-11-15T08:20:43-06:00] ERROR: Running exception handlers

[2013-11-15T08:20:43-06:00] ERROR: Exception handlers complete

[2013-11-15T08:20:43-06:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out

[2013-11-15T08:20:43-06:00] ERROR: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

[2013-11-15T08:20:43-06:00] ERROR: Sleeping for 1800 seconds before trying again

[...STACKTRACE...]

Generated at 2013-11-15 08:20:43 -0600

Mixlib::ShellOut::CommandTimeout: package[ntp] (ntp::default line 25) had an error: Mixlib::ShellOut::CommandTimeout: command timed out:

---- Begin output of /usr/bin/python /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py --options --installed-provides ----

STDOUT:

STDERR:

---- End output of /usr/bin/python /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py --options --installed-provides ----

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:64:in `run_command'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout.rb:225:in `run_command'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell_out.rb:30:in `shell_out'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell_out.rb:35:in `shell_out!'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:714:in `refresh'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:806:in `package_available?'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:1055:in `load_current_resource'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider.rb:97:in `run_action'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource.rb:625:in `run_action'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:49:in `run_action'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in `block (2 levels) in converge'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in `each'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in `block in converge'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:98:in `block in execute_each_resource'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in `call'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in `call_iterator_block'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:96:in `execute_each_resource'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:80:in `converge'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:429:in `converge'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:494:in `do_run'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:199:in `block in run'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in `fork'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in `run'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:183:in `run_chef_client'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:302:in `block in run_application'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in `loop'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in `run_application'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:66:in `run'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/bin/chef-client:26:in `<top (required)>'

/usr/bin/chef-client:23:in `load'

/usr/bin/chef-client:23:in `<main>'

--

http://philcryer.com/