[chef] Re: Re: Re: Re: Re: Re: Re: chef-client (still) randomly failing

[Daniel DeLeo < >]

On Thursday, November 21, 2013 at 11:35 AM, Phil Cryer wrote:

Daniel, sorry to say this still fails with the new mixlib-shellout. It fails, then fails every other time chef-client runs and never works. I think it may still be an issue with the yum.pid getting in the way. To fix, I ssh in and do:

-> stop chef-client

# /etc/init.d/chef-client stop

Stopping chef-client:                                      [  OK  ]

-> make sure all chef process are done

# ps -fe|grep chef

root     14303 14279  0 13:29 pts/0    00:00:00 grep chef

-> remove the stale yum pid

# rm /var/run/yum.pid

-> run chef-client

# chef-client

And it just works. So, here are the logs from when it was failing, notice it continued to try every hour...

=====

[2013-11-21T11:48:37-06:00] WARN: Current  template[/opt/splunkforwarder/etc/system/local/outputs.conf]: /var/chef/cache/cookbooks/logging/recipes/default.rb:51:in `from_file'

[2013-11-21T11:48:37-06:00] WARN: Cloning resource attributes for file[/etc/httpd/conf.d/ssl.conf] from prior resource (CHEF-3694)

[2013-11-21T11:48:37-06:00] WARN: Previous file[/etc/httpd/conf.d/ssl.conf]: /var/chef/cache/cookbooks/apache2/recipes/default.rb:84:in `block in from_file'

[2013-11-21T11:48:37-06:00] WARN: Current  file[/etc/httpd/conf.d/ssl.conf]: /var/chef/cache/cookbooks/apache2/recipes/mod_ssl.rb:28:in `from_file'

[2013-11-21T11:48:37-06:00] WARN: Cloning resource attributes for template[/etc/httpd/ports.conf] from prior resource (CHEF-3694)

[2013-11-21T11:48:37-06:00] WARN: Previous template[/etc/httpd/ports.conf]: /var/chef/cache/cookbooks/apache2/recipes/default.rb:185:in `from_file'

[2013-11-21T11:48:37-06:00] WARN: Current  template[/etc/httpd/ports.conf]: /var/chef/cache/cookbooks/apache2/recipes/mod_ssl.rb:34:in `from_file'

[2013-11-21T11:48:37-06:00] WARN: Cloning resource attributes for template[/etc/varnish/default.vcl] from prior resource (CHEF-3694)

[2013-11-21T11:48:37-06:00] WARN: Previous template[/etc/varnish/default.vcl]: /var/chef/cache/cookbooks/chef-varnish/recipes/default.rb:60:in `from_file'

[2013-11-21T11:48:37-06:00] WARN: Current  template[/etc/varnish/default.vcl]: /var/chef/cache/cookbooks/app-savviscom/recipes/default.rb:133:in `from_file'

[2013-11-21T11:48:37-06:00] WARN: Cloning resource attributes for service[moxi-server] from prior resource (CHEF-3694)

[2013-11-21T11:48:37-06:00] WARN: Previous service[moxi-server]: /var/chef/cache/cookbooks/couchbase/recipes/moxi.rb:42:in `from_file'

[2013-11-21T11:48:37-06:00] WARN: Current  service[moxi-server]: /var/chef/cache/cookbooks/couchbase/recipes/moxi.rb:56:in `from_file'

[2013-11-21T11:48:37-06:00] WARN: Cloning resource attributes for package[postfix] from prior resource (CHEF-3694)

[2013-11-21T11:48:37-06:00] WARN: Previous package[postfix]: /var/chef/cache/cookbooks/base/recipes/default.rb:82:in `from_file'

[2013-11-21T11:48:37-06:00] WARN: Current  package[postfix]: /var/chef/cache/cookbooks/app-savviscom/recipes/default.rb:186:in `from_file'

[2013-11-21T11:48:37-06:00] WARN: Cloning resource attributes for package[rsyslog] from prior resource (CHEF-3694)

[2013-11-21T11:48:37-06:00] WARN: Previous package[rsyslog]: /var/chef/cache/cookbooks/base/recipes/default.rb:90:in `block in from_file'

[2013-11-21T11:48:37-06:00] WARN: Current  package[rsyslog]: /var/chef/cache/cookbooks/app-savviscom/recipes/default.rb:189:in `from_file'

[2013-11-21T11:48:37-06:00] WARN: Cloning resource attributes for template[/etc/php.ini] from prior resource (CHEF-3694)

[2013-11-21T11:48:37-06:00] WARN: Previous template[/etc/php.ini]: /var/chef/cache/cookbooks/app-savviscom/recipes/default.rb:96:in `from_file'

[2013-11-21T11:48:37-06:00] WARN: Current  template[/etc/php.ini]: /var/chef/cache/cookbooks/php/recipes/package.rb:27:in `from_file'

[2013-11-21T11:48:37-06:00] INFO: Processing template[/root/.profile] action create (base::default line 14)

[2013-11-21T11:48:37-06:00] INFO: Processing template[/root/.curlrc] action create (base::default line 20)

[2013-11-21T11:48:37-06:00] INFO: Processing template[/root/.bash_profile] action create (base::default line 26)

[2013-11-21T11:48:37-06:00] INFO: Processing service[iptables] action disable (base::default line 32)

[2013-11-21T11:48:37-06:00] INFO: Processing service[iptables] action stop (base::default line 32)

[2013-11-21T11:48:37-06:00] INFO: Processing template[/etc/profile.d/proxy.sh] action create (base::default line 35)

[2013-11-21T11:48:37-06:00] INFO: Processing cookbook_file[/etc/selinux/config] action create (base::default line 41)

[2013-11-21T11:48:37-06:00] INFO: Processing cookbook_file[/etc/yum/pluginconf.d/fastestmirror.conf] action create (base::default line 57)

[2013-11-21T11:48:37-06:00] INFO: Processing cookbook_file[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] action create (base::default line 63)

[2013-11-21T11:48:37-06:00] INFO: Processing yum_key[RPM-GPG-KEY-EPEL-6] action add (yum::epel line 22)

[2013-11-21T11:48:37-06:00] INFO: Processing yum_repository[epel] action add (yum::epel line 27)

[2013-11-21T11:48:37-06:00] INFO: Processing package[ntp] action install (base::default line 77)

[2013-11-21T12:03:52-06:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py exceeded timeout 900

[2013-11-21T12:03:52-06:00] INFO: Retrying execution of package[ntp], 3 attempt(s) left

[2013-11-21T12:19:21-06:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py exceeded timeout 900

[2013-11-21T12:19:21-06:00] INFO: Retrying execution of package[ntp], 2 attempt(s) left

[2013-11-21T12:34:50-06:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py exceeded timeout 900

[2013-11-21T12:34:50-06:00] INFO: Retrying execution of package[ntp], 1 attempt(s) left

[2013-11-21T12:50:19-06:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py exceeded timeout 900

[2013-11-21T12:50:19-06:00] INFO: Retrying execution of package[ntp], 0 attempt(s) left

[2013-11-21T13:05:48-06:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py exceeded timeout 900

[2013-11-21T13:05:48-06:00] INFO: Running queued delayed notifications before re-raising exception

[2013-11-21T13:05:48-06:00] ERROR: Running exception handlers

[2013-11-21T13:05:48-06:00] ERROR: Exception handlers complete

[2013-11-21T13:05:48-06:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out

[2013-11-21T13:05:48-06:00] ERROR: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

[2013-11-21T13:05:48-06:00] ERROR: Sleeping for 1800 seconds before trying again

=====

Thanks

I would expect these error messages to look different with the new version of mixlib-shellout. Can you run this recipe:

require 'mixlib/shellout/version'

log "the shellout version is #{Mixlib::ShellOut::VERSION}" do

  level :warn

end

With chef-apply: chef-apply shellout-version.rb

the output should look like:

Recipe: (chef-apply cookbook)::(chef-apply recipe)

  * log[the shellout version is 1.3.0.rc.0] action write[2013-11-21T11:49:45-08:00] WARN: the shellout version is 1.3.0.rc.0

-- 

Daniel DeLeo