General discussion about Chef

[chef] Re: Re: Re: Re: Re: Re: Re: Re: chef-client (still) randomly failing


Chronological Thread 
  • From: Phil Cryer < >
  • To:
  • Subject: [chef] Re: Re: Re: Re: Re: Re: Re: Re: chef-client (still) randomly failing
  • Date: Thu, 21 Nov 2013 19:47:27 -0600
-> I threw this in the end of one of my recipes:
require 'mixlib/shellout/version'
log "the shellout version is #{Mixlib::ShellOut::VERSION}" do
  level :warn
end

-> ran it, and the log says:
  * log[the shellout version is 1.3.0.rc.0] action write[2013-11-21T19:42:11-06:00] WARN: the shellout version is 1.3.0.rc.0

happy to test it again, or another gem, whatever - just let me know. Thanks


On Thu, Nov 21, 2013 at 1:51 PM, Daniel DeLeo < " target="_blank"> > wrote:
On Thursday, November 21, 2013 at 11:35 AM, Phil Cryer wrote:
Daniel, sorry to say this still fails with the new mixlib-shellout. It fails, then fails every other time chef-client runs and never works. I think it may still be an issue with the yum.pid getting in the way. To fix, I ssh in and do:

-> stop chef-client
# /etc/init.d/chef-client stop
Stopping chef-client:                                      [  OK  ]

-> make sure all chef process are done
# ps -fe|grep chef
root     14303 14279  0 13:29 pts/0    00:00:00 grep chef

-> remove the stale yum pid
# rm /var/run/yum.pid

-> run chef-client
# chef-client

And it just works. So, here are the logs from when it was failing, notice it continued to try every hour...

=====
[2013-11-21T11:48:37-06:00] WARN: Current  template[/opt/splunkforwarder/etc/system/local/outputs.conf]: /var/chef/cache/cookbooks/logging/recipes/default.rb:51:in `from_file'
[2013-11-21T11:48:37-06:00] WARN: Cloning resource attributes for file[/etc/httpd/conf.d/ssl.conf] from prior resource (CHEF-3694)
[2013-11-21T11:48:37-06:00] WARN: Previous file[/etc/httpd/conf.d/ssl.conf]: /var/chef/cache/cookbooks/apache2/recipes/default.rb:84:in `block in from_file'
[2013-11-21T11:48:37-06:00] WARN: Current  file[/etc/httpd/conf.d/ssl.conf]: /var/chef/cache/cookbooks/apache2/recipes/mod_ssl.rb:28:in `from_file'
[2013-11-21T11:48:37-06:00] WARN: Cloning resource attributes for template[/etc/httpd/ports.conf] from prior resource (CHEF-3694)
[2013-11-21T11:48:37-06:00] WARN: Previous template[/etc/httpd/ports.conf]: /var/chef/cache/cookbooks/apache2/recipes/default.rb:185:in `from_file'
[2013-11-21T11:48:37-06:00] WARN: Current  template[/etc/httpd/ports.conf]: /var/chef/cache/cookbooks/apache2/recipes/mod_ssl.rb:34:in `from_file'
[2013-11-21T11:48:37-06:00] WARN: Cloning resource attributes for template[/etc/varnish/default.vcl] from prior resource (CHEF-3694)
[2013-11-21T11:48:37-06:00] WARN: Previous template[/etc/varnish/default.vcl]: /var/chef/cache/cookbooks/chef-varnish/recipes/default.rb:60:in `from_file'
[2013-11-21T11:48:37-06:00] WARN: Current  template[/etc/varnish/default.vcl]: /var/chef/cache/cookbooks/app-savviscom/recipes/default.rb:133:in `from_file'
[2013-11-21T11:48:37-06:00] WARN: Cloning resource attributes for service[moxi-server] from prior resource (CHEF-3694)
[2013-11-21T11:48:37-06:00] WARN: Previous service[moxi-server]: /var/chef/cache/cookbooks/couchbase/recipes/moxi.rb:42:in `from_file'
[2013-11-21T11:48:37-06:00] WARN: Current  service[moxi-server]: /var/chef/cache/cookbooks/couchbase/recipes/moxi.rb:56:in `from_file'
[2013-11-21T11:48:37-06:00] WARN: Cloning resource attributes for package[postfix] from prior resource (CHEF-3694)
[2013-11-21T11:48:37-06:00] WARN: Previous package[postfix]: /var/chef/cache/cookbooks/base/recipes/default.rb:82:in `from_file'
[2013-11-21T11:48:37-06:00] WARN: Current  package[postfix]: /var/chef/cache/cookbooks/app-savviscom/recipes/default.rb:186:in `from_file'
[2013-11-21T11:48:37-06:00] WARN: Cloning resource attributes for package[rsyslog] from prior resource (CHEF-3694)
[2013-11-21T11:48:37-06:00] WARN: Previous package[rsyslog]: /var/chef/cache/cookbooks/base/recipes/default.rb:90:in `block in from_file'
[2013-11-21T11:48:37-06:00] WARN: Current  package[rsyslog]: /var/chef/cache/cookbooks/app-savviscom/recipes/default.rb:189:in `from_file'
[2013-11-21T11:48:37-06:00] WARN: Cloning resource attributes for template[/etc/php.ini] from prior resource (CHEF-3694)
[2013-11-21T11:48:37-06:00] WARN: Previous template[/etc/php.ini]: /var/chef/cache/cookbooks/app-savviscom/recipes/default.rb:96:in `from_file'
[2013-11-21T11:48:37-06:00] WARN: Current  template[/etc/php.ini]: /var/chef/cache/cookbooks/php/recipes/package.rb:27:in `from_file'
[2013-11-21T11:48:37-06:00] INFO: Processing template[/root/.profile] action create (base::default line 14)
[2013-11-21T11:48:37-06:00] INFO: Processing template[/root/.curlrc] action create (base::default line 20)
[2013-11-21T11:48:37-06:00] INFO: Processing template[/root/.bash_profile] action create (base::default line 26)
[2013-11-21T11:48:37-06:00] INFO: Processing service[iptables] action disable (base::default line 32)
[2013-11-21T11:48:37-06:00] INFO: Processing service[iptables] action stop (base::default line 32)
[2013-11-21T11:48:37-06:00] INFO: Processing template[/etc/profile.d/proxy.sh] action create (base::default line 35)
[2013-11-21T11:48:37-06:00] INFO: Processing cookbook_file[/etc/selinux/config] action create (base::default line 41)
[2013-11-21T11:48:37-06:00] INFO: Processing cookbook_file[/etc/yum/pluginconf.d/fastestmirror.conf] action create (base::default line 57)
[2013-11-21T11:48:37-06:00] INFO: Processing cookbook_file[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] action create (base::default line 63)
[2013-11-21T11:48:37-06:00] INFO: Processing yum_key[RPM-GPG-KEY-EPEL-6] action add (yum::epel line 22)
[2013-11-21T11:48:37-06:00] INFO: Processing yum_repository[epel] action add (yum::epel line 27)
[2013-11-21T11:48:37-06:00] INFO: Processing package[ntp] action install (base::default line 77)
[2013-11-21T12:03:52-06:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py exceeded timeout 900
[2013-11-21T12:03:52-06:00] INFO: Retrying execution of package[ntp], 3 attempt(s) left
[2013-11-21T12:19:21-06:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py exceeded timeout 900
[2013-11-21T12:19:21-06:00] INFO: Retrying execution of package[ntp], 2 attempt(s) left
[2013-11-21T12:34:50-06:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py exceeded timeout 900
[2013-11-21T12:34:50-06:00] INFO: Retrying execution of package[ntp], 1 attempt(s) left
[2013-11-21T12:50:19-06:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py exceeded timeout 900
[2013-11-21T12:50:19-06:00] INFO: Retrying execution of package[ntp], 0 attempt(s) left
[2013-11-21T13:05:48-06:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py exceeded timeout 900
[2013-11-21T13:05:48-06:00] INFO: Running queued delayed notifications before re-raising exception
[2013-11-21T13:05:48-06:00] ERROR: Running exception handlers
[2013-11-21T13:05:48-06:00] ERROR: Exception handlers complete
[2013-11-21T13:05:48-06:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2013-11-21T13:05:48-06:00] ERROR: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
[2013-11-21T13:05:48-06:00] ERROR: Sleeping for 1800 seconds before trying again
=====

Thanks
I would expect these error messages to look different with the new version of mixlib-shellout. Can you run this recipe:

require 'mixlib/shellout/version'
log "the shellout version is #{Mixlib::ShellOut::VERSION}" do
  level :warn
end


With chef-apply: chef-apply shellout-version.rb

the output should look like:

Recipe: (chef-apply cookbook)::(chef-apply recipe)
  * log[the shellout version is 1.3.0.rc.0] action write[2013-11-21T11:49:45-08:00] WARN: the shellout version is 1.3.0.rc.0


-- 
Daniel DeLeo





--

Archive powered by MHonArc 2.6.16.

§