General discussion about Chef

[chef] Re: Re: Re: Re: Re: Re: chef-client (still) randomly failing


Chronological Thread 
  • From: Phil Cryer < >
  • To:
  • Subject: [chef] Re: Re: Re: Re: Re: Re: chef-client (still) randomly failing
  • Date: Thu, 21 Nov 2013 13:35:39 -0600
Daniel, sorry to say this still fails with the new mixlib-shellout. It fails, then fails every other time chef-client runs and never works. I think it may still be an issue with the yum.pid getting in the way. To fix, I ssh in and do:

-> stop chef-client
# /etc/init.d/chef-client stop
Stopping chef-client:                                      [  OK  ]

-> make sure all chef process are done
# ps -fe|grep chef
root     14303 14279  0 13:29 pts/0    00:00:00 grep chef

-> remove the stale yum pid
# rm /var/run/yum.pid

-> run chef-client
# chef-client

And it just works. So, here are the logs from when it was failing, notice it continued to try every hour...

=====
[2013-11-21T11:48:37-06:00] WARN: Current  template[/opt/splunkforwarder/etc/system/local/outputs.conf]: /var/chef/cache/cookbooks/logging/recipes/default.rb:51:in `from_file'
[2013-11-21T11:48:37-06:00] WARN: Cloning resource attributes for file[/etc/httpd/conf.d/ssl.conf] from prior resource (CHEF-3694)
[2013-11-21T11:48:37-06:00] WARN: Previous file[/etc/httpd/conf.d/ssl.conf]: /var/chef/cache/cookbooks/apache2/recipes/default.rb:84:in `block in from_file'
[2013-11-21T11:48:37-06:00] WARN: Current  file[/etc/httpd/conf.d/ssl.conf]: /var/chef/cache/cookbooks/apache2/recipes/mod_ssl.rb:28:in `from_file'
[2013-11-21T11:48:37-06:00] WARN: Cloning resource attributes for template[/etc/httpd/ports.conf] from prior resource (CHEF-3694)
[2013-11-21T11:48:37-06:00] WARN: Previous template[/etc/httpd/ports.conf]: /var/chef/cache/cookbooks/apache2/recipes/default.rb:185:in `from_file'
[2013-11-21T11:48:37-06:00] WARN: Current  template[/etc/httpd/ports.conf]: /var/chef/cache/cookbooks/apache2/recipes/mod_ssl.rb:34:in `from_file'
[2013-11-21T11:48:37-06:00] WARN: Cloning resource attributes for template[/etc/varnish/default.vcl] from prior resource (CHEF-3694)
[2013-11-21T11:48:37-06:00] WARN: Previous template[/etc/varnish/default.vcl]: /var/chef/cache/cookbooks/chef-varnish/recipes/default.rb:60:in `from_file'
[2013-11-21T11:48:37-06:00] WARN: Current  template[/etc/varnish/default.vcl]: /var/chef/cache/cookbooks/app-savviscom/recipes/default.rb:133:in `from_file'
[2013-11-21T11:48:37-06:00] WARN: Cloning resource attributes for service[moxi-server] from prior resource (CHEF-3694)
[2013-11-21T11:48:37-06:00] WARN: Previous service[moxi-server]: /var/chef/cache/cookbooks/couchbase/recipes/moxi.rb:42:in `from_file'
[2013-11-21T11:48:37-06:00] WARN: Current  service[moxi-server]: /var/chef/cache/cookbooks/couchbase/recipes/moxi.rb:56:in `from_file'
[2013-11-21T11:48:37-06:00] WARN: Cloning resource attributes for package[postfix] from prior resource (CHEF-3694)
[2013-11-21T11:48:37-06:00] WARN: Previous package[postfix]: /var/chef/cache/cookbooks/base/recipes/default.rb:82:in `from_file'
[2013-11-21T11:48:37-06:00] WARN: Current  package[postfix]: /var/chef/cache/cookbooks/app-savviscom/recipes/default.rb:186:in `from_file'
[2013-11-21T11:48:37-06:00] WARN: Cloning resource attributes for package[rsyslog] from prior resource (CHEF-3694)
[2013-11-21T11:48:37-06:00] WARN: Previous package[rsyslog]: /var/chef/cache/cookbooks/base/recipes/default.rb:90:in `block in from_file'
[2013-11-21T11:48:37-06:00] WARN: Current  package[rsyslog]: /var/chef/cache/cookbooks/app-savviscom/recipes/default.rb:189:in `from_file'
[2013-11-21T11:48:37-06:00] WARN: Cloning resource attributes for template[/etc/php.ini] from prior resource (CHEF-3694)
[2013-11-21T11:48:37-06:00] WARN: Previous template[/etc/php.ini]: /var/chef/cache/cookbooks/app-savviscom/recipes/default.rb:96:in `from_file'
[2013-11-21T11:48:37-06:00] WARN: Current  template[/etc/php.ini]: /var/chef/cache/cookbooks/php/recipes/package.rb:27:in `from_file'
[2013-11-21T11:48:37-06:00] INFO: Processing template[/root/.profile] action create (base::default line 14)
[2013-11-21T11:48:37-06:00] INFO: Processing template[/root/.curlrc] action create (base::default line 20)
[2013-11-21T11:48:37-06:00] INFO: Processing template[/root/.bash_profile] action create (base::default line 26)
[2013-11-21T11:48:37-06:00] INFO: Processing service[iptables] action disable (base::default line 32)
[2013-11-21T11:48:37-06:00] INFO: Processing service[iptables] action stop (base::default line 32)
[2013-11-21T11:48:37-06:00] INFO: Processing template[/etc/profile.d/proxy.sh] action create (base::default line 35)
[2013-11-21T11:48:37-06:00] INFO: Processing cookbook_file[/etc/selinux/config] action create (base::default line 41)
[2013-11-21T11:48:37-06:00] INFO: Processing cookbook_file[/etc/yum/pluginconf.d/fastestmirror.conf] action create (base::default line 57)
[2013-11-21T11:48:37-06:00] INFO: Processing cookbook_file[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] action create (base::default line 63)
[2013-11-21T11:48:37-06:00] INFO: Processing yum_key[RPM-GPG-KEY-EPEL-6] action add (yum::epel line 22)
[2013-11-21T11:48:37-06:00] INFO: Processing yum_repository[epel] action add (yum::epel line 27)
[2013-11-21T11:48:37-06:00] INFO: Processing package[ntp] action install (base::default line 77)
[2013-11-21T12:03:52-06:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py exceeded timeout 900
[2013-11-21T12:03:52-06:00] INFO: Retrying execution of package[ntp], 3 attempt(s) left
[2013-11-21T12:19:21-06:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py exceeded timeout 900
[2013-11-21T12:19:21-06:00] INFO: Retrying execution of package[ntp], 2 attempt(s) left
[2013-11-21T12:34:50-06:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py exceeded timeout 900
[2013-11-21T12:34:50-06:00] INFO: Retrying execution of package[ntp], 1 attempt(s) left
[2013-11-21T12:50:19-06:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py exceeded timeout 900
[2013-11-21T12:50:19-06:00] INFO: Retrying execution of package[ntp], 0 attempt(s) left
[2013-11-21T13:05:48-06:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py exceeded timeout 900
[2013-11-21T13:05:48-06:00] INFO: Running queued delayed notifications before re-raising exception
[2013-11-21T13:05:48-06:00] ERROR: Running exception handlers
[2013-11-21T13:05:48-06:00] ERROR: Exception handlers complete
[2013-11-21T13:05:48-06:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2013-11-21T13:05:48-06:00] ERROR: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
[2013-11-21T13:05:48-06:00] ERROR: Sleeping for 1800 seconds before trying again
=====

Thanks



On Tue, Nov 19, 2013 at 2:57 PM, Phil Cryer < " target="_blank"> > wrote:
Installed, thank you, will report back tomorrow.

~]# /opt/chef/embedded/bin/gem install mixlib-shellout --pre
Fetching: mixlib-shellout-1.3.0.rc.0.gem (100%)
Successfully installed mixlib-shellout-1.3.0.rc.0
1 gem installed
Installing ri documentation for mixlib-shellout-1.3.0.rc.0...
Installing RDoc documentation for mixlib-shellout-1.3.0.rc.0...


On Tue, Nov 19, 2013 at 2:32 PM, Daniel DeLeo < " target="_blank"> > wrote:

On Tuesday, November 19, 2013 at 12:30 PM, Phil Cryer wrote:

Ok, I've got that in place on my qa01 - I did a chef-client stop, then a ps to make sure all chef processes were gone, so we'll see how this one goes. If you do a prerelease I can install it on another node, just let me know - and thanks.

We decided to do the prerelease, so you can install that with:

  Omnibus: /opt/chef/embedded/bin/gem install mixlib-shellout —pre
  Otherwise: gem install mixlib-shellout —pre

Thanks for testing this.


-- 
Daniel DeLeo




--



--

Archive powered by MHonArc 2.6.16.

§