[chef] Re: Re: Re: Re: Re: Re: Re: Re: Re: chef-client (still) randomly failing

[Daniel DeLeo < >]

On Thursday, November 21, 2013 at 5:47 PM, Phil Cryer wrote:

-> I threw this in the end of one of my recipes:

require 'mixlib/shellout/version'

log "the shellout version is #{Mixlib::ShellOut::VERSION}" do

  level :warn

end

-> ran it, and the log says:

  * log[the shellout version is 1.3.0.rc.0] action write[2013-11-21T19:42:11-06:00] WARN: the shellout version is 1.3.0.rc.0

happy to test it again, or another gem, whatever - just let me know. Thanks

On Thu, Nov 21, 2013 at 1:51 PM, Daniel DeLeo < " target="_blank"> > wrote:

On Thursday, November 21, 2013 at 11:35 AM, Phil Cryer wrote:

Daniel, sorry to say this still fails with the new mixlib-shellout. It fails, then fails every other time chef-client runs and never works. I think it may still be an issue with the yum.pid getting in the way. To fix, I ssh in and do:

-> stop chef-client

# /etc/init.d/chef-client stop

Stopping chef-client:                                      [  OK  ]

-> make sure all chef process are done

# ps -fe|grep chef

root     14303 14279  0 13:29 pts/0    00:00:00 grep chef

-> remove the stale yum pid

# rm /var/run/yum.pid

-> run chef-client

# chef-client

And it just works. So, here are the logs from when it was failing, notice it continued to try every hour...

So is there a live process that has the yum lock? Or it’s truly stale and the pid in the file is a dead process? Also is this machine tight on RAM, and is there anything in the OOM killer log? At first glance I can’t think of what chef could do automatically for that case. Perhaps it could read the pid file and check if the process is alive, but that seems fraught with peril and prone to race conditions.

In any case if there are not any stale `yum-dump.py` processes hanging around then this behavior isn’t caused by MIXLIB-16, so I might have sent you on a wild goose chase.