[chef] Re: Re: Re: Re: 76 running chef-client processes exhausting system memory

[Daniel DeLeo < >]

On Thursday, October 3, 2013 at 9:56 AM, Phil Cryer wrote:

On Thu, Oct 3, 2013 at 10:41 AM, Daniel DeLeo < "> > wrote:

On Thursday, October 3, 2013 at 7:30 AM, Phil Cryer wrote:

Additionally, I'm currently running 6 nodes, and the 4 that this

always happens to are the web servers (which have 15+ recipes/packages

assigned to them), the 2 db servers (which only have 2 recipes

assigned) never have this issue, which makes me think something is

happening with yum to gum things up.

On Thu, Oct 3, 2013 at 9:19 AM, Phil Cryer < "> > wrote:

ISSUE

===========

76 running chef-client processes exhausting system memory, causing

other apps to crash (in this case zabbix), chef-client is set to run

once an hour, currently showing 76 running processes.

Are you running chef out of cron?

No, it's installed as a service, so it's 'running' all the time, here

you can see it's set to only run once an hour.

root 1663 1 0 00:15 ? 00:00:00

/opt/chef/embedded/bin/ruby /usr/bin/chef-client -d -c

/etc/chef/client.rb -L /var/log/chef/client.log -P

/var/run/chef/client.pid -i 3600 -s 300

Also, while scrolling the logs I did see this error, that I hadn't

caught before - seem to be related to the yum-dump.py script, looks

like I have something wrong with my yum config - "Problem parsing line

'Freeing read locks for locker 0xca9:" will look for that error online

* package[tzdata] action install[2013-10-03T10:00:39-05:00] WARN:

Problem parsing line 'Freeing read locks for locker 0xca7:

32758/140342062917376' from yum-dump.py! Please check your yum

configuration.

[2013-10-03T10:00:39-05:00] WARN: Problem parsing line 'Freeing read

locks for locker 0xca9: 32758/140342062917376' from yum-dump.py!

Please check your yum configuration.

(up to date)

FWIW, I use Debian almost exclusively (sometimes Ubuntu) so this

Centos stuff is new to me :)

Seems like you have 2 problems here. One looks a lot like https://tickets.opscode.com/browse/CHEF-4556

This issue applies when:

* You're running chef daemonized

* You're using the chef-client cookbook to manage the daemon

* You're using using a System V R4 type init system (i.e., plain-ol-init scripts and not daemontools/runit/upstart/systemd/etc.)

* You get into a state where the pid file used by chef's daemonization code is corrupt or stale.

You can generally rectify this by killing all chef processes, removing the pid file, and then starting chef via the init script. We narrowed the range of conditions in which this problem can occur in Chef 11.6, and are going to rewrite that portion of the daemonization code to fix it entirely in 11.8

The yum-dump.py bug is over my head, not being too familiar with yum/CentOS myself. Maybe take a look on the bug tracker to see if there's anything relevant, and if not, create a new issue?

-- 

Daniel DeLeo