[chef] 76 running chef-client processes exhausting system memory

[Phil Cryer < >]

ISSUE
===========
76 running chef-client processes exhausting system memory, causing
other apps to crash (in this case zabbix), chef-client is set to run
once an hour, currently showing 76 running processes.


 
 ~]# ps -fe | grep chef-client | wc -l
76

Also, when I have failed chef-client runs like this I always find this
in the ps; a chef-client worker process that is old (hung?), and a
yum-dump.py that always needs to be killed before a chef-client run
will be successful.

# ps -fe|grep chef
root      7082 13613 23 09:12 ?        00:00:00 chef-client worker:
ppid=13613;start=09:12:03;
root      7221  7082 26 09:12 ?        00:00:00 /usr/bin/python
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py
--options --installed-provides

After killing the 76 processes, and a kill -9 on the chef-client work
and yum-dump.py, a chef-client run will be successful, as will others
that are run automatically after that - it's just at some point down
the line one will get hung up and cause this cascade of processes.



SPECS
===========
# chef-client -v
Chef: 11.6.0

# cat /etc/issue.net
CentOS release 6.4 (Final)
Kernel \r on an \m

Vmware instance with one core and 3G RAM



LOGS
===========
[...]
[2013-10-03T09:06:37-05:00] INFO: Forking chef instance to converge...
[2013-10-03T09:06:37-05:00] INFO: *** Chef 11.6.0 ***
[2013-10-03T09:06:37-05:00] WARN: unable to detect ipaddress
[2013-10-03T09:06:37-05:00] WARN: unable to detect macaddress
[2013-10-03T09:06:37-05:00] WARN: unable to detect ip6address
[2013-10-03T09:06:37-05:00] INFO: Run List is [recipe[base],
recipe[pd-web], recipe[diptables]]
[2013-10-03T09:06:37-05:00] INFO: Run List expands to [base, pd-web, 
diptables]
[2013-10-03T09:06:37-05:00] INFO: Starting Chef Run for
dc2mgmtsavpd01.mgmt.sdirect
[2013-10-03T09:06:37-05:00] INFO: Running start handlers
[2013-10-03T09:06:37-05:00] INFO: Start handlers complete.
[2013-10-03T09:06:38-05:00] INFO: Loading cookbooks [apache2, apt,
aws, base, build-essential, chef-client, chef-varnish, chef_handler,
couchbase, cron, database, diptables, dmg, drush, firewall, git, line,
mysql, nfs, ntp, openssl, php, postfix, postgresql, rsyslog, runit,
savviscom-pd-web, splunk_handler, timezone-ii, ufw, windows, xfs, xml,
yum, zabbix]
[2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for
template[/etc/php.ini] from prior resource (CHEF-3694)
[2013-10-03T09:06:39-05:00] WARN: Previous template[/etc/php.ini]:
/var/chef/cache/cookbooks/pd-web/recipes/default.rb:4:in `from_file'
[2013-10-03T09:06:39-05:00] WARN: Current  template[/etc/php.ini]:
/var/chef/cache/cookbooks/php/recipes/package.rb:27:in `from_file'
[2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for
cookbook_file[/etc/httpd/conf/httpd.conf] from prior resource
(CHEF-3694)
[2013-10-03T09:06:39-05:00] WARN: Previous
cookbook_file[/etc/httpd/conf/httpd.conf]:
/var/chef/cache/cookbooks/savviscom-pd-web/recipes/default.rb:55:in
`from_file'
[2013-10-03T09:06:39-05:00] WARN: Current
cookbook_file[/etc/httpd/conf/httpd.conf]:
/var/chef/cache/cookbooks/apache2/recipes/default.rb:221:in
`from_file'
[2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for
service[apache2] from prior resource (CHEF-3694)
[2013-10-03T09:06:39-05:00] WARN: Previous service[apache2]:
/var/chef/cache/cookbooks/apache2/recipes/default.rb:24:in `from_file'
[2013-10-03T09:06:39-05:00] WARN: Current  service[apache2]:
/var/chef/cache/cookbooks/apache2/recipes/default.rb:228:in
`from_file'
[2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for
service[moxi-server] from prior resource (CHEF-3694)
[2013-10-03T09:06:39-05:00] WARN: Previous service[moxi-server]:
/var/chef/cache/cookbooks/couchbase/recipes/moxi.rb:42:in `from_file'
[2013-10-03T09:06:39-05:00] WARN: Current  service[moxi-server]:
/var/chef/cache/cookbooks/couchbase/recipes/moxi.rb:56:in `from_file'
[2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for
template[/etc/varnish/default.vcl] from prior resource (CHEF-3694)
[2013-10-03T09:06:39-05:00] WARN: Previous
template[/etc/varnish/default.vcl]:
/var/chef/cache/cookbooks/chef-varnish/recipes/default.rb:56:in
`from_file'
[2013-10-03T09:06:39-05:00] WARN: Current
template[/etc/varnish/default.vcl]:
/var/chef/cache/cookbooks/savviscom-pd-web/recipes/default.rb:113:in
`from_file'
[2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for
service[varnishlog] from prior resource (CHEF-3694)
[2013-10-03T09:06:39-05:00] WARN: Previous service[varnishlog]:
/var/chef/cache/cookbooks/chef-varnish/recipes/default.rb:81:in
`from_file'
[2013-10-03T09:06:39-05:00] WARN: Current  service[varnishlog]:
/var/chef/cache/cookbooks/savviscom-pd-web/recipes/default.rb:120:in
`from_file'
[2013-10-03T09:06:39-05:00] WARN: Don't know how to set up automatic
iptables on your distribution, sorry. Please submit a bug ticket at
https://github.com/wk8/cookbook-iptables/issues
[2013-10-03T09:06:39-05:00] INFO: Processing template[/root/.profile]
action create (base::default line 1)
[2013-10-03T09:06:39-05:00] INFO: Processing
template[/root/.bash_profile] action create (base::default line 8)
[2013-10-03T09:06:39-05:00] INFO: Processing template[/root/.curlrc]
action create (base::default line 15)
[2013-10-03T09:06:39-05:00] INFO: Processing
yum_key[RPM-GPG-KEY-EPEL-6] action add (yum::epel line 22)
[2013-10-03T09:06:39-05:00] INFO: Processing yum_repository[epel]
action add (yum::epel line 27)
[2013-10-03T09:06:39-05:00] INFO: Processing package[tzdata] action
install (timezone-ii::default line 16)
[2013-10-03T09:06:39-05:00] INFO: Running queued delayed notifications
before re-raising exception
[2013-10-03T09:06:39-05:00] ERROR: Running exception handlers
[2013-10-03T09:06:39-05:00] ERROR: Exception handlers complete
[2013-10-03T09:06:39-05:00] FATAL: Stacktrace dumped to
/var/chef/cache/chef-stacktrace.out
[2013-10-03T09:06:39-05:00] ERROR:
Chef::Exceptions::ChildConvergeError: Chef run process exited
unsuccessfully (exit code 1)
[2013-10-03T09:06:39-05:00] ERROR: Sleeping for 3600 seconds before trying 
again



STACKTRACE
===========
# cat /var/chef/cache/chef-stacktrace.out
Generated at 2013-10-03 09:06:39 -0500
Errno::ENOMEM: package[tzdata] (timezone-ii::default line 16) had an
error: Errno::ENOMEM: Cannot allocate memory - fork(2)
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:256:in
`fork'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:256:in
`fork_subprocess'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:40:in
`run_command'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout.rb:225:in
`run_command'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell_out.rb:30:in
`shell_out'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell_out.rb:35:in
`shell_out!'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:714:in
`refresh'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:806:in
`package_available?'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:1055:in
`load_current_resource'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider.rb:97:in
`run_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource.rb:625:in
`run_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:49:in
`run_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in
`block (2 levels) in converge'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in
`each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in
`block in converge'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:98:in
`block in execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in
`call'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in
`call_iterator_block'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:85:in
`step'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:104:in
`iterate'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:55:in
`each_with_index'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:96:in
`execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:80:in
`converge'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:429:in
`converge'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:494:in
`do_run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:199:in
`block in run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in
`fork'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in
`run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:183:in
`run_chef_client'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:302:in
`block in run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in
`loop'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in
`run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:66:in
`run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/bin/chef-client:26:in
`<top (required)>'
/usr/bin/chef-client:23:in `load'
/usr/bin/chef-client:23:in `<main>'


https://gist.github.com/philcryer/6810475
-- 
http://philcryer.com