- From: Phil Cryer <
>
- To:
- Subject: [chef] Re: 76 running chef-client processes exhausting system memory
- Date: Thu, 3 Oct 2013 09:30:09 -0500
Additionally, I'm currently running 6 nodes, and the 4 that this
always happens to are the web servers (which have 15+ recipes/packages
assigned to them), the 2 db servers (which only have 2 recipes
assigned) never have this issue, which makes me think something is
happening with yum to gum things up.
On Thu, Oct 3, 2013 at 9:19 AM, Phil Cryer
<
>
wrote:
>
ISSUE
>
===========
>
76 running chef-client processes exhausting system memory, causing
>
other apps to crash (in this case zabbix), chef-client is set to run
>
once an hour, currently showing 76 running processes.
>
>
>
~]# ps -fe | grep chef-client | wc -l
>
76
>
>
Also, when I have failed chef-client runs like this I always find this
>
in the ps; a chef-client worker process that is old (hung?), and a
>
yum-dump.py that always needs to be killed before a chef-client run
>
will be successful.
>
>
# ps -fe|grep chef
>
root 7082 13613 23 09:12 ? 00:00:00 chef-client worker:
>
ppid=13613;start=09:12:03;
>
root 7221 7082 26 09:12 ? 00:00:00 /usr/bin/python
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py
>
--options --installed-provides
>
>
After killing the 76 processes, and a kill -9 on the chef-client work
>
and yum-dump.py, a chef-client run will be successful, as will others
>
that are run automatically after that - it's just at some point down
>
the line one will get hung up and cause this cascade of processes.
>
>
>
>
SPECS
>
===========
>
# chef-client -v
>
Chef: 11.6.0
>
>
# cat /etc/issue.net
>
CentOS release 6.4 (Final)
>
Kernel \r on an \m
>
>
Vmware instance with one core and 3G RAM
>
>
>
>
LOGS
>
===========
>
[...]
>
[2013-10-03T09:06:37-05:00] INFO: Forking chef instance to converge...
>
[2013-10-03T09:06:37-05:00] INFO: *** Chef 11.6.0 ***
>
[2013-10-03T09:06:37-05:00] WARN: unable to detect ipaddress
>
[2013-10-03T09:06:37-05:00] WARN: unable to detect macaddress
>
[2013-10-03T09:06:37-05:00] WARN: unable to detect ip6address
>
[2013-10-03T09:06:37-05:00] INFO: Run List is [recipe[base],
>
recipe[pd-web], recipe[diptables]]
>
[2013-10-03T09:06:37-05:00] INFO: Run List expands to [base, pd-web,
>
diptables]
>
[2013-10-03T09:06:37-05:00] INFO: Starting Chef Run for
>
dc2mgmtsavpd01.mgmt.sdirect
>
[2013-10-03T09:06:37-05:00] INFO: Running start handlers
>
[2013-10-03T09:06:37-05:00] INFO: Start handlers complete.
>
[2013-10-03T09:06:38-05:00] INFO: Loading cookbooks [apache2, apt,
>
aws, base, build-essential, chef-client, chef-varnish, chef_handler,
>
couchbase, cron, database, diptables, dmg, drush, firewall, git, line,
>
mysql, nfs, ntp, openssl, php, postfix, postgresql, rsyslog, runit,
>
savviscom-pd-web, splunk_handler, timezone-ii, ufw, windows, xfs, xml,
>
yum, zabbix]
>
[2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for
>
template[/etc/php.ini] from prior resource (CHEF-3694)
>
[2013-10-03T09:06:39-05:00] WARN: Previous template[/etc/php.ini]:
>
/var/chef/cache/cookbooks/pd-web/recipes/default.rb:4:in `from_file'
>
[2013-10-03T09:06:39-05:00] WARN: Current template[/etc/php.ini]:
>
/var/chef/cache/cookbooks/php/recipes/package.rb:27:in `from_file'
>
[2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for
>
cookbook_file[/etc/httpd/conf/httpd.conf] from prior resource
>
(CHEF-3694)
>
[2013-10-03T09:06:39-05:00] WARN: Previous
>
cookbook_file[/etc/httpd/conf/httpd.conf]:
>
/var/chef/cache/cookbooks/savviscom-pd-web/recipes/default.rb:55:in
>
`from_file'
>
[2013-10-03T09:06:39-05:00] WARN: Current
>
cookbook_file[/etc/httpd/conf/httpd.conf]:
>
/var/chef/cache/cookbooks/apache2/recipes/default.rb:221:in
>
`from_file'
>
[2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for
>
service[apache2] from prior resource (CHEF-3694)
>
[2013-10-03T09:06:39-05:00] WARN: Previous service[apache2]:
>
/var/chef/cache/cookbooks/apache2/recipes/default.rb:24:in `from_file'
>
[2013-10-03T09:06:39-05:00] WARN: Current service[apache2]:
>
/var/chef/cache/cookbooks/apache2/recipes/default.rb:228:in
>
`from_file'
>
[2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for
>
service[moxi-server] from prior resource (CHEF-3694)
>
[2013-10-03T09:06:39-05:00] WARN: Previous service[moxi-server]:
>
/var/chef/cache/cookbooks/couchbase/recipes/moxi.rb:42:in `from_file'
>
[2013-10-03T09:06:39-05:00] WARN: Current service[moxi-server]:
>
/var/chef/cache/cookbooks/couchbase/recipes/moxi.rb:56:in `from_file'
>
[2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for
>
template[/etc/varnish/default.vcl] from prior resource (CHEF-3694)
>
[2013-10-03T09:06:39-05:00] WARN: Previous
>
template[/etc/varnish/default.vcl]:
>
/var/chef/cache/cookbooks/chef-varnish/recipes/default.rb:56:in
>
`from_file'
>
[2013-10-03T09:06:39-05:00] WARN: Current
>
template[/etc/varnish/default.vcl]:
>
/var/chef/cache/cookbooks/savviscom-pd-web/recipes/default.rb:113:in
>
`from_file'
>
[2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for
>
service[varnishlog] from prior resource (CHEF-3694)
>
[2013-10-03T09:06:39-05:00] WARN: Previous service[varnishlog]:
>
/var/chef/cache/cookbooks/chef-varnish/recipes/default.rb:81:in
>
`from_file'
>
[2013-10-03T09:06:39-05:00] WARN: Current service[varnishlog]:
>
/var/chef/cache/cookbooks/savviscom-pd-web/recipes/default.rb:120:in
>
`from_file'
>
[2013-10-03T09:06:39-05:00] WARN: Don't know how to set up automatic
>
iptables on your distribution, sorry. Please submit a bug ticket at
>
https://github.com/wk8/cookbook-iptables/issues
>
[2013-10-03T09:06:39-05:00] INFO: Processing template[/root/.profile]
>
action create (base::default line 1)
>
[2013-10-03T09:06:39-05:00] INFO: Processing
>
template[/root/.bash_profile] action create (base::default line 8)
>
[2013-10-03T09:06:39-05:00] INFO: Processing template[/root/.curlrc]
>
action create (base::default line 15)
>
[2013-10-03T09:06:39-05:00] INFO: Processing
>
yum_key[RPM-GPG-KEY-EPEL-6] action add (yum::epel line 22)
>
[2013-10-03T09:06:39-05:00] INFO: Processing yum_repository[epel]
>
action add (yum::epel line 27)
>
[2013-10-03T09:06:39-05:00] INFO: Processing package[tzdata] action
>
install (timezone-ii::default line 16)
>
[2013-10-03T09:06:39-05:00] INFO: Running queued delayed notifications
>
before re-raising exception
>
[2013-10-03T09:06:39-05:00] ERROR: Running exception handlers
>
[2013-10-03T09:06:39-05:00] ERROR: Exception handlers complete
>
[2013-10-03T09:06:39-05:00] FATAL: Stacktrace dumped to
>
/var/chef/cache/chef-stacktrace.out
>
[2013-10-03T09:06:39-05:00] ERROR:
>
Chef::Exceptions::ChildConvergeError: Chef run process exited
>
unsuccessfully (exit code 1)
>
[2013-10-03T09:06:39-05:00] ERROR: Sleeping for 3600 seconds before trying
>
again
>
>
>
>
STACKTRACE
>
===========
>
# cat /var/chef/cache/chef-stacktrace.out
>
Generated at 2013-10-03 09:06:39 -0500
>
Errno::ENOMEM: package[tzdata] (timezone-ii::default line 16) had an
>
error: Errno::ENOMEM: Cannot allocate memory - fork(2)
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:256:in
>
`fork'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:256:in
>
`fork_subprocess'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:40:in
>
`run_command'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout.rb:225:in
>
`run_command'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell_out.rb:30:in
>
`shell_out'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell_out.rb:35:in
>
`shell_out!'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:714:in
>
`refresh'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:806:in
>
`package_available?'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:1055:in
>
`load_current_resource'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider.rb:97:in
>
`run_action'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource.rb:625:in
>
`run_action'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:49:in
>
`run_action'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in
>
`block (2 levels) in converge'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in
>
`each'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in
>
`block in converge'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:98:in
>
`block in execute_each_resource'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in
>
`call'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in
>
`call_iterator_block'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:85:in
>
`step'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:104:in
>
`iterate'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:55:in
>
`each_with_index'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:96:in
>
`execute_each_resource'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:80:in
>
`converge'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:429:in
>
`converge'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:494:in
>
`do_run'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:199:in
>
`block in run'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in
>
`fork'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in
>
`run'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:183:in
>
`run_chef_client'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:302:in
>
`block in run_application'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in
>
`loop'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in
>
`run_application'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:66:in
>
`run'
>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/bin/chef-client:26:in
>
`<top (required)>'
>
/usr/bin/chef-client:23:in `load'
>
/usr/bin/chef-client:23:in `<main>'
>
>
>
https://gist.github.com/philcryer/6810475
>
--
>
http://philcryer.com
--
http://philcryer.com
Archive powered by MHonArc 2.6.16.