- From: Kenneth Stailey <
>
- To:
- Subject: [chef] Re: Re: Re: Re: Re: Re: Sudden Increase in Merb RAM Use
- Date: Wed, 3 Feb 2010 15:17:22 -0800 (PST)
- Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=npSBW5aWEJ1UVKKKe4ovRRqHkqlbFZRP29weYIJksA04aWjFDN3pBWKvqm0ElgeovcBV1UPusaexxn1VrqZ8YQO83nY/5exUjA+aoqgvg+ZNOqhRATEc4/ns4482feV0NPZ+NkHGkSm6C3zuaPrVJwCsaP1+0F21/H6NFe1HDj8=;
http://tickets.opscode.com/browse/CHEF-920
--- On Wed, 2/3/10, Adam Jacob
<
>
wrote:
>
From: Adam Jacob
>
<
>
>
Subject: [chef] Re: Re: Re: Re: Re: Sudden Increase in Merb RAM Use
>
To:
>
>
Date: Wednesday, February 3, 2010, 5:45 PM
>
Also, can you file a ticket?
>
Lets move this into the ticket tracking
>
system, so we can keep all the artifacts in the same
>
place.
>
>
Adam
>
>
On Wed, Feb 3, 2010 at 2:44 PM, Adam Jacob
>
<
>
>
wrote:
>
> Next step here is to get this running under something
>
that can give us
>
> some memory profiling - may I recommend bleak_house?
>
>
>
> http://blog.evanweaver.com/files/doc/fauna/bleak_house/files/README.html
>
>
>
> Is anyone else seeing this kind of growth on 64bit?
>
>
>
> It's "normal" to see ruby use roughly double the
>
memory on 64bit
>
> systems, at least in my experience. We tend to see
>
roughly 50-60MB
>
> resident across the board on 32bit, and it's pretty
>
stable... but we
>
> don't have any production systems at 9.10.
>
>
>
> Adam
>
>
>
> On Wed, Feb 3, 2010 at 2:25 PM, Kenneth Stailey
>
> <
>
>
wrote:
>
>> Hi Adam,
>
>>
>
>> $ dpkg -l | grep "ii ruby" | awk '{print $2 " "
>
$3 }'
>
>> ruby 4.2
>
>> ruby1.8 1.8.7.174-1ubuntu1
>
>> rubygems 1.3.5-1ubuntu2
>
>> rubygems1.8 1.3.5-1ubuntu2
>
>>
>
>> $ ruby --version
>
>> ruby 1.8.7 (2009-06-12 patchlevel 174)
>
[x86_64-linux]
>
>>
>
>> Actually looking at the charts shows a gradual
>
decline in free RAM for a while (hours and hours) and then a
>
sudden decline.
>
>>
>
>> There are 107 chef clients which go off twice an
>
hour but not all at once since they use splayed cron jobs.
>
The splay is done by using a modulo of the last octet of
>
the IP address not by any Chef splaying mechanism. This
>
yields an average of 2-3 a minute that check in nearly
>
simultaneously, maybe a few seconds apart.
>
>>
>
>> The Chef server is running in a VM which is not
>
hosting any other major applications. It has sshd etc.
>
>>
>
>> Thanks,
>
>> Ken
>
>>
>
>> --- On Wed, 2/3/10, Adam Jacob
>
>> <
>
>
wrote:
>
>>
>
>>> From: Adam Jacob
>
>>> <
>
>
>>> Subject: [chef] Re: Re: Re: Sudden Increase in
>
Merb RAM Use
>
>>> To:
>
>>>
>
>>> Date: Wednesday, February 3, 2010, 5:06 PM
>
>>> Well, it certainly smells like a leak
>
>>> - I'm not seeing this behavior,
>
>>> though, and we haven't had any other reports
>
of this kind
>
>>> of growth
>
>>> with the chef server. I'm also only running
>
32bit
>
>>> ruby, which may be
>
>>> something.
>
>>>
>
>>> My next request will be to line up the growth
>
in your
>
>>> graphs with the
>
>>> timing of particular servers checking in.
>
>>>
>
>>> If nothing obvious comes from that, it'll be
>
time to start
>
>>> talking
>
>>> about breaking out a debugger and seeing
>
what's what.
>
>>>
>
>>> Can you send the version of ruby your running
>
along as
>
>>> well?
>
>>>
>
>>> Best,
>
>>> Adam
>
>>>
>
>>> On Wed, Feb 3, 2010 at 1:58 PM, Kenneth
>
Stailey
>
<
>
>
>>> wrote:
>
>>> > Particulars:
>
>>> >
>
>>> > $ lsb_release -a
>
>>> > No LSB modules are available.
>
>>> > Distributor ID: Ubuntu
>
>>> > Description: Ubuntu 9.10
>
>>> > Release: 9.10
>
>>> > Codename: karmic
>
>>> >
>
>>> > $ uname -m
>
>>> > x86_64
>
>>> >
>
>>> > $ COLUMNS=40 dpkg -l | grep chef
>
>>> > ii chef 0.7.16-1
>
>>> configuration management system written in R
>
>>> > ii chef-indexer 0.7.16-1
>
Creates search
>
>>> indexes of Chef node attribut
>
>>> > ii chef-server 0.7.16-1
>
Merb
>
>>> application providing centralized manag
>
>>> > ii chef-server-sl 0.7.16-1
>
Merb app slice
>
>>> providing centralized managem
>
>>> > ii libchef-ruby 0.7.16-1
>
Ruby libraries
>
>>> for Chef configuration manage
>
>>> > ii libchef-ruby1. 0.7.16-1
>
Ruby 1.8
>
>>> libraries for Chef configuration ma
>
>>> >
>
>>> > Looking closer the increase in RAM usage
>
is not
>
>>> sudden. I was looking at a chart of "free
>
memory" *blush*
>
>>> so the "spikes" were really reboots resulting
>
in sudden
>
>>> increase of RAM.
>
>>> >
>
>>> > However, the merb worker was using an rss
>
of ~2.3 GiB
>
>>> >
>
>>> > $ ps vax --sort=-rss |head
>
>>> > PID TTY STAT TIME MAJFL
>
TRS DRS
>
>>> RSS %MEM COMMAND
>
>>> > 29513 ? R 2192:16 1141997
>
2 2549253
>
>>> 2352616 79.9 merb : worker (port 4000)
>
>>> > 9882 ? S 39:31 31673
>
2
>
>>> 172461 111188 3.7 ruby /usr/bin/chef-indexer
>
-d -c
>
>>> /etc/chef/indexer.rb
>
>>> > 29514 ? S 20:39 55304
>
2
>
>>> 132805 58284 1.9 merb : worker (port 4001)
>
>>> > 29512 ? S 811:15 11118
>
2
>
>>> 104457 29028 0.9 merb : spawner (ports
>
4000)
>
>>> >
>
>>> > Since the reboot the rss of the restarted
>
merb worker
>
>>> has climbed steadily from about 185 MiB to 320
>
MiB
>
>>> >
>
>>> > $ ps vax --sort=-rss |head -n 5
>
>>> > PID TTY STAT TIME MAJFL
>
TRS DRS
>
>>> RSS %MEM COMMAND
>
>>> > 1728 ? S 105:02
>
1 2
>
>>> 400393 322248 6.6 merb : worker (port 4000)
>
>>> > 1613 ? S 0:12
>
0 2
>
>>> 128161 78968 1.6 /usr/bin/ruby1.8
>
/usr/bin/stompserver -C
>
>>> /etc/stompserver.conf
>
>>> > 1729 ? S 1:08
>
0 2
>
>>> 109313 36864 0.7 merb : worker (port 4001)
>
>>> > 1727 ? S 41:55
>
3 2
>
>>> 105145 32532 0.6 merb : spawner (ports
>
4000)
>
>>> >
>
>>> > Does it have a memory leak or is it
>
merely a resource
>
>>> hog?
>
>>> >
>
>>> > Thanks,
>
>>> > Ken
>
>>> >
>
>>> > --- On Wed, 2/3/10, Adam Jacob
>
>>> > <
>
>
>>> wrote:
>
>>> >
>
>>> >> From: Adam Jacob
>
>>> >> <
>
>
>>> >> Subject: [chef] Re: Sudden Increase
>
in Merb RAM
>
>>> Use
>
>>> >> To:
>
>>> >> "
"
>
>>> <
>
>
>>> >> Cc:
>
>>> >> "
"
>
>>> <
>
>
>>> >> Date: Wednesday, February 3, 2010,
>
11:16 AM
>
>>> >> What's the frequency of the
>
>>> >> spike? Does it correlate to a
>
particular set of
>
>>> nodes
>
>>> >> checking in?
>
>>> >> Just to be sure, this is on 0.7.x?
>
>>> >>
>
>>> >> Sent from my iPhone
>
>>> >> On Feb 3, 2010, at 8:10 AM, Kenneth
>
Stailey
>
<
>
>
>>> >> wrote:
>
>>> >>
>
>>> >> Hi,
>
>>> >>
>
>>> >> merb RAM usage (rss) appears to stay
>
flat for a
>
>>> long time
>
>>> >> and then suddenly shoots up.
>
>>> >>
>
>>> >> After the RAM usage has spiked it
>
looks like
>
>>> this:
>
>>> >>
>
>>> >> $ ps vax --sort=-rss |head
>
>>> >> PID TTY
>
>>> >> STAT TIME MAJFL
>
>>> >> TRS DRS RSS %MEM COMMAND
>
>>> >> 29513 ?
>
>>> >> R 2192:16 1141997 2
>
2549253
>
>>> >> 2352616 79.9 merb : worker (port
>
>>> >>
>
>>> >> 4000)
>
>>> >>
>
>>> >> 9882 ?
>
>>> >> S 39:31
>
>>> >> 31673 2 172461 111188 3.7
>
ruby
>
>>> >> /usr/bin/chef-indexer -d -c
>
/etc/chef/indexer.rb
>
>>> >> 29514 ?
>
>>> >> S 20:39
>
>>> >> 55304 2 132805 58284 1.9
>
merb
>
>>> >> : worker (port
>
>>> >> 4001)
>
>>> >>
>
>>> >> 29512 ?
>
>>> >> S 811:15
>
>>> >> 11118 2 104457 29028 0.9
>
merb
>
>>> >> : spawner (ports
>
>>> >>
>
>>> >> 4000)
>
>>> >>
>
>>> >> 10213 ?
>
>>> >> S 3:37
>
>>> >> 119973 2 301561 18080 0.6
>
>>> >> /usr/bin/ruby1.8 /usr/bin/stompserver
>
-C
>
>>> >> /etc/stompserver.conf
>
>>> >> 15666 ?
>
>>> >> Sl 0:15 1473 1617
>
>>> >> 111930 10768 0.3
>
>>> >>
>
/usr/lib/erlang/erts-5.7.2/bin/beam.smp -Bd -K
>
>>> true -- -root
>
>>> >> /usr/lib/erlang -progname erl --
>
-home
>
>>> /var/lib/couchdb
>
>>> >> -noshell -noinput -smp auto -sasl
>
errlog_type
>
>>> error -pa
>
>>> >>
>
/usr/lib/couchdb/erlang/lib/couch-0.10.0/ebin
>
>>> >>
>
/usr/lib/couchdb/erlang/lib/mochiweb-r97/ebin
>
>>> >>
>
/usr/lib/couchdb/erlang/lib/ibrowse-1.5.2/ebin
>
>>> >>
>
/usr/lib/couchdb/erlang/lib/erlang-oauth/ebin
>
>>> -eval
>
>>> >> application:load(ibrowse) -eval
>
>>> application:load(oauth)
>
>>> >> -eval application:load(crypto) -eval
>
>>> application:load(couch)
>
>>> >> -eval crypto:start() -eval
>
ssl:start() -eval
>
>>> ibrowse:start()
>
>>> >> -eval couch_server:start([
>
>>> >> "/etc/couchdb/default.ini",
>
>>> >> "/etc/couchdb/local.ini",
>
>>> >> "/etc/couchdb/default.ini",
>
>>> >> "/etc/couchdb/local.ini"]), receive
>
done ->
>
>>> >> done end. -pidfile
>
/var/run/couchdb/couchdb.pid
>
>>> -heart
>
>>> >>
>
>>> >> Bumped up VM that is running Chef
>
server from 3 GB
>
>>> RAM to 5
>
>>> >> GB RAM.
>
>>> >>
>
>>> >> After the reboot merb is only using
>
about 185 MiB
>
>>> rss vs.
>
>>> >> 2.3 GiB
>
>>> >>
>
>>> >> $ ps vax --sort=-rss |head
>
>>> >> PID TTY
>
>>> >> STAT TIME MAJFL
>
>>> >> TRS DRS RSS %MEM COMMAND
>
>>> >> 29513 ?
>
>>> >> R 2192:16
>
>>> >> 1141997 2 2549253 2352616 79.9
>
merb :
>
>>> worker
>
>>> >> (port
>
>>> >> 4000)
>
>>> >>
>
>>> >> 9882 ?
>
>>> >> S 39:31
>
>>> >> 31673 2 172461 111188 3.7
>
ruby
>
>>> >> /usr/bin/chef-indexer -d -c
>
/etc/chef/indexer.rb
>
>>> >> 29514 ?
>
>>> >> S 20:39
>
>>> >> 55304 2 132805 58284 1.9
>
merb
>
>>> >> : worker (port
>
>>> >>
>
>>> >> 4001)
>
>>> >>
>
>>> >> 29512 ?
>
>>> >> S 811:15
>
>>> >> 11118 2 104457 29028 0.9
>
merb
>
>>> >> : spawner (ports
>
>>> >> 4000)
>
>>> >>
>
>>> >> 10213
>
>>> >> ?
>
>>> >> S 3:37
>
>>> >> 119973 2 301561 18080 0.6
>
>>> >> /usr/bin/ruby1.8 /usr/bin/stompserver
>
-C
>
>>> >> /etc/stompserver.conf
>
>>> >> 15666 ?
>
>>> >> Sl 0:15 1473 1617
>
>>> >> 111930 10768 0.3
>
>>> >>
>
/usr/lib/erlang/erts-5.7.2/bin/beam.smp -Bd -K
>
>>> true -- -root
>
>>> >> /usr/lib/erlang -progname erl --
>
-home
>
>>> /var/lib/couchdb
>
>>> >> -noshell -noinput -smp auto -sasl
>
errlog_type
>
>>> error -pa
>
>>> >>
>
/usr/lib/couchdb/erlang/lib/couch-0.10.0/ebin
>
>>> >>
>
/usr/lib/couchdb/erlang/lib/mochiweb-r97/ebin
>
>>> >>
>
/usr/lib/couchdb/erlang/lib/ibrowse-1.5.2/ebin
>
>>> >>
>
/usr/lib/couchdb/erlang/lib/erlang-oauth/ebin
>
>>> -eval
>
>>> >> application:load(ibrowse) -eval
>
>>> application:load(oauth)
>
>>> >> -eval application:load(crypto) -eval
>
>>> application:load(couch)
>
>>> >> -eval crypto:start() -eval
>
ssl:start() -eval
>
>>> ibrowse:start()
>
>>> >> -eval couch_server:start([
>
>>> >> "/etc/couchdb/default.ini",
>
>>> >> "/etc/couchdb/local.ini",
>
>>> >> "/etc/couchdb/default.ini",
>
>>> >> "/etc/couchdb/local.ini"]), receive
>
done ->
>
>>> >> done end. -pidfile
>
/var/run/couchdb/couchdb.pid
>
>>> -heart
>
>>> >>
>
>>> >> We are keeping charts of RAM use over
>
time via
>
>>> Nagios/RRD
>
>>> >> and they show the increase in RAM
>
usage happens
>
>>> suddenly not
>
>>> >> gradually. It also repeats over
>
time.
>
>>> >>
>
>>> >> Do other people see this issue?
>
>>> >>
>
>>> >> Thanks,
>
>>> >> Ken
>
>>> >>
>
>>> >>
>
>>> >
>
>>>
>
>>>
>
>>>
>
>>> --
>
>>> Opscode, Inc.
>
>>> Adam Jacob, CTO
>
>>> T: (206) 508-7449 E:
>
>>>
>
>>>
>
>>
>
>
>
>
>
>
>
> --
>
> Opscode, Inc.
>
> Adam Jacob, CTO
>
> T: (206) 508-7449 E:
>
>
>
>
>
>
>
>
--
>
Opscode, Inc.
>
Adam Jacob, CTO
>
T: (206) 508-7449 E:
>
>
Archive powered by MHonArc 2.6.16.