[chef] Re: Re: Re: Sudden Increase in Merb RAM Use

[Adam Jacob < >]

Well, it certainly smells like a leak - I'm not seeing this behavior,
though, and we haven't had any other reports of this kind of growth
with the chef server.  I'm also only running 32bit ruby, which may be
something.

My next request will be to line up the growth in your graphs with the
timing of particular servers checking in.

If nothing obvious comes from that, it'll be time to start talking
about breaking out a debugger and seeing what's what.

Can you send the version of ruby your running along as well?

Best,
Adam

On Wed, Feb 3, 2010 at 1:58 PM, Kenneth Stailey 
<
 >
 wrote:
> Particulars:
>
> $ lsb_release -a
> No LSB modules are available.
> Distributor ID: Ubuntu
> Description:    Ubuntu 9.10
> Release:        9.10
> Codename:       karmic
>
> $ uname -m
> x86_64
>
> $ COLUMNS=40 dpkg -l | grep chef
> ii  chef           0.7.16-1       configuration management system written 
> in R
> ii  chef-indexer   0.7.16-1       Creates search indexes of Chef node 
> attribut
> ii  chef-server    0.7.16-1       Merb application providing centralized 
> manag
> ii  chef-server-sl 0.7.16-1       Merb app slice providing centralized 
> managem
> ii  libchef-ruby   0.7.16-1       Ruby libraries for Chef configuration 
> manage
> ii  libchef-ruby1. 0.7.16-1       Ruby 1.8 libraries for Chef configuration 
> ma
>
> Looking closer the increase in RAM usage is not sudden.  I was looking at a 
> chart of "free memory" *blush* so the "spikes" were really reboots 
> resulting in sudden increase of RAM.
>
> However, the merb worker was using an rss of ~2.3 GiB
>
> $ ps vax --sort=-rss |head
>  PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
> 29513 ?        R    2192:16 1141997   2 2549253 2352616 79.9 merb : worker 
> (port 4000)
>  9882 ?        S     39:31  31673     2 172461 111188  3.7 ruby 
> /usr/bin/chef-indexer -d -c /etc/chef/indexer.rb
> 29514 ?        S     20:39  55304     2 132805 58284  1.9 merb : worker 
> (port 4001)
> 29512 ?        S    811:15  11118     2 104457 29028  0.9 merb : spawner 
> (ports 4000)
>
> Since the reboot the rss of the restarted merb worker has climbed steadily 
> from about 185 MiB to 320 MiB
>
> $ ps vax --sort=-rss |head -n 5
>  PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
>  1728 ?        S    105:02      1     2 400393 322248  6.6 merb : worker 
> (port 4000)
>  1613 ?        S      0:12      0     2 128161 78968  1.6 /usr/bin/ruby1.8 
> /usr/bin/stompserver -C /etc/stompserver.conf
>  1729 ?        S      1:08      0     2 109313 36864  0.7 merb : worker 
> (port 4001)
>  1727 ?        S     41:55      3     2 105145 32532  0.6 merb : spawner 
> (ports 4000)
>
> Does it have a memory leak or is it merely a resource hog?
>
> Thanks,
> Ken
>
> --- On Wed, 2/3/10, Adam Jacob 
> <
 >
>  wrote:
>
>> From: Adam Jacob 
>> <
 >
>> Subject: [chef] Re: Sudden Increase in Merb RAM Use
>> To: 
>> "
 "
>>  
>> <
 >
>> Cc: 
>> "
 "
>>  
>> <
 >
>> Date: Wednesday, February 3, 2010, 11:16 AM
>> What's the frequency of the
>> spike?  Does it correlate to a particular set of nodes
>> checking in?
>> Just to be sure, this is on 0.7.x?
>>
>> Sent from my iPhone
>> On Feb 3, 2010, at 8:10 AM, Kenneth Stailey 
>> <
 >
>> wrote:
>>
>> Hi,
>>
>> merb RAM usage (rss) appears to stay flat for a long time
>> and then suddenly shoots up.
>>
>> After the RAM usage has spiked it looks like this:
>>
>> $ ps vax --sort=-rss |head
>>   PID TTY
>> STAT   TIME  MAJFL
>> TRS   DRS   RSS %MEM COMMAND
>> 29513 ?
>> R    2192:16 1141997   2 2549253
>> 2352616 79.9 merb : worker (port
>>
>> 4000)
>>
>>  9882 ?
>>  S     39:31
>> 31673     2 172461 111188  3.7 ruby
>> /usr/bin/chef-indexer -d -c /etc/chef/indexer.rb
>> 29514 ?
>> S     20:39
>> 55304     2 132805 58284  1.9 merb
>> : worker (port
>> 4001)
>>
>> 29512 ?
>> S    811:15
>> 11118     2 104457 29028  0.9 merb
>> : spawner (ports
>>
>> 4000)
>>
>> 10213 ?
>> S      3:37
>> 119973     2 301561 18080  0.6
>> /usr/bin/ruby1.8 /usr/bin/stompserver -C
>> /etc/stompserver.conf
>> 15666 ?
>> Sl     0:15   1473  1617
>> 111930 10768  0.3
>> /usr/lib/erlang/erts-5.7.2/bin/beam.smp -Bd -K true -- -root
>> /usr/lib/erlang -progname erl -- -home /var/lib/couchdb
>> -noshell -noinput -smp auto -sasl errlog_type error -pa
>> /usr/lib/couchdb/erlang/lib/couch-0.10.0/ebin
>>  /usr/lib/couchdb/erlang/lib/mochiweb-r97/ebin
>> /usr/lib/couchdb/erlang/lib/ibrowse-1.5.2/ebin
>> /usr/lib/couchdb/erlang/lib/erlang-oauth/ebin -eval
>> application:load(ibrowse) -eval application:load(oauth)
>> -eval application:load(crypto) -eval application:load(couch)
>> -eval crypto:start() -eval ssl:start() -eval ibrowse:start()
>> -eval couch_server:start([
>> "/etc/couchdb/default.ini",
>> "/etc/couchdb/local.ini",
>> "/etc/couchdb/default.ini",
>> "/etc/couchdb/local.ini"]), receive done ->
>> done end. -pidfile /var/run/couchdb/couchdb.pid -heart
>>
>> Bumped up VM that is running Chef server from 3 GB RAM to 5
>> GB RAM.
>>
>> After the reboot merb is only using about 185 MiB rss vs.
>> 2.3 GiB
>>
>> $ ps vax --sort=-rss |head
>>   PID TTY
>> STAT   TIME  MAJFL
>> TRS   DRS   RSS %MEM COMMAND
>> 29513 ?
>> R    2192:16
>>  1141997   2 2549253 2352616 79.9 merb : worker
>> (port
>> 4000)
>>
>>  9882 ?
>> S     39:31
>> 31673     2 172461 111188  3.7 ruby
>> /usr/bin/chef-indexer -d -c /etc/chef/indexer.rb
>> 29514 ?
>> S     20:39
>> 55304     2 132805 58284  1.9 merb
>> : worker (port
>>
>> 4001)
>>
>> 29512 ?
>> S    811:15
>> 11118     2 104457 29028  0.9 merb
>> : spawner (ports
>> 4000)
>>
>> 10213
>>  ?
>> S      3:37
>> 119973     2 301561 18080  0.6
>> /usr/bin/ruby1.8 /usr/bin/stompserver -C
>> /etc/stompserver.conf
>> 15666 ?
>> Sl     0:15   1473  1617
>> 111930 10768  0.3
>> /usr/lib/erlang/erts-5.7.2/bin/beam.smp -Bd -K true -- -root
>> /usr/lib/erlang -progname erl -- -home /var/lib/couchdb
>> -noshell -noinput -smp auto -sasl errlog_type error -pa
>> /usr/lib/couchdb/erlang/lib/couch-0.10.0/ebin
>> /usr/lib/couchdb/erlang/lib/mochiweb-r97/ebin
>> /usr/lib/couchdb/erlang/lib/ibrowse-1.5.2/ebin
>> /usr/lib/couchdb/erlang/lib/erlang-oauth/ebin -eval
>> application:load(ibrowse) -eval application:load(oauth)
>> -eval application:load(crypto) -eval application:load(couch)
>> -eval crypto:start() -eval ssl:start() -eval ibrowse:start()
>> -eval couch_server:start([
>> "/etc/couchdb/default.ini",
>> "/etc/couchdb/local.ini",
>>  "/etc/couchdb/default.ini",
>> "/etc/couchdb/local.ini"]), receive done ->
>> done end. -pidfile /var/run/couchdb/couchdb.pid -heart
>>
>> We are keeping charts of RAM use over time via Nagios/RRD
>> and they show the increase in RAM usage happens suddenly not
>> gradually.  It also repeats over time.
>>
>> Do other people see this issue?
>>
>> Thanks,
>> Ken
>>
>>
>



-- 
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: