[chef] Re: Re: Re: Re: Re: Sudden Increase in Merb RAM Use

[Adam Jacob < >]

Next step here is to get this running under something that can give us
some memory profiling - may I recommend bleak_house?

http://blog.evanweaver.com/files/doc/fauna/bleak_house/files/README.html

Is anyone else seeing this kind of growth on 64bit?

It's "normal" to see ruby use roughly double the memory on 64bit
systems, at least in my experience. We tend to see roughly 50-60MB
resident across the board on 32bit, and it's pretty stable... but we
don't have any production systems at 9.10.

Adam

On Wed, Feb 3, 2010 at 2:25 PM, Kenneth Stailey 
<
 >
 wrote:
> Hi Adam,
>
> $ dpkg -l | grep "ii  ruby" | awk '{print $2 " " $3 }'
> ruby 4.2
> ruby1.8 1.8.7.174-1ubuntu1
> rubygems 1.3.5-1ubuntu2
> rubygems1.8 1.3.5-1ubuntu2
>
> $ ruby --version
> ruby 1.8.7 (2009-06-12 patchlevel 174) [x86_64-linux]
>
> Actually looking at the charts shows a gradual decline in free RAM for a 
> while (hours and hours) and then a sudden decline.
>
> There are 107 chef clients which go off twice an hour but not all at once 
> since they use splayed cron jobs.  The splay is done by using a modulo of 
> the last octet of the IP address not by any Chef splaying mechanism.  This 
> yields an average of 2-3 a minute that check in nearly simultaneously, 
> maybe a few seconds apart.
>
> The Chef server is running in a VM which is not hosting any other major 
> applications.  It has sshd etc.
>
> Thanks,
> Ken
>
> --- On Wed, 2/3/10, Adam Jacob 
> <
 >
>  wrote:
>
>> From: Adam Jacob 
>> <
 >
>> Subject: [chef] Re: Re: Re: Sudden Increase in Merb RAM Use
>> To: 
>> 
 
>> Date: Wednesday, February 3, 2010, 5:06 PM
>> Well, it certainly smells like a leak
>> - I'm not seeing this behavior,
>> though, and we haven't had any other reports of this kind
>> of growth
>> with the chef server.  I'm also only running 32bit
>> ruby, which may be
>> something.
>>
>> My next request will be to line up the growth in your
>> graphs with the
>> timing of particular servers checking in.
>>
>> If nothing obvious comes from that, it'll be time to start
>> talking
>> about breaking out a debugger and seeing what's what.
>>
>> Can you send the version of ruby your running along as
>> well?
>>
>> Best,
>> Adam
>>
>> On Wed, Feb 3, 2010 at 1:58 PM, Kenneth Stailey 
>> <
 >
>> wrote:
>> > Particulars:
>> >
>> > $ lsb_release -a
>> > No LSB modules are available.
>> > Distributor ID: Ubuntu
>> > Description:    Ubuntu 9.10
>> > Release:        9.10
>> > Codename:       karmic
>> >
>> > $ uname -m
>> > x86_64
>> >
>> > $ COLUMNS=40 dpkg -l | grep chef
>> > ii  chef           0.7.16-1
>> configuration management system written in R
>> > ii  chef-indexer   0.7.16-1       Creates search
>> indexes of Chef node attribut
>> > ii  chef-server    0.7.16-1       Merb
>> application providing centralized manag
>> > ii  chef-server-sl 0.7.16-1       Merb app slice
>> providing centralized managem
>> > ii  libchef-ruby   0.7.16-1       Ruby libraries
>> for Chef configuration manage
>> > ii  libchef-ruby1. 0.7.16-1       Ruby 1.8
>> libraries for Chef configuration ma
>> >
>> > Looking closer the increase in RAM usage is not
>> sudden.  I was looking at a chart of "free memory" *blush*
>> so the "spikes" were really reboots resulting in sudden
>> increase of RAM.
>> >
>> > However, the merb worker was using an rss of ~2.3 GiB
>> >
>> > $ ps vax --sort=-rss |head
>> >  PID TTY      STAT   TIME  MAJFL   TRS   DRS
>>   RSS %MEM COMMAND
>> > 29513 ?        R    2192:16 1141997   2 2549253
>> 2352616 79.9 merb : worker (port 4000)
>> >  9882 ?        S     39:31  31673     2
>> 172461 111188  3.7 ruby /usr/bin/chef-indexer -d -c
>> /etc/chef/indexer.rb
>> > 29514 ?        S     20:39  55304     2
>> 132805 58284  1.9 merb : worker (port 4001)
>> > 29512 ?        S    811:15  11118     2
>> 104457 29028  0.9 merb : spawner (ports 4000)
>> >
>> > Since the reboot the rss of the restarted merb worker
>> has climbed steadily from about 185 MiB to 320 MiB
>> >
>> > $ ps vax --sort=-rss |head -n 5
>> >  PID TTY      STAT   TIME  MAJFL   TRS   DRS
>>   RSS %MEM COMMAND
>> >  1728 ?        S    105:02      1     2
>> 400393 322248  6.6 merb : worker (port 4000)
>> >  1613 ?        S      0:12      0     2
>> 128161 78968  1.6 /usr/bin/ruby1.8 /usr/bin/stompserver -C
>> /etc/stompserver.conf
>> >  1729 ?        S      1:08      0     2
>> 109313 36864  0.7 merb : worker (port 4001)
>> >  1727 ?        S     41:55      3     2
>> 105145 32532  0.6 merb : spawner (ports 4000)
>> >
>> > Does it have a memory leak or is it merely a resource
>> hog?
>> >
>> > Thanks,
>> > Ken
>> >
>> > --- On Wed, 2/3/10, Adam Jacob 
>> > <
 >
>> wrote:
>> >
>> >> From: Adam Jacob 
>> >> <
 >
>> >> Subject: [chef] Re: Sudden Increase in Merb RAM
>> Use
>> >> To: 
>> >> "
 "
>> <
 >
>> >> Cc: 
>> >> "
 "
>> <
 >
>> >> Date: Wednesday, February 3, 2010, 11:16 AM
>> >> What's the frequency of the
>> >> spike?  Does it correlate to a particular set of
>> nodes
>> >> checking in?
>> >> Just to be sure, this is on 0.7.x?
>> >>
>> >> Sent from my iPhone
>> >> On Feb 3, 2010, at 8:10 AM, Kenneth Stailey 
>> >> <
 >
>> >> wrote:
>> >>
>> >> Hi,
>> >>
>> >> merb RAM usage (rss) appears to stay flat for a
>> long time
>> >> and then suddenly shoots up.
>> >>
>> >> After the RAM usage has spiked it looks like
>> this:
>> >>
>> >> $ ps vax --sort=-rss |head
>> >>   PID TTY
>> >> STAT   TIME  MAJFL
>> >> TRS   DRS   RSS %MEM COMMAND
>> >> 29513 ?
>> >> R    2192:16 1141997   2 2549253
>> >> 2352616 79.9 merb : worker (port
>> >>
>> >> 4000)
>> >>
>> >>  9882 ?
>> >>  S     39:31
>> >> 31673     2 172461 111188  3.7 ruby
>> >> /usr/bin/chef-indexer -d -c /etc/chef/indexer.rb
>> >> 29514 ?
>> >> S     20:39
>> >> 55304     2 132805 58284  1.9 merb
>> >> : worker (port
>> >> 4001)
>> >>
>> >> 29512 ?
>> >> S    811:15
>> >> 11118     2 104457 29028  0.9 merb
>> >> : spawner (ports
>> >>
>> >> 4000)
>> >>
>> >> 10213 ?
>> >> S      3:37
>> >> 119973     2 301561 18080  0.6
>> >> /usr/bin/ruby1.8 /usr/bin/stompserver -C
>> >> /etc/stompserver.conf
>> >> 15666 ?
>> >> Sl     0:15   1473  1617
>> >> 111930 10768  0.3
>> >> /usr/lib/erlang/erts-5.7.2/bin/beam.smp -Bd -K
>> true -- -root
>> >> /usr/lib/erlang -progname erl -- -home
>> /var/lib/couchdb
>> >> -noshell -noinput -smp auto -sasl errlog_type
>> error -pa
>> >> /usr/lib/couchdb/erlang/lib/couch-0.10.0/ebin
>> >>  /usr/lib/couchdb/erlang/lib/mochiweb-r97/ebin
>> >> /usr/lib/couchdb/erlang/lib/ibrowse-1.5.2/ebin
>> >> /usr/lib/couchdb/erlang/lib/erlang-oauth/ebin
>> -eval
>> >> application:load(ibrowse) -eval
>> application:load(oauth)
>> >> -eval application:load(crypto) -eval
>> application:load(couch)
>> >> -eval crypto:start() -eval ssl:start() -eval
>> ibrowse:start()
>> >> -eval couch_server:start([
>> >> "/etc/couchdb/default.ini",
>> >> "/etc/couchdb/local.ini",
>> >> "/etc/couchdb/default.ini",
>> >> "/etc/couchdb/local.ini"]), receive done ->
>> >> done end. -pidfile /var/run/couchdb/couchdb.pid
>> -heart
>> >>
>> >> Bumped up VM that is running Chef server from 3 GB
>> RAM to 5
>> >> GB RAM.
>> >>
>> >> After the reboot merb is only using about 185 MiB
>> rss vs.
>> >> 2.3 GiB
>> >>
>> >> $ ps vax --sort=-rss |head
>> >>   PID TTY
>> >> STAT   TIME  MAJFL
>> >> TRS   DRS   RSS %MEM COMMAND
>> >> 29513 ?
>> >> R    2192:16
>> >>  1141997   2 2549253 2352616 79.9 merb :
>> worker
>> >> (port
>> >> 4000)
>> >>
>> >>  9882 ?
>> >> S     39:31
>> >> 31673     2 172461 111188  3.7 ruby
>> >> /usr/bin/chef-indexer -d -c /etc/chef/indexer.rb
>> >> 29514 ?
>> >> S     20:39
>> >> 55304     2 132805 58284  1.9 merb
>> >> : worker (port
>> >>
>> >> 4001)
>> >>
>> >> 29512 ?
>> >> S    811:15
>> >> 11118     2 104457 29028  0.9 merb
>> >> : spawner (ports
>> >> 4000)
>> >>
>> >> 10213
>> >>  ?
>> >> S      3:37
>> >> 119973     2 301561 18080  0.6
>> >> /usr/bin/ruby1.8 /usr/bin/stompserver -C
>> >> /etc/stompserver.conf
>> >> 15666 ?
>> >> Sl     0:15   1473  1617
>> >> 111930 10768  0.3
>> >> /usr/lib/erlang/erts-5.7.2/bin/beam.smp -Bd -K
>> true -- -root
>> >> /usr/lib/erlang -progname erl -- -home
>> /var/lib/couchdb
>> >> -noshell -noinput -smp auto -sasl errlog_type
>> error -pa
>> >> /usr/lib/couchdb/erlang/lib/couch-0.10.0/ebin
>> >> /usr/lib/couchdb/erlang/lib/mochiweb-r97/ebin
>> >> /usr/lib/couchdb/erlang/lib/ibrowse-1.5.2/ebin
>> >> /usr/lib/couchdb/erlang/lib/erlang-oauth/ebin
>> -eval
>> >> application:load(ibrowse) -eval
>> application:load(oauth)
>> >> -eval application:load(crypto) -eval
>> application:load(couch)
>> >> -eval crypto:start() -eval ssl:start() -eval
>> ibrowse:start()
>> >> -eval couch_server:start([
>> >> "/etc/couchdb/default.ini",
>> >> "/etc/couchdb/local.ini",
>> >>  "/etc/couchdb/default.ini",
>> >> "/etc/couchdb/local.ini"]), receive done ->
>> >> done end. -pidfile /var/run/couchdb/couchdb.pid
>> -heart
>> >>
>> >> We are keeping charts of RAM use over time via
>> Nagios/RRD
>> >> and they show the increase in RAM usage happens
>> suddenly not
>> >> gradually.  It also repeats over time.
>> >>
>> >> Do other people see this issue?
>> >>
>> >> Thanks,
>> >> Ken
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Opscode, Inc.
>> Adam Jacob, CTO
>> T: (206) 508-7449 E: 
>> 
 
>>
>



-- 
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: