[chef] Re: Re: Re: Re: Sudden Increase in Merb RAM Use

[Kenneth Stailey < >]

Hi Adam,

$ dpkg -l | grep "ii  ruby" | awk '{print $2 " " $3 }'
ruby 4.2
ruby1.8 1.8.7.174-1ubuntu1
rubygems 1.3.5-1ubuntu2
rubygems1.8 1.3.5-1ubuntu2

$ ruby --version
ruby 1.8.7 (2009-06-12 patchlevel 174) [x86_64-linux]

Actually looking at the charts shows a gradual decline in free RAM for a 
while (hours and hours) and then a sudden decline.  

There are 107 chef clients which go off twice an hour but not all at once 
since they use splayed cron jobs.  The splay is done by using a modulo of the 
last octet of the IP address not by any Chef splaying mechanism.  This yields 
an average of 2-3 a minute that check in nearly simultaneously, maybe a few 
seconds apart.

The Chef server is running in a VM which is not hosting any other major 
applications.  It has sshd etc.

Thanks,
Ken

--- On Wed, 2/3/10, Adam Jacob 
<
 >
 wrote:

> From: Adam Jacob 
> <
 >
> Subject: [chef] Re: Re: Re: Sudden Increase in Merb RAM Use
> To: 
> 
 
> Date: Wednesday, February 3, 2010, 5:06 PM
> Well, it certainly smells like a leak
> - I'm not seeing this behavior,
> though, and we haven't had any other reports of this kind
> of growth
> with the chef server.  I'm also only running 32bit
> ruby, which may be
> something.
> 
> My next request will be to line up the growth in your
> graphs with the
> timing of particular servers checking in.
> 
> If nothing obvious comes from that, it'll be time to start
> talking
> about breaking out a debugger and seeing what's what.
> 
> Can you send the version of ruby your running along as
> well?
> 
> Best,
> Adam
> 
> On Wed, Feb 3, 2010 at 1:58 PM, Kenneth Stailey 
> <
 >
> wrote:
> > Particulars:
> >
> > $ lsb_release -a
> > No LSB modules are available.
> > Distributor ID: Ubuntu
> > Description:    Ubuntu 9.10
> > Release:        9.10
> > Codename:       karmic
> >
> > $ uname -m
> > x86_64
> >
> > $ COLUMNS=40 dpkg -l | grep chef
> > ii  chef           0.7.16-1      
> configuration management system written in R
> > ii  chef-indexer   0.7.16-1       Creates search
> indexes of Chef node attribut
> > ii  chef-server    0.7.16-1       Merb
> application providing centralized manag
> > ii  chef-server-sl 0.7.16-1       Merb app slice
> providing centralized managem
> > ii  libchef-ruby   0.7.16-1       Ruby libraries
> for Chef configuration manage
> > ii  libchef-ruby1. 0.7.16-1       Ruby 1.8
> libraries for Chef configuration ma
> >
> > Looking closer the increase in RAM usage is not
> sudden.  I was looking at a chart of "free memory" *blush*
> so the "spikes" were really reboots resulting in sudden
> increase of RAM.
> >
> > However, the merb worker was using an rss of ~2.3 GiB
> >
> > $ ps vax --sort=-rss |head
> >  PID TTY      STAT   TIME  MAJFL   TRS   DRS
>   RSS %MEM COMMAND
> > 29513 ?        R    2192:16 1141997   2 2549253
> 2352616 79.9 merb : worker (port 4000)
> >  9882 ?        S     39:31  31673     2
> 172461 111188  3.7 ruby /usr/bin/chef-indexer -d -c
> /etc/chef/indexer.rb
> > 29514 ?        S     20:39  55304     2
> 132805 58284  1.9 merb : worker (port 4001)
> > 29512 ?        S    811:15  11118     2
> 104457 29028  0.9 merb : spawner (ports 4000)
> >
> > Since the reboot the rss of the restarted merb worker
> has climbed steadily from about 185 MiB to 320 MiB
> >
> > $ ps vax --sort=-rss |head -n 5
> >  PID TTY      STAT   TIME  MAJFL   TRS   DRS
>   RSS %MEM COMMAND
> >  1728 ?        S    105:02      1     2
> 400393 322248  6.6 merb : worker (port 4000)
> >  1613 ?        S      0:12      0     2
> 128161 78968  1.6 /usr/bin/ruby1.8 /usr/bin/stompserver -C
> /etc/stompserver.conf
> >  1729 ?        S      1:08      0     2
> 109313 36864  0.7 merb : worker (port 4001)
> >  1727 ?        S     41:55      3     2
> 105145 32532  0.6 merb : spawner (ports 4000)
> >
> > Does it have a memory leak or is it merely a resource
> hog?
> >
> > Thanks,
> > Ken
> >
> > --- On Wed, 2/3/10, Adam Jacob 
> > <
 >
> wrote:
> >
> >> From: Adam Jacob 
> >> <
 >
> >> Subject: [chef] Re: Sudden Increase in Merb RAM
> Use
> >> To: 
> >> "
 "
> <
 >
> >> Cc: 
> >> "
 "
> <
 >
> >> Date: Wednesday, February 3, 2010, 11:16 AM
> >> What's the frequency of the
> >> spike?  Does it correlate to a particular set of
> nodes
> >> checking in?
> >> Just to be sure, this is on 0.7.x?
> >>
> >> Sent from my iPhone
> >> On Feb 3, 2010, at 8:10 AM, Kenneth Stailey 
> >> <
 >
> >> wrote:
> >>
> >> Hi,
> >>
> >> merb RAM usage (rss) appears to stay flat for a
> long time
> >> and then suddenly shoots up.
> >>
> >> After the RAM usage has spiked it looks like
> this:
> >>
> >> $ ps vax --sort=-rss |head
> >>   PID TTY
> >> STAT   TIME  MAJFL
> >> TRS   DRS   RSS %MEM COMMAND
> >> 29513 ?
> >> R    2192:16 1141997   2 2549253
> >> 2352616 79.9 merb : worker (port
> >>
> >> 4000)
> >>
> >>  9882 ?
> >>  S     39:31
> >> 31673     2 172461 111188  3.7 ruby
> >> /usr/bin/chef-indexer -d -c /etc/chef/indexer.rb
> >> 29514 ?
> >> S     20:39
> >> 55304     2 132805 58284  1.9 merb
> >> : worker (port
> >> 4001)
> >>
> >> 29512 ?
> >> S    811:15
> >> 11118     2 104457 29028  0.9 merb
> >> : spawner (ports
> >>
> >> 4000)
> >>
> >> 10213 ?
> >> S      3:37
> >> 119973     2 301561 18080  0.6
> >> /usr/bin/ruby1.8 /usr/bin/stompserver -C
> >> /etc/stompserver.conf
> >> 15666 ?
> >> Sl     0:15   1473  1617
> >> 111930 10768  0.3
> >> /usr/lib/erlang/erts-5.7.2/bin/beam.smp -Bd -K
> true -- -root
> >> /usr/lib/erlang -progname erl -- -home
> /var/lib/couchdb
> >> -noshell -noinput -smp auto -sasl errlog_type
> error -pa
> >> /usr/lib/couchdb/erlang/lib/couch-0.10.0/ebin
> >>  /usr/lib/couchdb/erlang/lib/mochiweb-r97/ebin
> >> /usr/lib/couchdb/erlang/lib/ibrowse-1.5.2/ebin
> >> /usr/lib/couchdb/erlang/lib/erlang-oauth/ebin
> -eval
> >> application:load(ibrowse) -eval
> application:load(oauth)
> >> -eval application:load(crypto) -eval
> application:load(couch)
> >> -eval crypto:start() -eval ssl:start() -eval
> ibrowse:start()
> >> -eval couch_server:start([
> >> "/etc/couchdb/default.ini",
> >> "/etc/couchdb/local.ini",
> >> "/etc/couchdb/default.ini",
> >> "/etc/couchdb/local.ini"]), receive done ->
> >> done end. -pidfile /var/run/couchdb/couchdb.pid
> -heart
> >>
> >> Bumped up VM that is running Chef server from 3 GB
> RAM to 5
> >> GB RAM.
> >>
> >> After the reboot merb is only using about 185 MiB
> rss vs.
> >> 2.3 GiB
> >>
> >> $ ps vax --sort=-rss |head
> >>   PID TTY
> >> STAT   TIME  MAJFL
> >> TRS   DRS   RSS %MEM COMMAND
> >> 29513 ?
> >> R    2192:16
> >>  1141997   2 2549253 2352616 79.9 merb :
> worker
> >> (port
> >> 4000)
> >>
> >>  9882 ?
> >> S     39:31
> >> 31673     2 172461 111188  3.7 ruby
> >> /usr/bin/chef-indexer -d -c /etc/chef/indexer.rb
> >> 29514 ?
> >> S     20:39
> >> 55304     2 132805 58284  1.9 merb
> >> : worker (port
> >>
> >> 4001)
> >>
> >> 29512 ?
> >> S    811:15
> >> 11118     2 104457 29028  0.9 merb
> >> : spawner (ports
> >> 4000)
> >>
> >> 10213
> >>  ?
> >> S      3:37
> >> 119973     2 301561 18080  0.6
> >> /usr/bin/ruby1.8 /usr/bin/stompserver -C
> >> /etc/stompserver.conf
> >> 15666 ?
> >> Sl     0:15   1473  1617
> >> 111930 10768  0.3
> >> /usr/lib/erlang/erts-5.7.2/bin/beam.smp -Bd -K
> true -- -root
> >> /usr/lib/erlang -progname erl -- -home
> /var/lib/couchdb
> >> -noshell -noinput -smp auto -sasl errlog_type
> error -pa
> >> /usr/lib/couchdb/erlang/lib/couch-0.10.0/ebin
> >> /usr/lib/couchdb/erlang/lib/mochiweb-r97/ebin
> >> /usr/lib/couchdb/erlang/lib/ibrowse-1.5.2/ebin
> >> /usr/lib/couchdb/erlang/lib/erlang-oauth/ebin
> -eval
> >> application:load(ibrowse) -eval
> application:load(oauth)
> >> -eval application:load(crypto) -eval
> application:load(couch)
> >> -eval crypto:start() -eval ssl:start() -eval
> ibrowse:start()
> >> -eval couch_server:start([
> >> "/etc/couchdb/default.ini",
> >> "/etc/couchdb/local.ini",
> >>  "/etc/couchdb/default.ini",
> >> "/etc/couchdb/local.ini"]), receive done ->
> >> done end. -pidfile /var/run/couchdb/couchdb.pid
> -heart
> >>
> >> We are keeping charts of RAM use over time via
> Nagios/RRD
> >> and they show the increase in RAM usage happens
> suddenly not
> >> gradually.  It also repeats over time.
> >>
> >> Do other people see this issue?
> >>
> >> Thanks,
> >> Ken
> >>
> >>
> >
> 
> 
> 
> -- 
> Opscode, Inc.
> Adam Jacob, CTO
> T: (206) 508-7449 E: 
> 
 
>