[chef] Re: Upstart problem with MongoDB recipe...

[Sam Pointer < >]

Hi Russell,

Whilst not a complete answer I realize, in my experience Upstart loosing track of process state is generally down to its insistence on fork following rather than pid files as a way of keeping track of what it thinks is running or not. The number of forks it expects to see is determined by the `expect fork|daemon` definition. 

The unfortunate thing about Upstart is that it only gives you two options, which causes problems with daemons who fork multiple times to really-really be detached from the parent. I guess it depends where you sit on the pragmatic/idealistic line as to whether this is a good thing or not.

My inclination would be to investigate exactly how many forks are involved starting and stopping the daemon via `service` and to see if they match the `expect` definition in the configuration file. The Upstart documentation has a good introduction to the strace incantations you can use to count forks: http://upstart.ubuntu.com/cookbook/#how-to-establish-fork-count. 

Sam Pointer

Lead Consulant

www.opsunit.com

On 3 October 2013 21:53, Russell Bateman < " target="_blank"> > wrote:

[This is a resend because it looks like it never made it to the mail list. I apologize if this is a double post.]

I have this MongoDB recipe that's working pretty well, but suffers from a problem that I've not been able to solve. The basic question is why does upstart stuff not work at the end of the recipe nor when the node(s) is(are) rebooted?

My use of upstart mechanisms is identical to what comes out of the MongoDB package and to other MongoDB community recipes. Or so I think.

First, particulars: Ubuntu 12.04 (LTS) everywhere, Chef server and clients (11.x).

The Chef recipe appears to work and finish perfectly, but upstart won't start the service. What I'm seeing is:

1. upstart service says MongoDB daemon not launched (mongodb stop/waiting)
2. performing service mongodb start by hand does not start it (mongodb stop/waiting)
3. executing the start-stop-daemon by hand just as in /etc/init/mongodb.conf will launch the daemon
4. once launched, upstart doesn't see it as the service is running (mongodb stop/waiting)
5. processor status (ps) shows the daemon is running

Is #4 something to do with pid?

I hope someone sees what I've done wrong here. I very much appreciate the help.

Russ

Console scrape illustrating the list above, from end of Chef run:

.
.
.
[2013-10-03T18:33:59+00:00] INFO: template[/data/mongodb/mongodb.conf] sending restart action to service[mongodb] (immediate)
  * service[mongodb] action restart[2013-10-03T18:33:59+00:00] INFO: Processing service[mongodb] action restart (mongodb::replica line 23)
[2013-10-03T18:33:59+00:00] INFO: service[mongodb] restarted

    - restart service service[mongodb]

  * service[mongodb] action enable[2013-10-03T18:33:59+00:00] INFO: Processing service[mongodb] action enable (mongodb::replica line 42)
 (up to date)
  * service[mongodb] action start[2013-10-03T18:33:59+00:00] INFO: Processing service[mongodb] action start (mongodb::replica line 42)
[2013-10-03T18:33:59+00:00] INFO: service[mongodb] started

    - start service service[mongodb]

[2013-10-03T18:33:59+00:00] INFO: Chef Run complete in 5.902176274 seconds
[2013-10-03T18:33:59+00:00] INFO: Running report handlers
[2013-10-03T18:33:59+00:00] INFO: Report handlers complete
Chef Client finished, 7 resources updated
:~# service mongodb status
mongodb stop/waiting
:~# service mongodb start
mongodb stop/waiting
:~# cat /etc/init/mongodb.conf
description "Keeps MongoDB running between boots"
limit nofile 20000 20000
kill timeout 300         # wait 300s between SIGTERM and SIGKILL.
start on runlevel [2345]
stop on runlevel [06]
expect fork
script
  ENABLE_MONGODB="yes"
  PIDFILE=/var/run/mongodb.pid
  if [ -f /etc/default/mongodb ]; then . /etc/default/mongodb; fi
  if [ "x$ENABLE_MONGODB" = "xyes" ]; then
    exec start-stop-daemon --start --quiet --chuid mongodb -m --pidfile $PIDFILE \
        --exec /usr/bin/mongod -- --config /data/mongodb/mongodb.conf
  fi
end script
:~# PIDFILE=/var/run/mongodb.pid
:~# exec start-stop-daemon --start --chuid mongodb -m --pidfile $PIDFILE --exec /usr/bin/mongod -- --config /data/mongodb/mongodb.conf
about to fork child process, waiting until server is ready for connections.
forked process: 3969
all output going to: /data/mongodb/mongodb.log
child process started successfully, parent exiting
:~$ service mongodb status
mongodb stop/waiting
:~$ ps -ef | grep [m]ongo
mongodb 3969  1  0 18:35 ?  00:00:00 /usr/bin/mongod --config /data/mongodb/mongodb.conf