May 06, 2011

Resque Worker Management with God

God is a process management tool, which can watch your Resque workers, ensuring they're running and spawn additional workers as necessary in the background.

We install God as below:

$ sudo gem install god

God uses configuration files for each of the processes it monitors. The way we're going to set it up is to have a single directory, /etc/god, containing all of the scripts we want to be running. Any file named .god will be run by the initialisation script we create later.

$ mkdir /etc/god

Resque comes with an example configuration file which is a good basis for writing your own app-specific configuration.

When you've got your configuration file, you can run God with the following command:

$ rvmsudo god -c /etc/god/resque.god -D

Running God Automatically

In a production environment, you'll want God to be running all of the time on system boot. We will need an init script to get this working, in /etc/init.d/god:

#!/bin/bash
#
# God
#
# chkconfig: - 99 1
# description: start, stop, restart God (bet you feel powerful)
#

# source function library
. /etc/rc.d/init.d/functions

RUBY_PATH="/usr/bin/ruby"
PATH=$RUBY_PATH:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON="/usr/bin/god"
PIDFILE=/var/run/god.pid
LOGFILE=/var/log/god.log
SCRIPTNAME=/etc/init.d/god
CONFIGFILEDIR=/etc/god

#DEBUG_OPTIONS="--log-level debug"
DEBUG_OPTIONS=""

# Gracefully exit if 'god' gem is not available.
test -x $DAEMON || exit 0

RETVAL=0

god_start() {
  start_cmd="$DAEMON -l $LOGFILE -P $PIDFILE $DEBUG_OPTIONS"
  #stop_cmd="kill -QUIT `cat $PIDFILE`"
  echo $start_cmd
  $start_cmd || echo -en "god already running"
  RETVAL=$?
  if [ "$RETVAL" == '0' ]; then
    sleep 2 # wait for server to load before loading config files
    if [ -d $CONFIGFILEDIR ]; then
      for file in `ls -1 $CONFIGFILEDIR/*.god`; do
        echo "god: loading $file ..."
        $DAEMON load $file
      done
    fi
  fi
  return $RETVAL
}

god_stop() {
  stop_cmd="god terminate"
  echo $stop_cmd
  $stop_cmd || echo -en "god not running"
}

case "$1" in
  start)
    god_start
    RETVAL=$?
    ;;
  stop)
    god_stop
    RETVAL=$?
    ;;
  restart)
    god_stop
    god_start
    RETVAL=$?
    ;;
  status)
    $DAEMON status
    RETVAL=$?
    ;;
  *)
    echo "Usage: god {start|stop|restart|status}"
    exit 1
    ;;
esac

exit $RETVAL

Make sure you change the paths at the beginning of the file to match your server's environment. To install the script, issue the following commands:

$ sudo chmod +x /etc/init.d/god
$ sudo touch /var/log/god.log
$ sudo update-rc.d -f god defaults

Congratulations; your workers are now fully managed by God!