Upstart script to manage a multi instance load balanced installation - galaxy-dev

16 Jul 2013


      After some work i've created an Upstart script which can manage a load
balanced galaxy configuration as described in the wiki. I thought that I
would put it on this list for other people to use. The script parses
universe_wsgi.ini just like run.sh and spawns all of the servers it finds.
It comes in two pieces galaxy.conf and galaxy-worker.conf. Once you place
them both in /etc/init and make the proper edits for the environment a
server can be started with "sudo start galaxy". The configuration starts
the server at boot time and puts all of the instances under the management
of upstart which deals with pids, logging to syslog and respawning if an
instance crashes.
I have just gotten this working reasonably well but have done basically no
testing so there are bugs to be found. Any comments are welcome if anyone
knows a better way to do something here.
- Seth
*galaxy.conf*
----------------------------
author "Seth Sims seth.sims@gmail.com"
version "0.0.1 test"
description "galaxy master process. Fetches eggs and spawns all of the
servers it finds configured"
start on started network-services
# make sure that any eggs we download are at least owned by the galaxy
group.
# we cannot use setuid in this script because only root can issue the
"start galaxy-worker"
# command. But this way the galaxy instances should still be able to use
their eggs.
setgid galaxy
# put galaxy root directory here
chdir /srv/galaxy-dist/
pre-start script
    date
    echo "checking python version"
    python ./scripts/check_python.py
    [ $? -ne 0 ] && exit 1
echo "pre-fetching tossing out expired eggs"
    python ./scripts/check_eggs.py -q
    if [ $? -eq 0 ]; then
        echo "Some eggs are out of date, attempting to fetch..."
        python ./scripts/fetch_eggs.py
        if [ $? -eq 0 ]; then
            echo "Fetch Successful."
        else
            echo "Fetch failed."
        fi
    fi
echo "starting servers"
    SERVERS=`sed -n 's/^[server:(.*)]/\1/  p' universe_wsgi.ini | xargs
echo`
    for SERVER in ${SERVERS} ; do
        echo "starting server ${SERVER}"
        start galaxy-worker SERVER_NAME=${SERVER}
    done
end script
post-stop script
    SERVERS=`sed -n 's/^[server:(.*)]/\1/  p' universe_wsgi.ini | xargs
echo`
    date
    echo "stopping galaxy servers".
    for SERVER in ${SERVERS} ; do
        echo "stopping ${SERVER}"
        stop galaxy-worker SERVER_NAME=${SERVER}
    done
end script
---------------------------
*galaxy-worker*
author "Seth Sims seth.sims@gmail.com"
version "0.0.1 test"
description "Starts a galaxy server instance. This is run from the
galaxy.conf file"
instance $SERVER_NAME
#make sure we are running as the galaxy user
setuid galaxy
setgid galaxy
#put the galaxy root directory here
chdir /srv/galaxy-dist/
#having multiple instances of galaxy using the same egg directory was
causing a race
#condition that was stopping the instances from starting correctly. So give
each instance
#its own directory under /tmp
env PYTHON_EGG_CACHE=/tmp/${SERVER_NAME}_egg/
respawn
script
    exec python ./scripts/paster.py serve universe_wsgi.ini
--server-name=${SERVER_NAME}
end script