Kimberly Begley wrote:
Hello,
The load balancing page is showing up blank at http://bitbucket.org/galaxy/galaxy-central/wiki/Config/LoadBalancing for me.
Can you direct me to the correct URL for this info?
Cheers,
Kimberly
Hi Kimberly,
Sorry about that. I haven't actually had a chance to write that page yet. I've fixed the linking page to simply refer people to galaxy-dev for load balancing questions.
Since you're asking, here's what we do:
You can create multiple web processes but there must be only one job runner. The configuration for the web processes can live in the same config file (which I create as 'universe_wsgi.webapp.ini'), and looks like this:
[server:main_0] use = egg:Paste#http port = 9980 host = 127.0.0.1 use_threadpool = true threadpool_workers = 7
[server:main_1] use = egg:Paste#http port = 9981 host = 127.0.0.1 use_threadpool = true threadpool_workers = 7
[server:main_2] use = egg:Paste#http port = 9982 host = 127.0.0.1 use_threadpool = true threadpool_workers = 7
[app:main]
enable_job_running = False
Then in a separate config file('universe_wsgi.runner.ini') for the job runner, the following:
[server:main] use = egg:Paste#http port = 9979 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5
[app:main]
enable_job_running = True track_jobs_in_database = True
All other options can be set to your liking, enable_job_running and track_jobs_in_database are the important differences.
Once configured, multiple instances can be started with a script like the following:
% cat run_all.sh #!/bin/bash
cd `dirname $0`
for i in {0..2}; do echo "starting server $i" python -ES ./scripts/paster.py serve universe_wsgi.webapp.ini --server-name=main_$i --pid-file=main_$i.pid --log-file=log/main_$i.log --daemon done
echo "starting job runner" python -ES ./scripts/paster.py serve universe_wsgi.runner.ini --server-name=main --pid-file=main.pid --log-file=log/main.log --daemon
Likewise, a 'stop_all.sh' would be the same with --stop-daemon instead of --daemon.
Your proxy server must be configured to balance across all the instances. We don't have an Apache config for this since we use nginx for balancing, and the nginx config is rather simple:
http://bitbucket.org/galaxy/galaxy-central/wiki/Config/nginxProxy
It can be done with Apache, however.