Implement scaling and load balancing galaxy on a 8-core host with cluster backend
Hi guys, Today my institute ran a galaxy training with about 20 people. We didn't implement multiple instances for galaxy python. When everyone started submitting jobs, the thread pool filled up very quickly, soon after the galaxy threw worker queue full error then became unresponsive. It had to be restarted. So we decide to implement the multiple instances overnight in prepare for 2nd day of the training. Our setup is a 8-core host runs the galaxy server itself, and a 500-core cluster handles all the jobs. So I am wondering how I should distribute 8 cores for different roles (web, manager, job handler). In the wiki, the author said he runs six web server processes on 8 CPUs server, I assume he only runs one manager and one job handler. Is one job handler is more than enough? Even for the public galaxy? Second question is, it makes sense to have all web roles listen on a public IP (with different ports). For manager, job handler, can I just set them to listen 127.0.0.1? Or they have to be listening the same IP as the web roles? Regards, Derrick
On Jun 4, 2012, at 1:06 AM, Derrick Lin wrote:
Hi guys,
Today my institute ran a galaxy training with about 20 people. We didn't implement multiple instances for galaxy python. When everyone started submitting jobs, the thread pool filled up very quickly, soon after the galaxy threw worker queue full error then became unresponsive. It had to be restarted.
So we decide to implement the multiple instances overnight in prepare for 2nd day of the training. Our setup is a 8-core host runs the galaxy server itself, and a 500-core cluster handles all the jobs.
So I am wondering how I should distribute 8 cores for different roles (web, manager, job handler). In the wiki, the author said he runs six web server processes on 8 CPUs server, I assume he only runs one manager and one job handler. Is one job handler is more than enough? Even for the public galaxy?
Hi Derrick, We run 6 job handlers for usegalaxy.org: 3 for typical jobs, 2 for NGS tools, and 1 for jobs originating from Trackster. Also, the processes won't be on-CPU constantly, so you can safely run more processes than you have cores.
Second question is, it makes sense to have all web roles listen on a public IP (with different ports). For manager, job handler, can I just set them to listen 127.0.0.1? Or they have to be listening the same IP as the web roles?
They can listen to any available IP, but I leave them as localhost to prevent anyone from bypassing the proxy and accessing them directly. --nate
Regards, Derrick ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
participants (2)
-
Derrick Lin -
Nate Coraor