python job manager
Hi, I run galaxy on a large SMP university machine. The machine is used by some folks for command line work, and others via galaxy. I was wondering if anyone had integrated into galaxy a job manager that monitors CPU load averages, and only runs new jobs when cpu resources are available? Thanks, James
On Wed, Mar 16, 2011 at 10:46:54AM -0400, James Lindsay wrote:
Hi, I run galaxy on a large SMP university machine. The machine is used by some folks for command line work, and others via galaxy. I was wondering if anyone had integrated into galaxy a job manager that monitors CPU load averages, and only runs new jobs when cpu resources are available?
James, you could achieve this using the galaxy cluster configurations. Even with a single (SMP) machine in use there's value in creating (for example) a torque queue on that machine and using Galaxy's torque support to have it submit jobs to that queue. Your non-Galaxy command line users can then also use the 'qsub' command to launch their jobs, and Torque will be able to balance resources across them according to your preferences. Having your galaxy jobs use the pbs (torque) job runner has the additional benefit of being able to restart galaxy without the jobs losing their parent process and dying. https://bitbucket.org/galaxy/galaxy-central/wiki/Config/ProductionServer https://bitbucket.org/galaxy/galaxy-central/wiki/Config/Cluster -- Ry4an Brase 612-626-6575 Software Developer Application Development University of Minnesota Supercomputing Institute http://www.msi.umn.edu
James Lindsay wrote:
Hi, I run galaxy on a large SMP university machine. The machine is used by some folks for command line work, and others via galaxy. I was wondering if anyone had integrated into galaxy a job manager that monitors CPU load averages, and only runs new jobs when cpu resources are available?
Hi James, It's not a part of Galaxy, but you can do this by installing a cluster resource manager like TORQUE and Maui. Even though you're only running on a single machine, the jobs will submit to the local system and Maui can be used to define when jobs are allowed to run, based on load, memory usage, etc. --nate
Thanks, James ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Nate Coraor wrote:
James Lindsay wrote:
Hi, I run galaxy on a large SMP university machine. The machine is used by some folks for command line work, and others via galaxy. I was wondering if anyone had integrated into galaxy a job manager that monitors CPU load averages, and only runs new jobs when cpu resources are available?
Hi James,
It's not a part of Galaxy, but you can do this by installing a cluster resource manager like TORQUE and Maui. Even though you're only running on a single machine, the jobs will submit to the local system and Maui can be used to define when jobs are allowed to run, based on load, memory usage, etc.
Whoops, I should look ahead in my email... Thanks for answering this, Ry4an.
--nate
Thanks, James ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
participants (3)
-
James Lindsay -
Nate Coraor -
Ry4an Brase