Well, you still can use my method, which I described at the beginning of the thread. But that means modifying some code.

If I'm not mistaken, Galaxy's built-in scheduler is a simple FIFO scheduler with no means to tune the needed resources. So if you set multithreaded tools, yeah I guess the nodes can expect surprises. That could happen with pbs too if you don't set the proper number of needed cpu per node / necessary amount of memory.

Or I missed something too.

Best,
L-A


Le 08/08/2011 21:07, Andrew Warren a écrit :
So would the current correct method for setting up multi-threaded jobs on a cluster be to specify custom runners in the [galaxy:tool_runners] section of the universe config file for EVERY tool that uses a multiple threads (assuming the default is set to one)?

For example, for the bowtie program and a queue named "galaxy":
bowtie = pbs:///galaxy/-l ppn=4,mem=16gb/

Is this currently the only way for galaxy to inform the queuing system how many threads a program will use?
And does this mean that without custom runners in the config file any muti-threaded program that has multiple instances in an asychronous workflow has the opportunity to overload a cluster node since the queuing system doesn't "know" how many threads the program will be using?

Just want to make sure I'm not missing out on the latest and greatest method for process management. :)

Thanks,
Andrew

Louise-Amélie Schmitt wrote:

> > 
> > default_cluster_job_runner will remain for backwards compatibility, but 
> > we'll ship a sample job_conf.xml that runs everything locally by 
> > default. 
> > 
> > --nate 
> 
> Haha, and I did that before realizing I could do just what I needed by 
> writing tool-specific pbs:// URLs at the end of the config file... I'm such 
> an idiot.

Haha, okay, I don't think i even noticed since I was distracted by your 
implementation being a step in the way we want to go with it. 

> But I really like what you did of it and I have a couple of questions. 
> 
> Concerning the single-threaded tools, what would happen if the number of 
> threads set in the xml file was >1 ? 

It'd consume extra slots, but the tool itself would just run as usual. 

> Could it be possible to forbid a tool to run on a given node? 

Hrm.  In PBS you could do it using node properties/neednodes or resource 
requirements.  I'd have to think a bit about how to do this in a more 
general way in the XML. 

--nate 

> 
> Thanks, 
> L-A 
> 
> 
> > 
> >> 
> >> Peter 
> >> 

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/