On Thu, 2 Jun 2011 13:31:04 -0400, Nate Coraor <nate@bx.psu.edu> wrote:
Peter Cock wrote:
On Thu, Jun 2, 2011 at 6:23 PM, Nate Coraor <nate@bx.psu.edu> wrote:
Peter Cock wrote:
On a related point, I've previously suggested a $variable could be defined for use in tool XML wrappers to set the number of threads. This number could come from a general configuration file, or perhaps via the cluster settings - the point is the tool doesn't need to know, it just gets told how many threads it is allowed.
Yeah, that's the idea.
job_conf.xml:
<?xml version="1.0"?> <job_conf> <destinations> <destination name="default" runner="local"/> <destination name="pbs_default" runner="pbs"> <native_param name="queue">batch</native_param> </destination> </destinations> <tools> <tool id="upload1" destination="default"/> <tool id="bowtie_wrapper" destination="pbs_default"> <resource type="cores">8</resource> </tool> </tools> </job_conf>
pbs.py then knows to translate '<resource type="cores">8</resource>' to '-l nodes=1:ppn=8'.
Your tool can access that value a bunch, like .cores.
The same should be possible for other consumables.
Sounds good. Would there be a global default setting somewhere in universe.ini or elsewhere for when job_conf.xml didn't set a value?
default_cluster_job_runner will remain for backwards compatibility, but we'll ship a sample job_conf.xml that runs everything locally by default.
--nate
Haha, and I did that before realizing I could do just what I needed by writing tool-specific pbs:// URLs at the end of the config file... I'm such an idiot. But I really like what you did of it and I have a couple of questions. Concerning the single-threaded tools, what would happen if the number of threads set in the xml file was >1 ? Could it be possible to forbid a tool to run on a given node? Thanks, L-A
Peter