Re: [galaxy-user] suggestion for multithreading
On Thu, 2 Jun 2011 13:31:04 -0400, Nate Coraor <nate(a)bx.psu.edu> wrote:
Peter Cock wrote:
> On Thu, Jun 2, 2011 at 6:23 PM, Nate Coraor <nate(a)bx.psu.edu> wrote:
> > Peter Cock wrote:
> >> On a related point, I've previously suggested a $variable could be
> >> defined
> >> for use in tool XML wrappers to set the number of threads. This
number
> >> could come from a general configuration file, or perhaps
via the
> >> cluster
> >> settings - the point is the tool doesn't need to know, it just gets
> >> told how
> >> many threads it is allowed.
> >
> > Yeah, that's the idea.
> >
> > job_conf.xml:
> >
> > <?xml version="1.0"?>
> > <job_conf>
> > <destinations>
> > <destination name="default" runner="local"/>
> > <destination name="pbs_default" runner="pbs">
> > <native_param
name="queue">batch</native_param>
> > </destination>
> > </destinations>
> > <tools>
> > <tool id="upload1" destination="default"/>
> > <tool id="bowtie_wrapper"
destination="pbs_default">
> > <resource type="cores">8</resource>
> > </tool>
> > </tools>
> > </job_conf>
> >
> > pbs.py then knows to translate '<resource
type="cores">8</resource>'
to
> > '-l nodes=1:ppn=8'.
> >
> > Your tool can access that value a bunch, like .cores.
> >
> > The same should be possible for other consumables.
> >
>
> Sounds good. Would there be a global default setting somewhere in
> universe.ini or elsewhere for when job_conf.xml didn't set a value?
default_cluster_job_runner will remain for backwards compatibility, but
we'll ship a sample job_conf.xml that runs everything locally by
default.
--nate
Haha, and I did that before realizing I could do just what I needed by
writing tool-specific pbs:// URLs at the end of the config file... I'm such
an idiot.
But I really like what you did of it and I have a couple of questions.
Concerning the single-threaded tools, what would happen if the number of
threads set in the xml file was >1 ?
Could it be possible to forbid a tool to run on a given node?
Thanks,
L-A
>
> Peter
>