Re: [galaxy-dev] suggestion for multithreading
(moved to galaxy-dev) Nate Coraor wrote, On 06/02/2011 01:31 PM:
Peter Cock wrote:
On Thu, Jun 2, 2011 at 6:23 PM, Nate Coraor <nate@bx.psu.edu> wrote:
pbs.py then knows to translate '<resource type="cores">8</resource>' to '-l nodes=1:ppn=8'.
Your tool can access that value a bunch, like $__resources__.cores.
The same should be possible for other consumables.
Just a thought here: The actual parameters that are passed to the scheduler are not necessarily hard-coded. Meaning, at least with SGE, specifying the number of cores can be: qsub -pe threads=8 or qsub -pe cores=8 or qsub -pe jiffies=8 and same thing for memory limitation (e.g. "-l virtual_free=800M"). The reason is that those resources (e.g. "threads", "cores", "virtual_free") are just identifiers, and they are created and configured by whomever installed SGE - they are not built-in or hard-coded). So just be careful in your design/implementation when automatically translating XML resources to hard-coded parameters. If you do hard-code them, just make sure the specifically document it (i.e. Galaxy expect the SGE threads parameter to be "-pe threads=8" and nothing else). -gordon
Assaf Gordon wrote:
(moved to galaxy-dev)
Nate Coraor wrote, On 06/02/2011 01:31 PM:
Peter Cock wrote:
On Thu, Jun 2, 2011 at 6:23 PM, Nate Coraor <nate@bx.psu.edu> wrote:
pbs.py then knows to translate '<resource type="cores">8</resource>' to '-l nodes=1:ppn=8'.
Your tool can access that value a bunch, like $__resources__.cores.
The same should be possible for other consumables.
Just a thought here:
The actual parameters that are passed to the scheduler are not necessarily hard-coded. Meaning, at least with SGE, specifying the number of cores can be: qsub -pe threads=8 or qsub -pe cores=8 or qsub -pe jiffies=8
and same thing for memory limitation (e.g. "-l virtual_free=800M").
The reason is that those resources (e.g. "threads", "cores", "virtual_free") are just identifiers, and they are created and configured by whomever installed SGE - they are not built-in or hard-coded).
So just be careful in your design/implementation when automatically translating XML resources to hard-coded parameters.
If you do hard-code them, just make sure the specifically document it (i.e. Galaxy expect the SGE threads parameter to be "-pe threads=8" and nothing else).
Hrm, I didn't realize that SGE didn't have a standard resource name for this. It's probably something we can just add into the XML as "cores" in Galaxy == "threads" in my SGE install. Thanks for the heads up.
-gordon
participants (2)
-
Assaf Gordon
-
Nate Coraor