Passing the number of mpi processors to SGE from xml file
Any thoughts on how to pass the number of processor I want to use from my xml wrapper to the job template. I am trying to wrap pBWA and use mpi. Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: ichorny@illumina.com<mailto:ichorny@illumina.com> Website: www.illumina.com<http://www.illumina.com>
I think with the current state of Galaxy this would need to be fixed. You would need to hard code the number of processors in the xml wrapper and in the corresponding job runner for that tool ________________________________ From: galaxy-dev-bounces@lists.bx.psu.edu [galaxy-dev-bounces@lists.bx.psu.edu] on behalf of Chorny, Ilya [ichorny@illumina.com] Sent: Tuesday, September 13, 2011 7:58 PM To: galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] Passing the number of mpi processors to SGE from xml file Any thoughts on how to pass the number of processor I want to use from my xml wrapper to the job template. I am trying to wrap pBWA and use mpi. Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: ichorny@illumina.com<mailto:ichorny@illumina.com> Website: www.illumina.com<http://www.illumina.com>
Nate, Any thoughts on how to do this? Can I add an xml tag that gets parsed and set's " -PE mpi $NSLOTS" in the job template. Can you point me to where the xml get's parsed? Thanks, Ilya From: Glen Beane [mailto:Glen.Beane@jax.org] Sent: Tuesday, September 13, 2011 5:00 PM To: Chorny, Ilya; galaxy-dev@lists.bx.psu.edu Subject: RE: [galaxy-dev] Passing the number of mpi processors to SGE from xml file I think with the current state of Galaxy this would need to be fixed. You would need to hard code the number of processors in the xml wrapper and in the corresponding job runner for that tool ________________________________ From: galaxy-dev-bounces@lists.bx.psu.edu<mailto:galaxy-dev-bounces@lists.bx.psu.edu> [galaxy-dev-bounces@lists.bx.psu.edu] on behalf of Chorny, Ilya [ichorny@illumina.com] Sent: Tuesday, September 13, 2011 7:58 PM To: galaxy-dev@lists.bx.psu.edu<mailto:galaxy-dev@lists.bx.psu.edu> Subject: [galaxy-dev] Passing the number of mpi processors to SGE from xml file Any thoughts on how to pass the number of processor I want to use from my xml wrapper to the job template. I am trying to wrap pBWA and use mpi. Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: ichorny@illumina.com<mailto:ichorny@illumina.com> Website: www.illumina.com<http://www.illumina.com>
On Sep 14, 2011, at 11:30 AM, Chorny, Ilya wrote: Nate, Any thoughts on how to do this? Can I add an xml tag that gets parsed and set’s “ –PE mpi $NSLOTS” in the job template. Can you point me to where the xml get’s parsed? Thanks, Ilya I'm not that familiar with SGE, but this would be pretty trivial with TORQUE assuming a fixed number of MPI processes for the tool. No modifications to the job template would be necessary: tool specific job runner specifies number of nodes/ppn or procs; mpiexec (built with TM support) launches appropriate number of processes on all nodes assigned to job by TORQUE. From: Glen Beane [mailto:Glen.Beane@jax.org] Sent: Tuesday, September 13, 2011 5:00 PM To: Chorny, Ilya; galaxy-dev@lists.bx.psu.edu<mailto:galaxy-dev@lists.bx.psu.edu> Subject: RE: [galaxy-dev] Passing the number of mpi processors to SGE from xml file I think with the current state of Galaxy this would need to be fixed. You would need to hard code the number of processors in the xml wrapper and in the corresponding job runner for that tool ________________________________ From: galaxy-dev-bounces@lists.bx.psu.edu<mailto:galaxy-dev-bounces@lists.bx.psu.edu> [galaxy-dev-bounces@lists.bx.psu.edu] on behalf of Chorny, Ilya [ichorny@illumina.com] Sent: Tuesday, September 13, 2011 7:58 PM To: galaxy-dev@lists.bx.psu.edu<mailto:galaxy-dev@lists.bx.psu.edu> Subject: [galaxy-dev] Passing the number of mpi processors to SGE from xml file Any thoughts on how to pass the number of processor I want to use from my xml wrapper to the job template. I am trying to wrap pBWA and use mpi. Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: ichorny@illumina.com<mailto:ichorny@illumina.com> Website: www.illumina.com<http://www.illumina.com> -- Glen L. Beane Senior Software Engineer The Jackson Laboratory (207) 288-6153
It's easy to hard code in the ini but I want to be able to set the number of processors as a parameter. From: Glen Beane [mailto:Glen.Beane@jax.org] Sent: Wednesday, September 14, 2011 10:34 AM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Passing the number of mpi processors to SGE from xml file On Sep 14, 2011, at 11:30 AM, Chorny, Ilya wrote: Nate, Any thoughts on how to do this? Can I add an xml tag that gets parsed and set's " -PE mpi $NSLOTS" in the job template. Can you point me to where the xml get's parsed? Thanks, Ilya I'm not that familiar with SGE, but this would be pretty trivial with TORQUE assuming a fixed number of MPI processes for the tool. No modifications to the job template would be necessary: tool specific job runner specifies number of nodes/ppn or procs; mpiexec (built with TM support) launches appropriate number of processes on all nodes assigned to job by TORQUE. From: Glen Beane [mailto:Glen.Beane@jax.org]<mailto:[mailto:Glen.Beane@jax.org]> Sent: Tuesday, September 13, 2011 5:00 PM To: Chorny, Ilya; galaxy-dev@lists.bx.psu.edu<mailto:galaxy-dev@lists.bx.psu.edu> Subject: RE: [galaxy-dev] Passing the number of mpi processors to SGE from xml file I think with the current state of Galaxy this would need to be fixed. You would need to hard code the number of processors in the xml wrapper and in the corresponding job runner for that tool ________________________________ From: galaxy-dev-bounces@lists.bx.psu.edu<mailto:galaxy-dev-bounces@lists.bx.psu.edu> [galaxy-dev-bounces@lists.bx.psu.edu] on behalf of Chorny, Ilya [ichorny@illumina.com] Sent: Tuesday, September 13, 2011 7:58 PM To: galaxy-dev@lists.bx.psu.edu<mailto:galaxy-dev@lists.bx.psu.edu> Subject: [galaxy-dev] Passing the number of mpi processors to SGE from xml file Any thoughts on how to pass the number of processor I want to use from my xml wrapper to the job template. I am trying to wrap pBWA and use mpi. Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: ichorny@illumina.com<mailto:ichorny@illumina.com> Website: www.illumina.com<http://www.illumina.com> -- Glen L. Beane Senior Software Engineer The Jackson Laboratory (207) 288-6153
On Sep 14, 2011, at 2:07 PM, Chorny, Ilya wrote: It’s easy to hard code in the ini but I want to be able to set the number of processors as a parameter. that would be a nice modification to galaxy. I've always wanted parameterized job runners, so I could include variables in a job runner specification in the .ini file and then that would get filled in based on some variable set by the job template -- Glen L. Beane Senior Software Engineer The Jackson Laboratory (207) 288-6153
Yeah, not possible now. Ilya, running in a controlled environment you can just have the wrapper read the number of allocated processors from the environment once it's scheduled and run. --nate Glen Beane wrote:
On Sep 14, 2011, at 2:07 PM, Chorny, Ilya wrote:
It’s easy to hard code in the ini but I want to be able to set the number of processors as a parameter.
that would be a nice modification to galaxy. I've always wanted parameterized job runners, so I could include variables in a job runner specification in the .ini file and then that would get filled in based on some variable set by the job template
-- Glen L. Beane Senior Software Engineer The Jackson Laboratory (207) 288-6153
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
participants (3)
-
Chorny, Ilya -
Glen Beane -
Nate Coraor