Hello, I'm trying to move our Galaxy to the our SGE cluster, and I need couple of fine-tuning tweaks. 1. set a project name (in addition to the queue) - globally, and per tool. 2. set extra QSUB options per tool 3. Specify the actual user who submitted the job - in the "email" field and as the job name. Details: The project name is needed for accounting (requested by our SGE admins). To specify a project name, add a fourth slash to the sge URL in universe_wsgi.ini, as so: --- ## Global SGE project name , for all tools default_cluster_job_runner = sge:////GalaxyProject/ ## Tool specific project name Cut1 = sge:////OtherGalaxyProject/ ---- The extra QSUB options per tool allow requesting nodes with more memory or request more cores (=threads) per job. To specify extra options, add a fifth slash to the sge tool runner URL in univer_wsge.ini, as so: ---- ## Tool specific parameters - ## run 'cut' on a node with at least 7GB of ram Cut1 = sge:////GalaxyProject/-l virtual_free=7G/ ## run 'select first lines' on a node with 4 cores, and reserve all the cores for the tool ## allowing multi-threaded 'select lines' operation ;) Show beginning1 = sge:////GalaxyProject/-R y -pe threads 4/ ---- Note: specifying extra parameters ALWAYS overrides the project name (but the project name can be an empty string). The actual user is needed to quickly see (with qstat) who runs what (currently, the job names are "galaxy_NNNN.sh"). Comments are welcomed, -gordon.
Hi Gordon, That's very useful. I have particularly been after a way to alter the SGE parameters on a tool by tool basis as clearly an NGS alignment tool requires many more resources than a 'cut'. Our galaxy server is just for internal use (currently), so the username may not be as useful. I'll have a go at implementing it. Cheers, Chris Assaf Gordon wrote:
Hello,
I'm trying to move our Galaxy to the our SGE cluster, and I need couple of fine-tuning tweaks.
1. set a project name (in addition to the queue) - globally, and per tool. 2. set extra QSUB options per tool 3. Specify the actual user who submitted the job - in the "email" field and as the job name.
Details: The project name is needed for accounting (requested by our SGE admins).
To specify a project name, add a fourth slash to the sge URL in universe_wsgi.ini, as so: --- ## Global SGE project name , for all tools default_cluster_job_runner = sge:////GalaxyProject/
## Tool specific project name Cut1 = sge:////OtherGalaxyProject/ ----
The extra QSUB options per tool allow requesting nodes with more memory or request more cores (=threads) per job. To specify extra options, add a fifth slash to the sge tool runner URL in univer_wsge.ini, as so: ---- ## Tool specific parameters - ## run 'cut' on a node with at least 7GB of ram Cut1 = sge:////GalaxyProject/-l virtual_free=7G/
## run 'select first lines' on a node with 4 cores, and reserve all the cores for the tool ## allowing multi-threaded 'select lines' operation ;) Show beginning1 = sge:////GalaxyProject/-R y -pe threads 4/ ---- Note: specifying extra parameters ALWAYS overrides the project name (but the project name can be an empty string).
The actual user is needed to quickly see (with qstat) who runs what (currently, the job names are "galaxy_NNNN.sh").
Comments are welcomed, -gordon.
Can confirm it works fine on our setup. Very nifty! I would certainly advocate adding this to the main development tree, although I'd prefer not to change the job name to the user's email (I've commented out that bit of the code). If anything, I'd rather the job name reflected the tool being used. Thanks again for this patch, Gordon. Chris Chris Cole wrote:
Hi Gordon,
That's very useful. I have particularly been after a way to alter the SGE parameters on a tool by tool basis as clearly an NGS alignment tool requires many more resources than a 'cut'.
Our galaxy server is just for internal use (currently), so the username may not be as useful.
I'll have a go at implementing it. Cheers,
Chris
Assaf Gordon wrote:
Hello,
I'm trying to move our Galaxy to the our SGE cluster, and I need couple of fine-tuning tweaks.
1. set a project name (in addition to the queue) - globally, and per tool. 2. set extra QSUB options per tool 3. Specify the actual user who submitted the job - in the "email" field and as the job name.
Details: The project name is needed for accounting (requested by our SGE admins).
To specify a project name, add a fourth slash to the sge URL in universe_wsgi.ini, as so: --- ## Global SGE project name , for all tools default_cluster_job_runner = sge:////GalaxyProject/
## Tool specific project name Cut1 = sge:////OtherGalaxyProject/ ----
The extra QSUB options per tool allow requesting nodes with more memory or request more cores (=threads) per job. To specify extra options, add a fifth slash to the sge tool runner URL in univer_wsge.ini, as so: ---- ## Tool specific parameters - ## run 'cut' on a node with at least 7GB of ram Cut1 = sge:////GalaxyProject/-l virtual_free=7G/
## run 'select first lines' on a node with 4 cores, and reserve all the cores for the tool ## allowing multi-threaded 'select lines' operation ;) Show beginning1 = sge:////GalaxyProject/-R y -pe threads 4/ ---- Note: specifying extra parameters ALWAYS overrides the project name (but the project name can be an empty string).
The actual user is needed to quickly see (with qstat) who runs what (currently, the job names are "galaxy_NNNN.sh").
Comments are welcomed, -gordon.
Chris Cole wrote, On 08/14/2009 11:38 AM:
Can confirm it works fine on our setup. Very nifty!
I'm glad to hear it's working fine.
although I'd prefer not to change the job name to the user's email (I've commented out that bit of the code). If anything, I'd rather the job name reflected the tool being used.
To put the tool name as the job's name, replace the line that says: job_name = str(job.history.user.email) with: job_name = job_wrapper.tool.name Leave the rest of the sanitation code in place, because I'm not sure which characters are valid as SGE job names. -gordon
Assaf Gordon wrote:
Chris Cole wrote, On 08/14/2009 11:38 AM:
Can confirm it works fine on our setup. Very nifty!
I'm glad to hear it's working fine.
although I'd prefer not to change the job name to the user's email (I've commented out that bit of the code). If anything, I'd rather the job name reflected the tool being used.
To put the tool name as the job's name, replace the line that says:
job_name = str(job.history.user.email)
with:
job_name = job_wrapper.tool.name
Leave the rest of the sanitation code in place, because I'm not sure which characters are valid as SGE job names.
Perfect, thanks! Chris
participants (2)
-
Assaf Gordon -
Chris Cole