Thanks Nate!
I did think that job_conf.xml would overwrite settings in universe_wsgi.ini. I rewrote it and see that the galaxy job gets to run on the cluster. However, the TMPDIR clause defined in nativeSpecification didn't seems to be effective :(
the console output "log" does not show what the final nativeSpecification was parsed as. Given I didn't see any errors, I can assume that it was taken as defined and it is SGE somehow not heeding the parameter?
Any other ideas of how I may fix this? Or should I look at cluster configuration to change it without relaying on galaxy?
relevant console output:
galaxy.jobs DEBUG 2013-08-16 02:50:51,622 Loading job configuration from ./job_conf.xml
galaxy.jobs DEBUG 2013-08-16 02:50:51,622 Read definition for handler 'main'
galaxy.jobs INFO 2013-08-16 02:50:51,623 Setting <handlers> default to child with id 'main'
galaxy.jobs DEBUG 2013-08-16 02:50:51,623 <destinations> default set to child with id or tag 'sge'
galaxy.jobs DEBUG 2013-08-16 02:50:51,623 Done loading job configuration
...
galaxy.jobs.manager DEBUG 2013-08-16 02:51:01,190 Starting job handler
galaxy.jobs.runners DEBUG 2013-08-16 02:51:01,192 Starting 4 LocalRunner workers
galaxy.jobs DEBUG 2013-08-16 02:51:01,194 Loaded job runner 'galaxy.jobs.runners.local:LocalJobRunner' as 'local'
galaxy.jobs.runners DEBUG 2013-08-16 02:51:01,242 Starting 4 DRMAARunner workers
galaxy.jobs DEBUG 2013-08-16 02:51:01,243 Loaded job runner 'galaxy.jobs.runners.drmaa:DRMAAJobRunner' as 'drmaa'
galaxy.jobs.handler DEBUG 2013-08-16 02:51:01,243 Loaded job runners plugins: drmaa:local
galaxy.jobs.handler INFO 2013-08-16 02:51:01,244 job handler stop queue started
galaxy.jobs.handler INFO 2013-08-16 02:51:01,253 job handler queue started
...
galaxy.jobs DEBUG 2013-08-16 02:51:22,316 (218) Working directory for job is: /usr/prog/galaxy/galaxy-dist/database/job_working_directory/000/218
galaxy.jobs.handler DEBUG 2013-08-16 02:51:22,320 (218) Dispatching to drmaa runner
galaxy.jobs DEBUG 2013-08-16 02:51:22,399 (218) Persisting job destination (destination id: sge)
galaxy.jobs.handler INFO 2013-08-16 02:51:22,415 (218) Job dispatched
galaxy.tools DEBUG 2013-08-16 02:51:22,648 Building dependency shell command for dependency 'bwa'
galaxy.jobs.runners.drmaa DEBUG 2013-08-16 02:51:22,764 (218) submitting file /usr/prog/galaxy/galaxy-dist/database/job_working_directory/000/218/galaxy_218.sh
galaxy.jobs.runners.drmaa DEBUG 2013-08-16 02:51:22,765 (218) command is: PACKAGE_BASE=/usr/prog/galaxy/tool_dependency/bwa/0.5.9/devteam/bwa_wrappers/150b3fe44caa; export PACKAGE_BASE; . /usr/prog/galaxy/tool_dependency/bwa/0.5.9/devteam/bwa_wrappers/150b3fe44caa/env.sh; python /usr/prog/galaxy/shed_tools/
toolshed.g2.bx.psu.edu/repos/devteam/bwa_wrappers/150b3fe44caa/bwa_wrappers/bwa_wrapper.py --threads="4" --fileSource="indexed" --ref="/db/ngs-bioinfo/genomes/human/hg19/Homo_sapiens_assembly19.fasta" --do_not_build_index --input1="/usr/prog/galaxy/galaxy-dist/database/files/000/dataset_268.dat" --output="/usr/prog/galaxy/galaxy-dist/database/files/000/dataset_287.dat" --genAlignType="single" --params="full" --maxEditDist="0" --fracMissingAligns="0.04" --maxGapOpens="0" --maxGapExtens="-1" --disallowLongDel="16" --disallowIndel="5" --seed="-1" --maxEditDistSeed="2" --mismatchPenalty="3" --gapOpenPenalty="11" --gapExtensPenalty="4" --suboptAlign="" --noIterSearch="false" --outputTopN="3" --outputTopNDisc="10" --maxInsertSize="500" --maxOccurPairing="100000" --suppressHeader="true"
galaxy.jobs.runners.drmaa INFO 2013-08-16 02:51:22,773 (218) queued as 37234
galaxy.jobs DEBUG 2013-08-16 02:51:22,805 (218) Persisting job destination (destination id: sge)
galaxy.jobs.runners.drmaa DEBUG 2013-08-16 02:51:23,273 (218/37234) state change: job is queued and active
galaxy.jobs.runners.drmaa DEBUG 2013-08-16 02:51:24,301 (218/37234) state change: job is running
the updated job_conf.xml ::
<job_conf>
<plugins>
<plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="4"/>
<plugin id="drmaa" type="runner" load="galaxy.jobs.runners.drmaa:DRMAAJobRunner"/>
</plugins>
<handlers>
<handler id="main"/>
</handlers>
<destinations default="sge">
<destination id="sge" runner="drmaa">
<!-- runner need to refer to a plugin id -->
<param id="nativeSpecification">-q default.q -V -v TMPDIR=/scratch</param>
</destination>
<destination id="local" runner="local"/>
</destinations>
</job_conf>
Much thanks again,
-Tin