Attempting to use parallelism in BWA
Hi, I am using Python 2.7. I am running with an hg checked out, up to date version of "https://bitbucket.org/galaxy/galaxy-dist/". I am very new to Galaxy. I have an instance setup *and running* locally. I am able to perform routine actions such as uploading files, processing with Picard, and aligning with BWA. I have been able to make the jobs execute out our SGE cluster. In order to better utilize my cluster, I attempted to enable parallelism of the tasks that support it (such as BWA) by configuring the * universe_wsgi.ini* file with the parameter use_tasked_jobs = True When I do this and attempt execute a BWA alignment on a 4.1Gb fastq file, I am given the error Result 256 from mv /pbtech_mounts/fclab_store006/gobyweb_dat/galaxy-python/database/job_working_directory/000/21/task_1/dataset_21.dat /pbtech_mounts/fclab_store006/gobyweb_dat/galaxy-python/database/files/000/dataset_21.dat /pbtech_mounts/fclab_store006/gobyweb_dat/galaxy-python/database/job_working_directory/000/21/task_0: Traceback (most recent call last): File "./scripts/extract_dataset_part.py", line 25, in ? import galaxy.model.mapping #need to load this before we unpickle, in order to setup properties assigned by the mappers File "/pbtech_mounts/fclab_store006/gobyweb_dat/galaxy-python/lib/galaxy/model/__init__.py", line 13, in ? import galaxy.datatypes.registry File "/pbtech_mounts/fclab_store006/gobyweb_dat/galaxy-python/lib/galaxy/datatypes/registry.py", line 146 finally: ^ SyntaxError: invalid syntax The alignment failed. If you need the complete error message, contact me and I can supply you with it. It's too large to paste in-line and attachments aren't allowed. Is there more I need to do to enable parallel jobs (splitting of input, align, then merge)? Any suggestions are greatly appreciated. Kevin
Hi Kevin- That's truly odd. The error happens when the offending file is imported, which means that it is a syntax issue that should have been seen earlier. I'll take a look at this soon (unless someone else beats me to it). In the meantime, of course, do not use the use_tasked_jobs on this instance of Galaxy. If others have seen similar issues, then please speak up. -Scott ----- Original Message -----
Hi,
I am using Python 2.7. I am running with an hg checked out, up to date version of " https://bitbucket.org/galaxy/galaxy-dist/ ".
I am very new to Galaxy. I have an instance setup and running locally. I am able to perform routine actions such as uploading files, processing with Picard, and aligning with BWA. I have been able to make the jobs execute out our SGE cluster.
In order to better utilize my cluster, I attempted to enable parallelism of the tasks that support it (such as BWA) by configuring the universe_wsgi.ini file with the parameter
use_tasked_jobs = True
When I do this and attempt execute a BWA alignment on a 4.1Gb fastq file, I am given the error
Result 256 from mv /pbtech_mounts/fclab_store006/gobyweb_dat/galaxy-python/database/job_working_directory/000/21/task_1/dataset_21.dat /pbtech_mounts/fclab_store006/gobyweb_dat/galaxy-python/database/files/000/dataset_21.dat
/pbtech_mounts/fclab_store006/gobyweb_dat/galaxy-python/database/job_working_directory/000/21/task_0:
Traceback (most recent call last):
File "./scripts/extract_dataset_part.py", line 25, in ?
import galaxy.model.mapping #need to load this before we unpickle, in order to setup properties assigned by the mappers
File "/pbtech_mounts/fclab_store006/gobyweb_dat/galaxy-python/lib/galaxy/model/__init__.py", line 13, in ?
import galaxy.datatypes.registry
File "/pbtech_mounts/fclab_store006/gobyweb_dat/galaxy-python/lib/galaxy/datatypes/registry.py", line 146
finally:
^
SyntaxError: invalid syntax
The alignment failed.
If you need the complete error message, contact me and I can supply you with it. It's too large to paste in-line and attachments aren't allowed.
Is there more I need to do to enable parallel jobs (splitting of input, align, then merge)?
Any suggestions are greatly appreciated.
Kevin ___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Hi Kevin, I'm not able to reproduce this with python 2.7. Can you confirm that galaxy is indeed using the installed python 2.7 and not, perhaps, 2.4? The structure of python's try/except/finally clause changed with python 2.5 and if you were unintentionally running 2.4 that would explain this error. And lastly, regarding the parallelization of BWA, there's nothing else you need to do; setting use_tasked_jobs = True should handle it. -Dannon On Jul 17, 2012, at 12:27 PM, Kevin C. Dorff wrote:
Hi,
I am using Python 2.7. I am running with an hg checked out, up to date version of "https://bitbucket.org/galaxy/galaxy-dist/".
I am very new to Galaxy. I have an instance setup and running locally. I am able to perform routine actions such as uploading files, processing with Picard, and aligning with BWA. I have been able to make the jobs execute out our SGE cluster.
In order to better utilize my cluster, I attempted to enable parallelism of the tasks that support it (such as BWA) by configuring the universe_wsgi.ini file with the parameter
use_tasked_jobs = True
When I do this and attempt execute a BWA alignment on a 4.1Gb fastq file, I am given the error
Result 256 from mv /pbtech_mounts/fclab_store006/gobyweb_dat/galaxy-python/database/job_working_directory/000/21/task_1/dataset_21.dat /pbtech_mounts/fclab_store006/gobyweb_dat/galaxy-python/database/files/000/dataset_21.dat /pbtech_mounts/fclab_store006/gobyweb_dat/galaxy-python/database/job_working_directory/000/21/task_0: Traceback (most recent call last): File "./scripts/extract_dataset_part.py", line 25, in ? import galaxy.model.mapping #need to load this before we unpickle, in order to setup properties assigned by the mappers File "/pbtech_mounts/fclab_store006/gobyweb_dat/galaxy-python/lib/galaxy/model/__init__.py", line 13, in ? import galaxy.datatypes.registry File "/pbtech_mounts/fclab_store006/gobyweb_dat/galaxy-python/lib/galaxy/datatypes/registry.py", line 146 finally: ^ SyntaxError: invalid syntax The alignment failed.
If you need the complete error message, contact me and I can supply you with it. It's too large to paste in-line and attachments aren't allowed.
Is there more I need to do to enable parallel jobs (splitting of input, align, then merge)?
Any suggestions are greatly appreciated.
Kevin ___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
participants (3)
-
Dannon Baker -
Kevin C. Dorff -
Scott McManus