May 2013 - galaxy-dev - lists.galaxyproject.org

How to transfer or upload datasets to Galaxy server automatically?
by 王聃Don 09 May '13

09 May '13

Hello, Galaxy developers and maintainers! I'm a user in Chinese mainland and I'd like to deploy our Galaxy server locally. There's some requirements I cannot realize and I still have no idea after searching over all Galaxy Wiki pages and Galaxy Developer docs. The problem is, I need to jump from another external webpage to Galaxy index page through a JS action and a specific included dataset has to be transferred to Galaxy at the same time. In other words, users click a button on an external webpage and will transfer to Galaxy index page with an automatic action of user-login and dataset-uploading in Galaxy server. I've no idea how to realize someway and really need some help on it. Thanks Wang Dan

2 1

Error cleaning up Condor jobs
by Branden Timm 08 May '13

08 May '13

Hi All, I've been working to configure a new Galaxy instance to run jobs under Condor. Things are 99% working at this point, but what seems to be happening is after the Condor job finishes Galaxy tries to clean up a cluster file that isn't there, namely the .ec (exit code) file. Relevant log info: galaxy.jobs DEBUG 2013-05-07 15:02:49,364 (1985) Working directory for job is: /home/GLBRCORG/galaxy/database/job_working_directory/001/1985 galaxy.jobs.handler DEBUG 2013-05-07 15:02:49,387 (1985) Dispatching to condor runner galaxy.jobs DEBUG 2013-05-07 15:02:49,720 (1985) Persisting job destination (destination id: condor) galaxy.jobs.handler INFO 2013-05-07 15:02:49,761 (1985) Job dispatched galaxy.jobs.runners.condor DEBUG 2013-05-07 15:02:56,368 (1985) submitting file /home/GLBRCORG/galaxy/database/condor/galaxy_1985.sh galaxy.jobs.runners.condor DEBUG 2013-05-07 15:02:56,369 (1985) command is: python /home/GLBRCORG/galaxy/galaxy-central/tools/fastq/fastq_to_fasta.py '/home/GLBRCORG/galaxy/database/files/000/dataset_3.dat' '/home/GLBRCORG/galaxy/database/files/002/dataset_2842.dat' ''; cd /home/GLBRCORG/galaxy/galaxy-central; /home/GLBRCORG/galaxy/galaxy-central/set_metadata.sh /home/GLBRCORG/galaxy/database/files /home/GLBRCORG/galaxy/database/job_working_directory/001/1985 . /home/GLBRCORG/galaxy/galaxy-central/universe_wsgi.ini /home/GLBRCORG/galaxy/database/tmp/tmpGe1JZJ /home/GLBRCORG/galaxy/database/job_working_directory/001/1985/galaxy.json /home/GLBRCORG/galaxy/database/job_working_directory/001/1985/metadata_in_HistoryDatasetAssociation_3161_are5Bg,/home/GLBRCORG/galaxy/database/job_working_directory/001/1985/metadata_kwds_HistoryDatasetAssociation_3161_p73Yus,/home/GLBRCORG/galaxy/database/job_working_directory/001/1985/metadata_out_HistoryDatasetAssociation_3161_tLqep6,/home/GLBRCORG/galaxy/database/job_working_directory/001/1985/metadata_results_HistoryDatasetAssociation_3161_3QSW5X,,/home/GLBRCORG/galaxy/database/job_working_directory/001/1985/metadata_override_HistoryDatasetAssociation_3161_JUFvmk galaxy.jobs.runners.condor INFO 2013-05-07 15:02:58,960 (1985) queued as 15 galaxy.jobs DEBUG 2013-05-07 15:02:59,110 (1985) Persisting job destination (destination id: condor) galaxy.jobs.runners.condor DEBUG 2013-05-07 15:02:59,536 (1985/15) job is now running galaxy.jobs.runners.condor DEBUG 2013-05-07 15:07:16,966 (1985/15) job is now running galaxy.jobs.runners.condor DEBUG 2013-05-07 15:07:17,279 (1985/15) job has completed galaxy.jobs.runners DEBUG 2013-05-07 15:07:17,417 (1985/15) Unable to cleanup /home/GLBRCORG/galaxy/database/condor/galaxy_1985.ec: [Errno 2] No such file or directory: '/home/GLBRCORG/galaxy/database/condor/galaxy_1985.ec' galaxy.jobs DEBUG 2013-05-07 15:07:17,560 setting dataset state to ERROR galaxy.jobs DEBUG 2013-05-07 15:07:17,961 job 1985 ended galaxy.datatypes.metadata DEBUG 2013-05-07 15:07:17,961 Cleaning up external metadata files I've done a watch on the condor job directory, and as far as I can tell galaxy_1985.ec never gets created. From a cursory look at lib/galaxy/jobs/runners/__init__.py and condor.py, it looks like the cleanup is happening in the AsynchronousJobState::cleanup method, which iterates on the cleanup_file_attributes list. I naively tried to override cleanup_file_attributes in CondorJobState to disinclude 'exit_code_file', to no avail. I'm hoping somebody can spot where the hiccup is here. Another question that is on my mind is should a failure to cleanup cluster files set the dataset state to ERROR? An inspection of the output file from my job leads me to believe it finished just fine, and indicating failure to the user because Galaxy couldn't cleanup a 1b error code file seems a little extreme to me. Thanks! -- Branden Timm btimm(a)energy.wisc.edu

2 4

question on fastqc
by Kathryn Sun 08 May '13

08 May '13

Hello, I've set up galaxy at the local linux. After data configuration, I run fastqc to check how it runs. Here I got error message as below -- error An error occurred running this job: Traceback (most recent call last): File "../galaxy-dist/tools/rgenetics/rgFastQC.py", line 158, in assert os.path.isfile(opts.executable),'##rgFastQC.py error - cannot find executable %s' % opts.executable AssertionError: ##r Here is the detail of the error -- Tool: FastQC:Read QC Name: FastQC_FASTQ Groomer on data 6.html Created: May 02, 2013 Filesize: 0 bytes Dbkey: mm9 Format: html Tool Version: Tool Standard Output: stdout Tool Standard Error: stderr Tool Exit Code: 1 Input Parameter Value Short read data from your current history 8: FASTQ Groomer on data 6 Title for the output file - to remind you what the job was for FastQC Contaminant list No dataset What's the problem? Thank you! Kathryn

1 0

routing to a cluster or not on a per-tool basis
by Dan Tenenbaum 08 May '13

08 May '13

Hi, I have some tools that run really quickly without using any kind of cluster. I would prefer not to run these tools on a cluster, as the overhead of submitting these jobs makes them take much longer than they otherwise would. I have other tools that are computationally intensive and need to be run on a cluster. I would like to expose all these tools in the same Galaxy instance, but have some tools run on the cluster and others not. Is this possible? Thanks, Dan

2 3

Difficulties using <repeat> tagset with min attribute
by Cory Spencer 08 May '13

08 May '13

Hi all - I've been trying to get the <repeat>...</repeat> tag working with a min attribute for some time now, though without any success. It works in other tools distributed with Galaxy, but when I attempt to use it in one of our custom tools, it dies with a "AttributeError: 'ExpressionContext' object has no attribute 'keys'" exception. Can anybody offer any insight? The full traceback is: ⇝ AttributeError: 'ExpressionContext' object has no attribute 'keys' URL: http://localhost:8080/tool_runner?tool_id=scde-list-compare Module weberror.evalexception.middleware:364 in respond view >> app_iter = self.application(environ, detect_start_response) Module paste.debug.prints:98 in __call__ view >> environ, self.app) Module paste.wsgilib:539 in intercept_output view >> app_iter = application(environ, replacement_start_response) Module paste.recursive:80 in __call__ view >> return self.application(environ, start_response) Module paste.httpexceptions:632 in __call__ view >> return self.application(environ, start_response) Module galaxy.web.framework.base:160 in __call__ view >> body = method( trans, **kwargs ) Module galaxy.web.controllers.tool_runner:68 in index view >> template, vars = tool.handle_input( trans, params.__dict__ ) Module galaxy.tools:1320 in handle_input view >> state = self.new_state( trans ) Module galaxy.tools:1248 in new_state view >> self.fill_in_new_state( trans, inputs, state.inputs ) Module galaxy.tools:1257 in fill_in_new_state view >> state[ input.name ] = input.get_initial_value( trans, context ) Module galaxy.tools.parameters.grouping:100 in get_initial_value view >> rval_dict[ input.name ] = input.get_initial_value( trans, context ) Module galaxy.tools.parameters.basic:1016 in get_initial_value view >> return SelectToolParameter.get_initial_value( self, trans, context ) Module galaxy.tools.parameters.basic:785 in get_initial_value view >> if self.need_late_validation( trans, context ): Module galaxy.tools.parameters.basic:1022 in need_late_validation view >> if super( ColumnListParameter, self ).need_late_validation( trans, context ): Module galaxy.tools.parameters.basic:766 in need_late_validation view >> for layer in context.itervalues(): Module UserDict:116 in itervalues view >> for _, v in self.iteritems(): Module UserDict:109 in iteritems view >> for k in self: Module UserDict:96 in __iter__ view >> for k in self.keys(): AttributeError: 'ExpressionContext' object has no attribute 'keys'

2 2

Step 3 of "get Microbial Data" install
by Sarah Maman 08 May '13

08 May '13

Hello, I try to set "Get Microbial Data" in my local instance of Galaxy. So, as explain in /path/to//src/galaxy/galaxy-dist/scripts/microbes/README.txt , step 1 and 2 or OK, binary 'faToNib' is available on our cluster, but step 3 generate empty files : drwxr-xr-x 4 galaxy wbioinfo 100 18 avril 14:26 .. -rw-r--r-- 1 galaxy wbioinfo 203 18 avril 14:26 harvest.txt -rw-r--r-- 1 galaxy wbioinfo 69 18 avril 14:34 ncbi_to_ucsc.txt -rw-r--r-- 1 galaxy wbioinfo 0 19 avril 15:47 sequence.txt -rw-r--r-- 1 galaxy wbioinfo 0 19 avril 15:47 seq.loc drwxr-xr-x 2 galaxy wbioinfo 118 19 avril 15:47 . Could you please, hel me ? Thnaks in advance, Sarah Maman -- --*-- Sarah Maman INRA - LGC - SIGENAE http://www.sigenae.org/ Chemin de Borde-Rouge - Auzeville - BP 52627 31326 Castanet-Tolosan cedex - FRANCE Tel: +33(0)5.61.28.57.08 Fax: +33(0)5.61.28.57.53 --*--

2 1

Error installing migrated tool deps
by Branden Timm 07 May '13

07 May '13

I recently upgraded to the latest galaxy-central, and was advised on first run that two tools in tool_conf.xml had been removed from the distribution, but could be installed from the tool shed. I ran the script that it generated, however the script fails with the following messages: No handlers could be found for logger "galaxy.tools.parameters.dynamic_options" /home/GLBRCORG/galaxy/galaxy-central/eggs/pysam-0.4.2_kanwei_b10f6e722e9a-py2.6-linux-x86_64-ucs4.egg/pysam/__init__.py:1: RuntimeWarning: __builtin__.file size changed, may indicate binary incompatibility from csamtools import * Repositories will be installed into configured tool_path location ../shed_tools Skipping automatic install of repository ' bowtie_wrappers ' because it has already been installed in location ../shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/bowtie_wrappers/0c7e4eadfb3c Traceback (most recent call last): File "./scripts/migrate_tools/migrate_tools.py", line 21, in <module> app = MigrateToolsApplication( sys.argv[ 1 ] ) File "/home/GLBRCORG/galaxy/galaxy-central/lib/tool_shed/galaxy_install/migrate/common.py", line 76, in __init__ install_dependencies=install_dependencies ) File "/home/GLBRCORG/galaxy/galaxy-central/lib/tool_shed/galaxy_install/install_manager.py", line 75, in __init__ self.install_repository( repository_elem, install_dependencies ) File "/home/GLBRCORG/galaxy/galaxy-central/lib/tool_shed/galaxy_install/install_manager.py", line 319, in install_repository install_dependencies=install_dependencies ) File "/home/GLBRCORG/galaxy/galaxy-central/lib/tool_shed/galaxy_install/install_manager.py", line 203, in handle_repository_contents persist=True ) File "/home/GLBRCORG/galaxy/galaxy-central/lib/tool_shed/util/metadata_util.py", line 594, in generate_metadata_for_changeset_revision tool, valid, error_message = tool_util.load_tool_from_config( app, app.security.encode_id( repository.id ), full_path ) AttributeError: 'MigrateToolsApplication' object has no attribute 'security' Any help would be greatly appreciated. Thanks! -- Branden Timm btimm(a)energy.wisc.edu

2 2

Error cleaning up Condor jobs
by Branden Timm 07 May '13

07 May '13

Hi All, I've been working to configure a new Galaxy instance to run jobs under Condor. Things are 99% working at this point, but what seems to be happening is after the Condor job finishes Galaxy tries to clean up a cluster file that isn't there, namely the .ec (exit code) file. Relevant log info: galaxy.jobs DEBUG 2013-05-07 15:02:49,364 (1985) Working directory for job is: /home/GLBRCORG/galaxy/database/job_working_directory/001/1985 galaxy.jobs.handler DEBUG 2013-05-07 15:02:49,387 (1985) Dispatching to condor runner galaxy.jobs DEBUG 2013-05-07 15:02:49,720 (1985) Persisting job destination (destination id: condor) galaxy.jobs.handler INFO 2013-05-07 15:02:49,761 (1985) Job dispatched galaxy.jobs.runners.condor DEBUG 2013-05-07 15:02:56,368 (1985) submitting file /home/GLBRCORG/galaxy/database/condor/galaxy_1985.sh galaxy.jobs.runners.condor DEBUG 2013-05-07 15:02:56,369 (1985) command is: python /home/GLBRCORG/galaxy/galaxy-central/tools/fastq/fastq_to_fasta.py '/home/GLBRCORG/galaxy/database/files/000/dataset_3.dat' '/home/GLBRCORG/galaxy/database/files/002/dataset_2842.dat' ''; cd /home/GLBRCORG/galaxy/galaxy-central; /home/GLBRCORG/galaxy/galaxy-central/set_metadata.sh /home/GLBRCORG/galaxy/database/files /home/GLBRCORG/galaxy/database/job_working_directory/001/1985 . /home/GLBRCORG/galaxy/galaxy-central/universe_wsgi.ini /home/GLBRCORG/galaxy/database/tmp/tmpGe1JZJ /home/GLBRCORG/galaxy/database/job_working_directory/001/1985/galaxy.json /home/GLBRCORG/galaxy/database/job_working_directory/001/1985/metadata_in_HistoryDatasetAssociation_3161_are5Bg,/home/GLBRCORG/galaxy/database/job_working_directory/001/1985/metadata_kwds_HistoryDatasetAssociation_3161_p73Yus,/home/GLBRCORG/galaxy/database/job_working_directory/001/1985/metadata_out_HistoryDatasetAssociation_3161_tLqep6,/home/GLBRCORG/galaxy/database/job_working_directory/001/1985/metadata_results_HistoryDatasetAssociation_3161_3QSW5X,,/home/GLBRCORG/galaxy/database/job_working_directory/001/1985/metadata_override_HistoryDatasetAssociation_3161_JUFvmk galaxy.jobs.runners.condor INFO 2013-05-07 15:02:58,960 (1985) queued as 15 galaxy.jobs DEBUG 2013-05-07 15:02:59,110 (1985) Persisting job destination (destination id: condor) galaxy.jobs.runners.condor DEBUG 2013-05-07 15:02:59,536 (1985/15) job is now running galaxy.jobs.runners.condor DEBUG 2013-05-07 15:07:16,966 (1985/15) job is now running galaxy.jobs.runners.condor DEBUG 2013-05-07 15:07:17,279 (1985/15) job has completed galaxy.jobs.runners DEBUG 2013-05-07 15:07:17,417 (1985/15) Unable to cleanup /home/GLBRCORG/galaxy/database/condor/galaxy_1985.ec: [Errno 2] No such file or directory: '/home/GLBRCORG/galaxy/database/condor/galaxy_1985.ec' galaxy.jobs DEBUG 2013-05-07 15:07:17,560 setting dataset state to ERROR galaxy.jobs DEBUG 2013-05-07 15:07:17,961 job 1985 ended galaxy.datatypes.metadata DEBUG 2013-05-07 15:07:17,961 Cleaning up external metadata files I've done a watch on the condor job directory, and as far as I can tell galaxy_1985.ec never gets created. From a cursory look at lib/galaxy/jobs/runners/__init__.py and condor.py, it looks like the cleanup is happening in the AsynchronousJobState::cleanup method, which iterates on the cleanup_file_attributes list. I naively tried to override cleanup_file_attributes in CondorJobState to disinclude 'exit_code_file', to no avail. I'm hoping somebody can spot where the hiccup is here. Another question that is on my mind is should a failure to cleanup cluster files set the dataset state to ERROR? An inspection of the output file from my job leads me to believe it finished just fine, and indicating failure to the user because Galaxy couldn't cleanup a 1b error code file seems a little extreme to me. Thanks! -- Branden Timm btimm(a)energy.wisc.edu

1 0

Galaxy Keeps Freezing
by Kim Spradling 07 May '13

07 May '13

I have been trying to join the intervals of two datasets side-by-side using the public Main instance, but the job never finishes. I've gotten a message stating that Galaxy is unavailable a couple of times, but I have tried running this job multiple times with no success. I also tried to upload a text file to see if it was an issue with using that specific tool, but it won't complete either. Any help will be greatly appreciated. Thanks, Kim

2 1

Galaxy Jobs Waiting to Run for Several Days
by Elnagheeb, Marwa 07 May '13

07 May '13

Hello, I am a student, and I am using the psu galaxy server for an assignment. My workflow for each of my 5 data sets has been waiting to run 3 jobs (bwa alignment, sam to bam, and bam statistics). One of my datasets reached completion two days ago, but it has been 5 days for the others. Several others in my class have had the same issue. Is there something going on with the server this week? I really need to meet a deadline. Any chance I can get my jobs to run quicker without cloud? Thanks, Marwa

2 1