- galaxy-dev - lists.galaxyproject.org

Always seem to see this " [Errno 39] Directory not empty" error when a step fails
by Anthonius deBoer 08 Nov '12

08 Nov '12

Hi, I always see this error pop up in the logs when one of the steps failed... It seems to have a problem deleting the working directory since it claims they are not empty. Ofcourse when I check, the directory is empty, but I am left with a gazilion empty working directories, alhtough no problems with the installation otherwise... Using the latest pull Any ideas? Thanks, Thon galaxy.datatypes.metadata DEBUG 2012-11-08 15:43:29,532 Cleaning up external metadata files galaxy.objectstore CRITICAL 2012-11-08 15:43:29,621 /mnt/ngs/analysis/svcgalaxy/DATA/job_working_directory/000/105 delet e error [Errno 39] Directory not empty: '/mnt/ngs/analysis/svcgalaxy/DATA/job_working_directory/000/105' ls -la /mnt/ngs/analysis/svcgalaxy/DATA/job_working_directory/000/105 total 36K drwxrwsr-x 2 svcgalaxy 0 Nov 8 15:42 . drwxrwsr-x 30 svcgalaxy 565 Nov 8 15:42 .. (galaxy_env)[svcgalaxy@srv151 g]$

1 0

Problems with latest Galaxy 8167:b0c916349070
by Scooter Willis 08 Nov '12

08 Nov '12

Running a CloudMan created instance on Amazon. Updated using cloudman from http://bitbucket.org/galaxy/galaxy-central to version 8167:b0c916349070 Tried to run bowtie and get the following error. Searched the message archives an someone else reported the same problem but thought it was related to a new version of cufflinks. Same problem also exists when running abyss. Given the nature of the error message not even sure what the actual path should be. Also for the abyss install from toolshed the toolshed.g2.bx.psu.edu directory doesn't exist or the six directories underneath. However Abyss was/is installed at the following location galaxy@ip-10-110-143-10:/mnt/galaxyTools/tools/abyss/1.2.5/bin$ ls ABYSS abyss-adjtodot abyss-joindist abyss-tofastq DistanceEst MergePaths PathConsensus SimpleGraph abyss2ace abyss-fac ABYSS-P AdjList KAligner Overlap PathOverlap abyss2afg abyss-fixmate abyss-pe Consensus MergeContigs ParseAligns PopBubbles Would be great if someone could give some guidance on how to resolve the problems. I understand if I need to do something under the covers but the expectation is that the one click install/updates should not then require major command line work that isn't documented. Thanks Scooter Bowtie Stdout Sequence file aligned. ERROR: Unable to read builds file: [Errno 2] No such file or directory: '/mnt/galaxyTools/galaxy-central/lib/galaxy/util/../../../tool-data/shared/ucsc/publicbuilds.txt' ERROR: Unable to read builds file: [Errno 2] No such file or directory: '/mnt/galaxyTools/galaxy-central/lib/galaxy/util/../../../tool-data/shared/ensembl/builds.txt' ERROR: Unable to read builds file: [Errno 2] No such file or directory: '/mnt/galaxyTools/galaxy-central/lib/galaxy/util/../../../tool-data/shared/ncbi/builds.txt' Stderr WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'blastxml': 'module' object has no attribute 'BlastXml' WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.xml:BlastXml' to sniff_order: 'module' object has no attribute 'BlastXml' Using toolshed installed Abyss and get the same error in Stdout. Stdout ERROR: Unable to read builds file: [Errno 2] No such file or directory: '/mnt/galaxyTools/galaxy-central/lib/galaxy/util/../../../tool-data/shared/ucsc/publicbuilds.txt' ERROR: Unable to read builds file: [Errno 2] No such file or directory: '/mnt/galaxyTools/galaxy-central/lib/galaxy/util/../../../tool-data/shared/ensembl/builds.txt' ERROR: Unable to read builds file: [Errno 2] No such file or directory: '/mnt/galaxyTools/galaxy-central/lib/galaxy/util/../../../tool-data/shared/ncbi/builds.txt' Stderr COMMAND FAILURE: /mnt/galaxyTools/shed_tools/toolshed.g2.bx.psu.edu/repos/msjeon/abyss_tool/ce99b6666d1f/abyss_tool/abyss_toolsuite-92636934a189/abyss/abyss_wrapper.sh: line 9: ABYSS: command not found WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'blastxml': 'module' object has no attribute 'BlastXml' WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.xml:BlastXml' to sniff_order: 'module' object has no attribute 'BlastXml' Looking at the util directory based on the expected path of the read build file galaxy@ip-10-110-143-10:/mnt/galaxyTools/galaxy-central/lib/galaxy/util$ ls aliaspickler.py docutils_ext heartbeat.py json.pyc odict.py streamball.py aliaspickler.pyc docutils_template.txt inflection.py lrucache.py odict.pyc streamball.pyc backports expressions.py inflection.pyc lrucache.pyc sanitize_html.py template.py bunch.py expressions.pyc __init__.py memdump.py sanitize_html.pyc template.pyc bunch.pyc hash_util.py __init__.pyc none_like.py shed_util.py topsort.py debugging.py hash_util.pyc json.py none_like.pyc shed_util.pyc topsort.pyc

2 12

BWA error post upgrade - local galaxy
by Kelkar, Hemant 08 Nov '12

08 Nov '12

We recently upgraded our local galaxy install and followed the procedure that now uses "bwa" from the galaxy toolshed. Since the upgrade "bwa" alignments are generating the following error (exact file paths redacted): [bwt_restore_bwt] fail to open file '/path_to/bwa_path/mm9_all.fa.rbwt'. Abort! /bin/sh: line 1: 14044 Aborted (core dumped) bwa aln -t 4 /path_to/bwa_path/mm9_all.fa /path_to/galaxy/002/dataset_2377.dat > /path_to/galaxy-prod/galaxy-dist/database/tmp/tmpxb3Bvr/tmpJeewN0 We are using indexes made by bwa (0.6.0 also tried 0.6.2) but I do not see the "*.rbwt" file mentioned above in the indexes we have. BWA was working prior to the upgrade with the indexes currently in place. We are using "LSF" for job queuing on the cluster. All other tools appear to be working fine. Thanks, Hemant

2 1

Manage libraries permissions with API
by Fabien Mareuil 08 Nov '12

08 Nov '12

Dear all, I would like to know if it is possible to manage libraries permissions with the API galaxy, and how to do that. Thank you for your answer, Best regards, Fabien Mareuil -- Fabien Mareuil | Centre d'Informatique pour la Biologie fabien.mareuil(a)pasteur.fr | Institut Pasteur 25,28 rue du Docteur Roux 75015 Paris, France

1 0

Purge datasets without purging the associated history?
by Luobin Yang 08 Nov '12

08 Nov '12

Hi, guys, Is it possible to purge some datasets in a history without purging the whole history? Thanks, Luobin

2 1

can't use bowtie mapper
by Shuqing Xu 08 Nov '12

08 Nov '12

Dear Galaxy developers, I am using a locally installed galaxy system. I have tested many different function, it works pretty well. But now i am having a problem with NGS tools. I have installed bowtie2 and tophat2 etc locally, but when i run bowtie to align the short reads to reference genome (selected from history), it does not work, just shows waiting for running. I then checked the files from bowtie2_wrapper.py with my own files as following under command line: python2.7 bowtie2_wrapper.py --num-threads=4 --output=test.out --own-file=Nagenome.fasta --input1=control_R1.fq --input2=control_R2.fq However, it didn't run well, it shows following error message: Settings: Output files: "/tmp/tmplBdrjc/Nagenome.*.bt2" Line rate: 6 (line is 64 bytes) Lines per side: 1 (side is 64 bytes) Offset rate: 4 (one in 16) FTable chars: 10 Strings: unpacked Max bucket size: default Max bucket size, sqrt multiplier: default Max bucket size, len divisor: 4 Difference-cover sample period: 1024 Endianness: little Actual local endianness: little Sanity checking: disabled Assertions: disabled Random seed: 0 Sizeofs: void*:8, int:4, long:8, size_t:8 Input files DNA, FASTA: Nagenome.fasta Total time for call to driver() for forward index: 00:00:00 Error indexing reference sequence Error: could not open Nagenome.fasta Error: Encountered internal Bowtie 2 exception (#1) Command: bowtie2-build -f Nagenome.fasta /tmp/tmplBdrjc/Nagenome I also tested with pre-build reference genome index, it worked well. So, i guess there is something wrong with building index. However, i couldn't figure out what could be the reason. The genome size is pretty small, only 2000 sequences. I am looking forward to hearing your feedback and helps. best wishes, Shuqing -- ################################################# Department of Molecular Ecology Max Planck Institute for Chemical Ecology Hans-Knöll-Straße 8 D-07745 Jena Germany E-mail: sxu(a)ice.mpg.de Phone:+49 (0)3641 57 1129 #################################################

2 1

Dataset upload fail
by Vladimir Yamshchikov 07 Nov '12

07 Nov '12

Local install of Galaxy on SciLinux55. Fails to upload 5.2 GB fastq file from local HD, while normally loading smaller fastq and fasta datasets (less than 1 GB). Chunks of 1.2 GB size remain in */database/tmp, which all represent beginning of the file that fails to upload. Several attempts to upload made and several chunks of the same size are present. Can I just copy the file dataset to database directory instead of uploading through web interface? This shows up when clicking the button "Run this job again" ⇝ Exception: Failed to get job information for dataset hid 5 clear this clear this URL: http://127.0.0.1:8080/tool_runner/rerun?id=8 Module weberror.evalexception.middleware:364 in respond view << try: __traceback_supplement__ = errormiddleware.Supplement, self, environ app_iter = self.application(environ, detect_start_response) try: return_iter = list(app_iter)>> app_iter = self.application(environ, detect_start_response) Module paste.debug.prints:98 in __call__ view << try: status, headers, body = wsgilib.intercept_output( environ, self.app) if status is None: # Some error occurred>> environ, self.app) Module paste.wsgilib:539 in intercept_output view << data.append(headers) return output.write app_iter = application(environ, replacement_start_response) if data[0] is None: return (None, None, app_iter)>> app_iter = application(environ, replacement_start_response) Module paste.recursive:80 in __call__ view << environ['paste.recursive.script_name'] = my_script_name try: return self.application(environ, start_response) except ForwardRequestException, e: middleware = CheckForRecursionMiddleware(>> return self.application(environ, start_response) Module paste.httpexceptions:632 in __call__ view << []).append(HTTPException) try: return self.application(environ, start_response) except HTTPException, exc: return exc(environ, start_response)>> return self.application(environ, start_response) Module galaxy.web.framework.base:160 in __call__ view << kwargs.pop( '_', None ) try: body = method( trans, **kwargs ) except Exception, e: body = self.handle_controller_exception( e, trans, **kwargs )>> body = method( trans, **kwargs ) Module galaxy.webapps.galaxy.controllers.tool_runner:129 in rerun view << job = data.creating_job if not job: raise Exception("Failed to get job information for dataset hid %d" % data.hid) # Get the tool object tool_id = job.tool_id>> raise Exception("Failed to get job information for dataset hid %d" % data.hid) Exception: Failed to get job information for dataset hid 5 URL: http://127.0.0.1:8080/tool_runner/rerun?id=8 File '/home/yaximik/galaxy-dist/eggs/WebError-0.8a-py2.7.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/home/yaximik/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/home/yaximik/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/home/yaximik/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/home/yaximik/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/home/yaximik/galaxy-dist/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/home/yaximik/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/tool_runner.py', line 129 in rerun raise Exception("Failed to get job information for dataset hid %d" % data.hid) Exception: Failed to get job information for dataset hid 5 <?xml version="1.0" ?> <traceback> <sysinfo> <language version="2.7"> Python </language> </sysinfo> <stack> <frame> <module> weberror.evalexception.middleware </module> <filename> /home/yaximik/galaxy-dist/eggs/WebError-0.8a-py2.7.egg/weberror/evalexception/middleware.py </filename> <line> 364 </line> <function> respond </function> <operation> app_iter = self.application(environ, detect_start_response) </operation> <operation_context> try: __traceback_supplement__ = errormiddleware.Supplement, self, environ app_iter = self.application(environ, detect_start_response) try: return_iter = list(app_iter) </operation_context> </frame> <frame> <module> paste.debug.prints </module> <filename> /home/yaximik/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/debug/prints.py </filename> <line> 98 </line> <function> __call__ </function> <operation> environ, self.app) </operation> <operation_context> try: status, headers, body = wsgilib.intercept_output( environ, self.app) if status is None: # Some error occurred </operation_context> </frame> <frame> <module> paste.wsgilib </module> <filename> /home/yaximik/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/wsgilib.py </filename> <line> 539 </line> <function> intercept_output </function> <operation> app_iter = application(environ, replacement_start_response) </operation> <operation_context> data.append(headers) return output.write app_iter = application(environ, replacement_start_response) if data[0] is None: return (None, None, app_iter) </operation_context> </frame> <frame> <module> paste.recursive </module> <filename> /home/yaximik/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/recursive.py </filename> <line> 80 </line> <function> __call__ </function> <operation> return self.application(environ, start_response) </operation> <operation_context> environ['paste.recursive.script_name'] = my_script_name try: return self.application(environ, start_response) except ForwardRequestException, e: middleware = CheckForRecursionMiddleware( </operation_context> </frame> <frame> <module> paste.httpexceptions </module> <filename> /home/yaximik/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/httpexceptions.py </filename> <line> 632 </line> <function> __call__ </function> <operation> return self.application(environ, start_response) </operation> <operation_context> []).append(HTTPException) try: return self.application(environ, start_response) except HTTPException, exc: return exc(environ, start_response) </operation_context> </frame> <frame> <module> galaxy.web.framework.base </module> <filename> /home/yaximik/galaxy-dist/lib/galaxy/web/framework/base.py </filename> <line> 160 </line> <function> __call__ </function> <operation> body = method( trans, **kwargs ) </operation> <operation_context> kwargs.pop( '_', None ) try: body = method( trans, **kwargs ) except Exception, e: body = self.handle_controller_exception( e, trans, **kwargs ) </operation_context> </frame> <frame> <module> galaxy.webapps.galaxy.controllers.tool_runner </module> <filename> /home/yaximik/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/tool_runner.py </filename> <line> 129 </line> <function> rerun </function> <operation> raise Exception("Failed to get job information for dataset hid %d" % data.hid) </operation> <operation_context> job = data.creating_job if not job: raise Exception("Failed to get job information for dataset hid %d" % data.hid) # Get the tool object tool_id = job.tool_id </operation_context> </frame> </stack> <exception> <type> Exception </type> <value> Failed to get job information for dataset hid 5 </value> </exception> </traceback> <?xml version="1.0" ?> <traceback> <sysinfo> <language version="2.7"> Python </language> </sysinfo> <stack> <frame> <module> weberror.evalexception.middleware </module> <filename> /home/yaximik/galaxy-dist/eggs/WebError-0.8a-py2.7.egg/weberror/evalexception/middleware.py </filename> <line> 364 </line> <function> respond </function> <operation> app_iter = self.application(environ, detect_start_response) </operation> <operation_context> try: __traceback_supplement__ = errormiddleware.Supplement, self, environ app_iter = self.application(environ, detect_start_response) try: return_iter = list(app_iter) </operation_context> </frame> <frame> <module> paste.debug.prints </module> <filename> /home/yaximik/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/debug/prints.py </filename> <line> 98 </line> <function> __call__ </function> <operation> environ, self.app) </operation> <operation_context> try: status, headers, body = wsgilib.intercept_output( environ, self.app) if status is None: # Some error occurred </operation_context> </frame> <frame> <module> paste.wsgilib </module> <filename> /home/yaximik/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/wsgilib.py </filename> <line> 539 </line> <function> intercept_output </function> <operation> app_iter = application(environ, replacement_start_response) </operation> <operation_context> data.append(headers) return output.write app_iter = application(environ, replacement_start_response) if data[0] is None: return (None, None, app_iter) </operation_context> </frame> <frame> <module> paste.recursive </module> <filename> /home/yaximik/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/recursive.py </filename> <line> 80 </line> <function> __call__ </function> <operation> return self.application(environ, start_response) </operation> <operation_context> environ['paste.recursive.script_name'] = my_script_name try: return self.application(environ, start_response) except ForwardRequestException, e: middleware = CheckForRecursionMiddleware( </operation_context> </frame> <frame> <module> paste.httpexceptions </module> <filename> /home/yaximik/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/httpexceptions.py </filename> <line> 632 </line> <function> __call__ </function> <operation> return self.application(environ, start_response) </operation> <operation_context> []).append(HTTPException) try: return self.application(environ, start_response) except HTTPException, exc: return exc(environ, start_response) </operation_context> </frame> <frame> <module> galaxy.web.framework.base </module> <filename> /home/yaximik/galaxy-dist/lib/galaxy/web/framework/base.py </filename> <line> 160 </line> <function> __call__ </function> <operation> body = method( trans, **kwargs ) </operation> <operation_context> kwargs.pop( '_', None ) try: body = method( trans, **kwargs ) except Exception, e: body = self.handle_controller_exception( e, trans, **kwargs ) </operation_context> </frame> <frame> <module> galaxy.webapps.galaxy.controllers.tool_runner </module> <filename> /home/yaximik/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/tool_runner.py </filename> <line> 129 </line> <function> rerun </function> <operation> raise Exception("Failed to get job information for dataset hid %d" % data.hid) </operation> <operation_context> job = data.creating_job if not job: raise Exception("Failed to get job information for dataset hid %d" % data.hid) # Get the tool object tool_id = job.tool_id </operation_context> </frame> </stack> <exception> <type> Exception </type> <value> Failed to get job information for dataset hid 5 </value> </exception> </traceback> -- Vladimir Yamshchikov <yaximik(a)gmail.com>

3 3

Think I found a bug
by Juan González-Vallinas 07 Nov '12

07 Nov '12

Hi galaxy devs, We are in the final stages of deploying a new Galaxy instance. We downloaded the galaxy-dist version a month ago aprox. We realized that, when launching a job and deleting it in the middle of the execution (hitting the Delete button in the history bar) will output the following error in the logger: galaxy.jobs.handler DEBUG 2012-11-05 15:56:33,537 Stopping job 45: galaxy.jobs.handler DEBUG 2012-11-05 15:56:33,537 stopping job 45 in local runner galaxy.jobs.handler ERROR 2012-11-05 15:56:33,538 Exception in monitor_step Traceback (most recent call last): File "/usr/local/www/galaxy-dist/lib/galaxy/jobs/handler.py", line 298, in monitor self.monitor_step() File "/usr/local/www/galaxy-dist/lib/galaxy/jobs/handler.py", line 341, in monitor_step self.dispatcher.stop( job ) File "/usr/local/www/galaxy-dist/lib/galaxy/jobs/handler.py", line 438, in stop self.job_runners[runner_name].stop_job( job ) File "/usr/local/www/galaxy-dist/lib/galaxy/jobs/runners/local.py", line 180, in stop_job if job.get_external_output_metadata(): AttributeError: 'Job' object has no attribute 'get_external_output_metadata' By copying the definition of get_external_output_data from the Task class into the Job class, we fixed the problem and now the jobs are terminated successfully. This is the code I added: class Job( object ): ... def get_external_output_metadata( self ): return None ... In this file: https://bitbucket.org/galaxy/galaxy-central/src/9d6a61f060d359a0289b3163990… Do you think this fix is correct, or is it a potential problem? Also, sorry I sent you through this channel, I couldn't find the way of reporting this on BitBucket. Regards, -- Juan González-Vallinas PhD Student Regulatory Genomics Group Research Unit in Biomedical Informatics (GRIB) Universitat Pompeu Fabra (UPF) Barcelona Spain (Currently @ UPenn)

3 4

Syntax error (!) in latest pull?!
by Anthonius deBoer 07 Nov '12

07 Nov '12

Got this in the latest pull of galaxy central... Ideas? Traceback (most recent call last): File "./scripts/set_metadata.py", line 24, in ? import galaxy.model.mapping #need to load this before we unpickle, in order to setup properties assigned by the mappers File "/mnt/ngs/analysis/svcgalaxy/galaxy-central/lib/galaxy/model/__init__.py", line 13, in ? import galaxy.datatypes.registry File "/mnt/ngs/analysis/svcgalaxy/galaxy-central/lib/galaxy/datatypes/registry.py", line 145 finally: ^ SyntaxError: invalid syntax

5 5

missing BWA tools on AWS AMI
by Quang Trinh 07 Nov '12

07 Nov '12

Hi dev, I launched an instance of Galaxy on Amazon ( AMI ami-da58aab3 ) this morning and noticed bwa is no longer available. Is this temporary or if we want to use bwa, we have to add the bwa ourselves? Thanks, Q

2 1