September 2014 - galaxy-dev - lists.galaxyproject.org

BAMtools in Galaxy
by Anton Nekrutenko 11 Sep '14

11 Sep '14

Derek: Just to let you know that I just wrapped BAMTools for Galaxy: GitHub -> https://github.com/nekrut/bamtools GalaxyToolShed -> https://toolshed.g2.bx.psu.edu/view/anton/bamtools Galaxt Test Server -> https://test.galaxyproject.org (look for BAMTools on the left pane). We will roll them out on the main site soon. Thanks for a great package! anton -- Anton Nekrutenko Professor of Biochemistry and Molecular Biology http://nekrut.bx.psu.edu (814) 826-3051

1 0

Keeping Galaxy up to date
by Thomas Bellembois 11 Sep '14

11 Sep '14

Hello, I try to keep our Galaxy instance up to date. Very often, some tools disappear and other do not work anymore. This breaks some users workflows. I may miss something. Is there a "best update pratice" to keep a Galaxy instance and the tool-sheds fully functionnal ? Regards, Thomas -- Thomas Bellembois, Network and System Administrator, IGFL (France) http://perso.ens-lyon.fr/thomas.bellembois - +33 4 26 73 13 67 (IGFL internal IT doc: http://itdoc.igfl.ens-lyon.fr/itdoc)

4 6

ToolShed test failure: NotFound: cannot find 'ucsc_display_sites' while searching for 'APP.config.ucsc_display_sites'
by Peter Cock 11 Sep '14

11 Sep '14

Hi all, I'm wondering why my samtools_depad repository tests have failed, and since I have not changed this recently presume this is due to a Galaxy change or general TestToolShed problem not specific to my tool: https://testtoolshed.g2.bx.psu.edu/view/peterjc/samtools_depad Tests that failed Tool id: samtools_depad Tool version: samtools_depad Test: test_tool_000000 (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterj…) Stderr: Traceback: Traceback (most recent call last): File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 114, in test_tool self.do_it( td ) File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 35, in do_it self._verify_outputs( testdef, test_history, shed_tool_id, data_list, galaxy_interactor ) File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 75, in _verify_outputs galaxy_interactor.verify_output( history, output_data, output_testdef=output_testdef, shed_tool_id=shed_tool_id, maxseconds=maxseconds ) File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/base/interactor.py", line 82, in verify_output self._verify_metadata( history_id, hid, attributes ) File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/base/interactor.py", line 103, in _verify_metadata raise Exception( msg ) Exception: Dataset metadata verification for [file_ext] failed, expected [bam] but found [None]. Traceback (most recent call last): File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/web/framework/decorators.py", line 243, in decorator rval = func( self, trans, *args, **kwargs) File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/webapps/galaxy/api/history_contents.py", line 188, in show return self.__show_dataset( trans, id, **kwd ) File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/webapps/galaxy/api/history_contents.py", line 214, in __show_dataset hda_dict[ 'display_apps' ] = self.get_display_apps( trans, hda ) File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/web/base/controller.py", line 855, in get_display_apps for display_app in hda.get_display_applications( trans ).itervalues(): File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/model/__init__.py", line 1754, in get_display_applications return self.datatype.get_display_applications_by_dataset( self, trans ) File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/datatypes/data.py", line 445, in get_display_applications_by_dataset value = value.filter_by_dataset( dataset, trans ) File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/datatypes/display_applications/application.py", line 200, in filter_by_dataset if link_value.filter_by_dataset( data, trans ): File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/datatypes/display_applications/application.py", line 78, in filter_by_dataset if fill_template( filter_elem.text, context = context ) != filter_elem.get( 'value', 'True' ): File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/util/template.py", line 9, in fill_template return str( Template( source=template_text, searchList=[context] ) ) File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/eggs/Cheetah-2.2.2-py2.7-linux-x86_64-ucs4.egg/Cheetah/Template.py", line 1004, in __str__ return getattr(self, mainMethName)() File "cheetah_DynamicallyCompiledCheetahTemplate_1410263883_33_43576.py", line 82, in respond NotFound: cannot find 'ucsc_display_sites' while searching for 'APP.config.ucsc_display_sites' requests.packages.urllib3.connectionpool: DEBUG: "GET /api/histories/993bad2fe35335db/contents/7fbe67cfae825002?key=edc04240db9605fb7edc7bab44d3404c HTTP/1.1" 500 None requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 127.0.0.1 requests.packages.urllib3.connectionpool: DEBUG: "GET /api/histories/993bad2fe35335db/contents/7fbe67cfae825002/provenance?key=edc04240db9605fb7edc7bab44d3404c HTTP/1.1" 200 None requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 127.0.0.1 requests.packages.urllib3.connectionpool: DEBUG: "GET /api/histories/993bad2fe35335db/contents/7fbe67cfae825002/provenance?key=edc04240db9605fb7edc7bab44d3404c HTTP/1.1" 200 None Any thoughts? Thanks, Peter

2 2

problems with the varscan somatic tool
by Karin Schlangen 11 Sep '14

11 Sep '14

Dear Galaxy team, i tried to run the varscan somatic tool from fcaramia on two pileup files but i got the following error: Unknown Input: COMMAND::java -jar /home/karin/galaxy/dependency_dir/VarScan/2.3.5/fcaramia/varscan/51969e284317/jars/VarScan.v2.3.5.jar somatic The command was the following: Job Command-Line: perl /home/karin/shed_tools/ toolshed.g2.bx.psu.edu/repos/fcaramia/varscan/51969e284317/varscan/varscan_… "COMMAND::java -jar $JAVA_JAR_PATH/VarScan.v2.3.5.jar somatic" "NORMAL::/home/karin/galaxy-dist/database/files/000/dataset_107.dat" "TUMOR::/home/karin/galaxy-dist/database/files/000/dataset_109.dat" "OUTPUT::/home/karin/galaxy-dist/database/files/000/dataset_145.dat" "OPTION::--min-coverage 8" "OPTION::--min-coverage-normal 8" "OPTION::--min-coverage-tumor 8" "OPTION::--min-var-freq 0.1" "OPTION::--min-freq-for-hom 0.75" "OPTION::--normal-purity 1.0" "OPTION::--tumor-purity 1.0" "OPTION::--p-value 0.99" "OPTION::--somatic-p-value 0.05" "OPTION::--strand-filter 1" "OPTION::--validation 0" "OPTION::--output-vcf 1" I also tried to change the varscan_somatic.xml file in such a way that i removed the "COMMAND::" tags etc. and used it without the varscan_somatic.pl file. After doing that, the tool ran without errors but i obtained empty files. Redoing the same command on my local computer ran properly and i obtained the wanted outputs: java -jar $JAVA_JAR_PATH/VarScan.v2.3.5.jar somatic /home/karin/galaxy-dist/database/files/000/dataset_107.dat /home/karin/galaxy-dist/database/files/000/dataset_109.dat dataset_131.dat --output-snp dataset_131.dat --output-indel dataset_132.dat --min-coverage 8 --min-coverage-normal 8 --min-coverage-tumor 8 --min-var-freq 0.1 --min-freq-for-hom 0.75 --normal-purity 1.0 --tumor-purity 1.0 --p-value 0.99 --somatic-p-value 0.05 --strand-filter 1 2> testlog in dataset_131.dat the snps, indels: dataset_132.dat, and some information (varscan directs this info on stderr) in testlog. Any idea how to fix that? i appreciate any information on how this may be fixed! Thanks a lot and best wishes, Karin

1 0

BAM to SAM tool no way to get an unsorted SAM?
by Alistair Chilcott 10 Sep '14

10 Sep '14

Hello all, My users are trying to use a tool called bismark and it requires an unsorted SAM file for one of its steps Previous steps produce BAM files so we would like to include the tool BAM-to-SAM as a step in the workflow unfortunately it seems to be set to sort by default and there is no way to alter this behaviour. Is there another tool that would convert a BAM to a SAM without sorting it or can the BAM-to-SAM tool be altered to include a checkbox for an unsorted result? Regards, Alistair

2 4

How to escape % characters in tool parameters?
by Melissa Cline 09 Sep '14

09 Sep '14

Hi folks, I'm working on a tool that queries a database based on user input, where this user input is a text parameter. The wildcard for the database is the percent (%) character. The user input, which is ultimately translated into the query, should be able to contain a % character. For example, one input field lets the user indicate which cancer cohort he or she is interested in, in a table of available cohorts, and the user should be able to give input like 'TCGA%" to indicate all TCGA cohorts. The problem I'm having is that when % is given in the input field, it's translated (to X) by the time it reaches my tool's python script. I haven't been able to figure out the right way to escape the input so that % is not translated. So, how do I escape this? Thanks! Melissa

2 2

Error executing workflow via API
by Neil.Burdett＠csiro.au 08 Sep '14

08 Sep '14

Hi, I execute a number of workflows via the API which all work fine. However, the longest one I use (consists of three linked tasks), returns an error. I call the workflow in the python script using popen and has the format: /home/galaxy/milxcloud-new/scripts/api/workflow_execute.py d2fcd3feb4c6318c496d55fa8869b67c http://barium-rbh/new/api/workflows f597429621d6eb2b hist_id=f597429621d6eb2b 4=hda=c8c00aa41dc69085 The task appears on the history panel and all tasks eventually complete in the workflow, however, the task that calls the workflow using the above popen call returns the following error: Traceback (most recent call last): File "/home/galaxy/milxcloud-new/scripts/api/workflow_execute.py", line 31, in <module> main() File "/home/galaxy/milxcloud-new/scripts/api/workflow_execute.py", line 28, in main submit( sys.argv[1], sys.argv[2], data ) File "/home/galaxy/milxcloud-new/scripts/api/common.py", line 117, in submit r = post( api_key, url, data ) File "/home/galaxy/milxcloud-new/scripts/api/common.py", line 51, in post return json.loads( urllib2.urlopen( req ).read() ) File "/usr/lib/python2.7/socket.py", line 351, in read data = self._sock.recv(rbufsize) File "/usr/lib/python2.7/httplib.py", line 541, in read return self._read_chunked(amt) File "/usr/lib/python2.7/httplib.py", line 586, in _read_chunked raise IncompleteRead(''.join(value)) httplib.IncompleteRead: IncompleteRead(147 bytes read) This worked on older versions of Galaxy. I need this to work so I can "monitor" the workflow so know when all tasks are complete. has anyone seen this error? Or got an idea how to solve it. hg summary reports: % hg summary parent: 13771:7a4d321c0e38 tip Updated tag latest_2014.06.02 for changeset 8c30e91bc9ae branch: stable commit: (clean) update: (current) Thanks Neil

2 3

Output Excel file
by Patrick Leyshock 08 Sep '14

08 Sep '14

Hello, I am trying to write a Galaxy tool that will output an Excel file. Currently the tool wrapper calls an R script, which uses the "xlsx" package to read and write to and from Excel files. After being invoked by Galaxy, the script is able to successfully read an input Excel file: suppressPackageStartupMessages(library(xlsx, quietly=TRUE)); raw.data <- read.xlsx(commandArgs(trailingOnly=TRUE)[1]), sheetName="input_records"); The script then does its work on the data just read in. Then, when the work is done, I'd like to output the results to an Excel file. Here's where I run into a problem. I first tried to output the result like this: write.xlsx(processed.data, file=commandArgs(trailingOnly=TRUE)[2]); but write.xlsx throws an error. Looks like ".xls" and ".xlsx" are the only legal file extensions for the write.xlsx() function. Inspecting commandArgs(trailingOnly=TRUE)[2] shows that Galaxy provided a filename extension of ".dat". I tried a workaround using this: write.xlsx(processed.data, file=paste(commandArgs(trailingOnly=TRUE)[2]), ".xls", sep=""); The write.xlsx function no longer throws an error (since the file name supplied as a parameter has an acceptable file extension) but now Galaxy won't display the result in the History. If I look in Galaxy's database I see two files there corresponding to my tool's output. Supposing that Galaxy assigned the result the name "dataset_87", then looking in the Galaxy database I see: dataset_87.dat dataset_87.dat.xls "dataset_87.dat" is empty but visible to Galaxy (and so displayed as an empty dataset in the History window). "dataset_87.dat.xls" has the results I want but isn't displayed by Galaxy in the History window. There might be R libraries that can write xls or xlsx files without requiring a ".xls" or ".xlsx" file extension. That's a path I'm willing to explore. That said, is there a configuration option I can set that'll let me continue to use write.xlsx()? I've been working from the suggestions made on this helpful thread: http://lists.bx.psu.edu/pipermail/galaxy-dev/2011-December/007807.html so the relevant entries in datatypes_conf.xml are: <datatype extension="xls" type="galaxy.datatypes.binary.Xls" display_in_upload="true" /> <datatype extension="xlsx" type="galaxy.datatypes.binary.Xlsx" display_in_upload="true" /> and the relevant additions to binary.py are: class Xls(Binary): '''Class describing an Excel 2003 (xls) file''' file_ext='xls' class Xlsx(Binary): '''Class describing an Excel 2007 (xlsx) file''' file_ext='xlsx' Any suggestions appreciated. Thanks, Patrick

3 3

Re: [galaxy-dev] Concept for a Galaxy Versioned Fasta Data Retrieval Tool
by Dooley, Damion 05 Sep '14

05 Sep '14

Well well, thanks very much for that reference! I can see how your system to enable a workflow to process delta (diff) data (and merge the results back with a previous run's output) would greatly lighten the processing power for keeping results current. Interesting choice of technologies too. Damion Message: 5 Date: Fri, 5 Sep 2014 10:41:37 +0000 From: Pedersen Edvard <edvard.pedersen(a)uit.no> To: "galaxy-dev(a)lists.bx.psu.edu" <galaxy-dev(a)lists.bx.psu.edu> Subject: Re: [galaxy-dev] Concept for a Galaxy Versioned Fasta Data Retrieval Tool My PhD work may be of interest for this subject, although the primary focus has been on generating databases comprising the changes from a specific timeframe, and was not designed specifically for Galaxy. The similarities between my system and the system you are proposing are that it can generate a BLAST database from any date (that has been added to the system), as well as "diffs" between two dates, and supports FASTA, the Uniprot EMBL variant, full files (which does not give compression benefits) and several others. The system uses delta compression to make sure that non-updated fields do not take up extra space. It uses the Hadoop stack (HBase, HDFS and MapReduce) for parallelism in generating the databases (the blast database generation from FASTA files is not parallel). You can find one of the publications here: <http://bdps.cs.uit.no/papers/hibb13.pdf> http://bdps.cs.uit.no/papers/hibb13.pdf I hope this can be of some use to you. Regards, Edvard Pedersen

1 0

Join tool failure
by Iry Witham 05 Sep '14

05 Sep '14

Hi Team, I have upgraded to the latest galaxy distribution on my test server. Now I am running into issues and can't figure this one out. I am attempting to run the join.py tool to join two datasets at specified fields. I am getting the following error: Traceback (most recent call last): File "/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/tools/filters/join.py", line 18, in <module> from galaxy.util.bunch import Bunch File "/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/galaxy/__init__.py", line 95, in <module> import galaxy.eggs File "/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/galaxy/eggs/__init__.py", line 5, in <module> import os, sys, shutil, glob, urllib, urllib2, ConfigParser, HTMLParser, zipimport, zipfile File "/usr/lib64/python2.6/urllib2.py", line 93, in <module> import hashlib File "/usr/lib64/python2.6/hashlib.py", line 88, in <module> import _hashlib ImportError: /usr/lib64/libssl.so.10: symbol EC_KEY_get0_group, version OPENSSL_1.0.1_EC not defined in file libcrypto.so.10 with link time reference Traceback (most recent call last): File "./scripts/set_metadata.py", line 29, in <module> from galaxy import eggs File "/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/galaxy/__init__.py", line 95, in <module> import galaxy.eggs File "/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/galaxy/eggs/__init__.py", line 5, in <module> import os, sys, shutil, glob, urllib, urllib2, ConfigParser, HTMLParser, zipimport, zipfile File "/usr/lib64/python2.6/urllib2.py", line 93, in <module> import hashlib File "/usr/lib64/python2.6/hashlib.py", line 88, in <module> import _hashlib ImportError: /usr/lib64/libssl.so.10: symbol EC_KEY_get0_group, version OPENSSL_1.0.1_EC not defined in file libcrypto.so.10 with link time reference I am at a loss on the resolution for this. Thanks, Iry The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.

2 1