April 2015 - galaxy-dev - lists.galaxyproject.org

fml_gff3togtf: GTF-to-BED failes on numpy in Galaxy
by Curtis Hendrickson (Campus) 06 Apr '15

06 Apr '15

Folks I installed this very useful looking tool repository (fml_gff3togtf<https://toolshed.g2.bx.psu.edu/view/vipints/fml_gff3togtf>) to convert GFF3->BED12 based on this BioStars. https://www.biostars.org/p/85869 It installed fine, but when I run it on ftp://ftp.ncbi.nih.gov/genomes/Viruses/Human_herpesvirus_5_uid14559/NC_0062… I get Traceback (most recent call last): File "/share/apps/galaxy/test-shed-tools/toolshed.g2.bx.psu.edu/repos/vipints/fml_gff3togtf/6e589f267c14/fml_gff3togtf/gff_to_bed.py", line 18, in <module> import GFFParser File "/share/apps/galaxy/test-shed-tools/toolshed.g2.bx.psu.edu/repos/vipints/fml_gff3togtf/6e589f267c14/fml_gff3togtf/GFFParser.py", line 20, in <module> import scipy.io as sio ImportError: No module named scipy.io Anyone know the trick to get the python libraries set up correctly? (also note that this failed nightly tests on 2015-04-06 - but that looks like a lack of test data/configuration) I see some other packages (like htseq_count<https://toolshed.g2.bx.psu.edu/view/lparsons/htseq_count>) have numpy as a dependency.... Using the devteam's numpy 1.7 package: https://toolshed.g2.bx.psu.edu/view/devteam/package_numpy_1_7 Regards, Curtis Research Associate, Informatics Unit Center for Clinical and Translational Science http://bioinformatics.uab.edu University of Alabama at Birmingham

3 2

Workflow bug when *.loc entries are missing
by Peter Cock 06 Apr '15

06 Apr '15

Hi all, I was testing a new workflow which I developed on the current stable release, Galaxy 15.03, and uploaded to the Tool Shed yesterday: https://github.com/peterjc/galaxy_blast/tree/master/workflows/blast_top_hit… http://toolshed.g2.bx.psu.edu/view/peterjc/blast_top_hit_species I then switched to my development instance running the dev branch from github, 1. Log in as an Admin 2. Install http://toolshed.g2.bx.psu.edu/view/peterjc/blast_top_hit_species 3. Upload suggested test dataset (in workflow README file): http://nematode.net/Data/nacobbus_aberrans_transcript_assembly/N.abberans_r… 4. Click on "All Workflows" (bottom of left pane tool list) 5. Click on "Species of top BLAST hits (imported from uploaded file)" 6. On preview screen, examine step 3, blastx 7. Notice it will run against Protein BLAST database "nr" 8. Click "Run workflow" As noted in the README file, this workflow assumes the local blastdb_p.loc file includes an entry "nr" for a current/recent mirror of the NCBI "non-redundant" protein BLAST database. Initially my development galaxy had an empty blastdb_p.loc file, so I expected the workflow to spot this problem at the preview step (before ever clicking "run workflow"). Instead it proceeds and attempted to run BLASTX without no database (which of course failed): blastx -query "/mnt/galaxy/repositories/galaxy/database/files/000/dataset_81.dat" -db "" -query_gencode 1 -evalue 0.001 -out "/mnt/galaxy/repositories/galaxy/database/files/000/dataset_82.dat" -outfmt "6 qseqid sseqid pident length mismatch gapopen qstart qend sstart send evalue bitscore staxids sscinames scomnames sblastnames sskingdoms" -num_threads "${GALAXY_SLOTS:-8}" -strand both -matrix BLOSUM62 -seg yes -max_target_seqs 1 That's the bad news: It fails, but it fails with a reasonably clear error: Fatal error: Exit code 2 () BLAST Database error: Database name is required. Good news: If I update blastdb_p.loc to add the "nr" database it works. Very bad news: If I update blastdb_p.loc to add other databases (but not an entry for "nr"), the workflow defaults to the first database in blastdb_p.loc Summary: Workflow requesting key entry "xxx" from example.loc (a) example.loc empty, gets empty path, fails cleanly. (b) example.loc contains xxx, gets correct path, works. (c) example.loc non empty but lacks xxx, runs with WRONG path. Peter

4 10

changing usage indicator
by Nikhil Joshi 03 Apr '15

03 Apr '15

Hi all, Is there a way to change the "Using" indicator at the top right to be a Using/Total for the whole disk (not just Galaxy usage)? - Nik. -- Nikhil Joshi Bioinformatics Analyst/Programmer UC Davis Bioinformatics Core http://bioinformatics.ucdavis.edu/ najoshi -at- ucdavis -dot- edu 530.752.2698 (w)

2 3

NGINX uWSGI with require_login needs uwsgi_param SCRIPT_NAME '';
by David Trudgian 02 Apr '15

02 Apr '15

Hi, When I enabled 'require_login = True' on a galaxy instance running with NGINX+uWSGI I receive internal server error pages due to a Key error for File 'lib/galaxy/web/framework/base.py', line 356 in path return self.environ['SCRIPT_NAME'] + self.environ['PATH_INFO'] KeyError: 'SCRIPT_NAME' Works fine with 'require_login = False' - I'm not sure why require_login makes it take a code path which needs SCRIPT_NAME. Anyway - the addition of... uwsgi_param SCRIPT_NAME ''; ... to the nginx.conf prevents this error. Should I try and add this to the uWSGI config snippets on the wiki? Thanks, -- David Trudgian Ph.D., Computational Scientist UT Southwestern BioHPC -- David Trudgian Ph.D., Computational Scientist UT Southwestern BioHPC ________________________________ UT Southwestern Medical Center The future of medicine, today.

2 2

April! Galactic News!
by Dave Clements 02 Apr '15

02 Apr '15

1 0

using DataManager to add custom genomes
by Nikhil Joshi 02 Apr '15

02 Apr '15

Hi all, Is it possible to use a DataManager (or anything) to allow a user (or admin) to upload their genome of choice and have it be indexed and be used as a built-in/cached genome *without* having to go on the command-line? I.e., only using the Galaxy interface? - Nik. -- Nikhil Joshi Bioinformatics Analyst/Programmer UC Davis Bioinformatics Core http://bioinformatics.ucdavis.edu/ najoshi -at- ucdavis -dot- edu 530.752.2698 (w)

2 2

test/base/twilltestcase.py - Converting local (test-data) bam to sam failed
by Peter Cock 02 Apr '15

02 Apr '15

Hi all, I have a query about a failing tool test on the Tool Shed, where it seems Galaxy is trying to convert both the expected BAM output and the tool's BAM output into SAM for comparison - but this fails. https://toolshed.g2.bx.psu.edu/view/peterjc/samtools_depad 2015-01-27 02:13:51 ... Tool id: samtools_depad Tool version: samtools_depad Test: test_tool_000000 (functional.test_toolbox.TestForTool_toolshed.g2.bx.psu.edu/repos/peterjc/sa…) Stderr: Traceback: Traceback (most recent call last): File "/tmp/buildslave/buildbot-install-test-main-tool-shed-py27/build/test/functional/test_toolbox.py", line 122, in test_tool self.do_it( td ) File "/tmp/buildslave/buildbot-install-test-main-tool-shed-py27/build/test/functional/test_toolbox.py", line 40, in do_it self._verify_outputs( testdef, test_history, jobs, shed_tool_id, data_list, galaxy_interactor ) File "/tmp/buildslave/buildbot-install-test-main-tool-shed-py27/build/test/functional/test_toolbox.py", line 80, in _verify_outputs galaxy_interactor.verify_output( history, jobs, output_data, output_testdef=output_testdef, shed_tool_id=shed_tool_id, maxseconds=maxseconds ) File "/tmp/buildslave/buildbot-install-test-main-tool-shed-py27/build/test/base/interactor.py", line 66, in verify_output self.twill_test_case.verify_hid( outfile, hda_id=hid, attributes=attributes, dataset_fetcher=fetcher, shed_tool_id=shed_tool_id ) File "/tmp/buildslave/buildbot-install-test-main-tool-shed-py27/build/test/base/twilltestcase.py", line 2341, in verify_hid raise AssertionError( errmsg ) AssertionError: History item different than expected, difference (using diff): ( /tmp/shed_tools/toolshed.g2.bx.psu.edu/repos/peterjc/samtools_depad/2d303f2e09e0/samtools_depad/test-data/sam_spec_padded.depad.bam v. /tmp/buildslave/buildbot-install-test-main-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/repositories_with_tools/tmp/tmpExQWRD/tmpSknKqLsam_spec_padded.depad.bam ) Converting local (test-data) bam to sam failed (and same error on the second test) The same occurs on the Test Tool Shed (with a very slightly updated version of this tool), although with a more concise traceback: https://testtoolshed.g2.bx.psu.edu/view/peterjc/samtools_depad 2015-01-25 08:21:59 ... Tool id: samtools_depad Tool version: samtools_depad Test: test_tool_000000 (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterj…) Stderr: Traceback: Traceback (most recent call last): File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 268, in test_tool self.do_it( td ) File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 67, in do_it raise e JobOutputsError: History item different than expected, difference (using diff): ( /tmp/shed_tools/testtoolshed.g2.bx.psu.edu/repos/peterjc/samtools_depad/8e1e3a1ecad7/samtools_depad/test-data/sam_spec_padded.depad.bam v. /tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/repositories_with_tools/tmp/tmpNvCGJi/tmphX3A_Fsam_spec_padded.depad.bam ) Converting local (test-data) bam to sam failed Both tests should produce a BAM file matching the provided test-data/sam_spec_padded.depad.bam example. The error comes from test/base/twilltestcase.py, p = subprocess.Popen( args='samtools view -h -o "%s" "%s"' % ( temp_local.name, local_name ), shell=True ) assert not p.wait(), 'Converting local (test-data) bam to sam failed' Running this command locally at the terminal seems to work fine, both with samtools 0.1.19 and 1.1. $ ~/bin/samtools_0.1.19 view -h -o temp.sam sam_spec_padded.depad.bam $ more temp.sam @HD VN:1.5 SO:coordinate @SQ SN:ref LN:45 ref 516 ref 1 0 45M * 0 0 AGCATGTTAGATAAGATAGCTGTGCTAGTAGGCAGTCAGCGCCAT * r001 99 ref 7 30 8M2I4M1D3M = 37 41 TTAGATAAAGGATACTG * * 768 ref 8 30 1M * 0 0 * * CT:Z:.;Warning;Note=Ref wrong? r002 0 ref 9 30 3S6M1P1I4M * 0 0 AAAAGATAAGGATA * PT:Z:1;4;+;homopolymer r003 0 ref 9 30 5H6M * 0 0 AGCTAA * NM:i:1 r004 0 ref 16 30 6M14D5M * 0 0 ATAGCTTCAGC * r003 2064 ref 29 30 6H5M * 0 0 TAGGC * NM:i:0 r001 147 ref 37 30 9M = 7 -41 CAGCGGCAT * NM:i:1 $ ~/bin/samtools_1.1 view -h -o temp.sam sam_spec_padded.depad.bam [galaxy@ppserver test-data]$ more temp.sam @HD VN:1.5 SO:coordinate @SQ SN:ref LN:45 ref 516 ref 1 0 45M * 0 0 AGCATGTTAGATAAGATAGCTGTGCTAGTAGGCAGTCAGCGCCAT * r001 99 ref 7 30 8M2I4M1D3M = 37 41 TTAGATAAAGGATACTG * * 768 ref 8 30 1M * 0 0 * * CT:Z:.;Warning;Note=Ref wrong? r002 0 ref 9 30 3S6M1P1I4M * 0 0 AAAAGATAAGGATA * PT:Z:1;4;+;homopolymer r003 0 ref 9 30 5H6M * 0 0 AGCTAA * NM:i:1 r004 0 ref 16 30 6M14D5M * 0 0 ATAGCTTCAGC * r003 2064 ref 29 30 6H5M * 0 0 TAGGC * NM:i:0 r001 147 ref 37 30 9M = 7 -41 CAGCGGCAT * NM:i:1 Which version of samtools is on the $PATH on the test system? Could the test system have run out of disk space again? The tests are passing on my local development instance (before and after updating to latest galaxy-central), and on TravisCI using the latest Galaxy: https://travis-ci.org/peterjc/pico_galaxy/builds/48485673 Thanks, Peter

2 5

value_from_basic - KeyError: 'base_name', was: Tests not being run on toolsheds?
by Peter Cock 02 Apr '15

02 Apr '15

Hi Dave, The following looks like a regression on the Test Tool Shed, the tests pass locally (using a recent revision), and on TravisCI using the current galaxy dev branch on GitHub: https://travis-ci.org/peterjc/pico_galaxy/builds/54870500 Problem tool: https://testtoolshed.g2.bx.psu.edu/view/peterjc/sample_seqs This may be unrelated, but by chance this is the first time I have uploaded a tool to the Tool Shed which uses the new functionality to test the stdout/stderr strings, and more importantly it includes a test expected to fail via <test expect_failure="true" expect_exit_code="1"> Revision on my development repository: https://github.com/peterjc/pico_galaxy/commit/55ebb308b911b4acef912cc3b03f4… Test Tool Shed output from last night: Automated test environment Time tested: 2015-03-18 02:46:55 System: Linux 3.13.0-36-generic Architecture: x86_64 Python version: 2.7.6 Galaxy revision: 17050:6395e7035143 Galaxy database version: 128 Tool shed revision: 16867:0468d285f89c Tool shed database version: 25 Tool shed mercurial version: 3.2.4 Tests that failed Tool id: sample_seqs Tool version: sample_seqs Test: test_tool_000000 (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterj…) Stderr: Traceback: Traceback (most recent call last): File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 268, in test_tool self.do_it( td ) File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 37, in do_it stage_data_in_history( galaxy_interactor, testdef.test_data(), test_history, shed_tool_id ) File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/base/interactor.py", line 38, in stage_data_in_history upload_wait() File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/base/interactor.py", line 279, in wait while not self.__history_ready( history_id ): File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/base/interactor.py", line 297, in __history_ready return self._state_ready( state, error_msg="History in error state." ) File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/base/interactor.py", line 356, in _state_ready raise Exception( error_msg ) Exception: History in error state. Traceback (most recent call last): File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/jobs/runners/__init__.py", line 158, in prepare_job job_wrapper.prepare() File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/jobs/__init__.py", line 828, in prepare tool_evaluator.set_compute_environment( compute_environment, get_special=get_special ) File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/tools/evaluation.py", line 53, in set_compute_environment incoming = self.tool.params_from_strings( incoming, self.app ) File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/tools/__init__.py", line 1873, in params_from_strings return params_from_strings( self.inputs, params, app, ignore_errors ) File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/tools/parameters/__init__.py", line 103, in params_from_strings value = params[key].value_from_basic( value, app, ignore_errors ) File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/tools/parameters/grouping.py", line 512, in value_from_basic raise e KeyError: 'base_name' requests.packages.urllib3.connectionpool: DEBUG: "GET /api/histories/c33ffb67c9176a6c?key=ae52a59b996b793c4c06a11fc3c33481 HTTP/1.1" 200 None requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 127.0.0.1 galaxy.datatypes.metadata: DEBUG: Cleaning up external metadata files requests.packages.urllib3.connectionpool: DEBUG: "GET /api/histories/c33ffb67c9176a6c?key=ae52a59b996b793c4c06a11fc3c33481 HTTP/1.1" 200 None requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 127.0.0.1 requests.packages.urllib3.connectionpool: DEBUG: "GET /api/histories/c33ffb67c9176a6c/contents?key=ae52a59b996b793c4c06a11fc3c33481 HTTP/1.1" 200 None requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 127.0.0.1 requests.packages.urllib3.connectionpool: DEBUG: "GET /api/histories/c33ffb67c9176a6c/contents/d80a923b055ee8e9?key=ae52a59b996b793c4c06a11fc3c33481 HTTP/1.1" 200 None requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 127.0.0.1 requests.packages.urllib3.connectionpool: DEBUG: "GET /api/histories/c33ffb67c9176a6c/contents/d80a923b055ee8e9/provenance?key=ae52a59b996b793c4c06a11fc3c33481 HTTP/1.1" 200 None (All the tests for this tool seem to have failed the same way) Regards, Peter On Wed, Mar 18, 2015 at 1:46 PM, Dave Bouvier <dave(a)bx.psu.edu> wrote: > Gentlemen, > > The issue with the nightly testing was due to a stalled test run blocking > subsequent tests. I've cleared out that blockage and a manual test run > appears to have completed successfully, as should future automated test > runs. As always, feel free to let us know if you encounter any additional > inexplicable behavior. > > --Dave B. > > On 03/18/2015 07:00 AM, Peter Cock wrote: >> >> ... I can also confirm that the Test Tool Shed example I >> gave was tested overnight, although it looks like a novel failure: >> https://testtoolshed.g2.bx.psu.edu/view/peterjc/sample_seqs >>

2 4

wrapper accepts fastqcssanger when it shouldn't
by Nikhil Joshi 02 Apr '15

02 Apr '15

Hi all, So in my wrapper for my trimming tool, sickle, I only allow fastq, fastqsanger, fastqillumina, or fastqsolexa input data. However, if a fastqsanger file's datatype has been erroneously changed to fastqcssanger, my tool will still show that file in the dropdown for the possible inputs. Why does this happen? - Nik. -- Nikhil Joshi Bioinformatics Analyst/Programmer UC Davis Bioinformatics Core http://bioinformatics.ucdavis.edu/ najoshi -at- ucdavis -dot- edu 530.752.2698 (w)

2 1

dat quota
by Williams, Charles Houston 01 Apr '15

01 Apr '15

After reading threads. I have deleted and purged all but 33gb of my datasets and a single history. However, my quota is still 99%. What can I do to fix this issue. My user address is Charles.h.williams@vanderbilt. Edu Thanks, Chuck Charles H Williams III Graduate Student Hong Lab Department of Cell and Developmental Biology Vanderbilt University Light Hall 1155a

2 1