March 2015 - galaxy-dev - lists.galaxyproject.org

Using synchronous method to upload multiple files
by Daniele P Colobraro 16 Apr '15

16 Apr '15

Hi all, I've set up a simple tool using the synchronous method to upload a file on galaxy. I'm able to visit the external link, select and upload a file by the list. The script that I use is data_source.py and the 'command' part in the xml configuration is the same as the ucsc xml config (ucsc_tablebrowser.xml). But my purpose is to import multiple files. Have you any idea how I can set a way to import/upload multiple files by using the synchronous method? Thank in advance, Best regards Daniele

3 5

problem with revision updates for repositories.
by Anna Terry 14 Apr '15

14 Apr '15

Hi, I have a few installed tool shed repositories which are showing that updates are available. When I click get repository updates, instead of the usual upate page, I get a page that simply says Found The resource was found at http://toolshed.g2.bx.psu.edu/repository/check_for_updates?galaxy_url=https…; you should be redirected automatically. This is for bam_to_sam, but same problem for blast_datatypes, pileup_interval and sam_to_bam too. Anyone have any ideas why I'm seeing this? Cheers Anna

2 9

Visualization in anonymous mode ?
by Loraine Guéguen 09 Apr '15

09 Apr '15

Hi, I'd like to use the visualization in anonymous mode, but the visualize icon on the dataset is not available in anonymous mode (although it works fine when connected). Is it possible to activate the dataset visualization in anonymous mode ? Thanks, Loraine -- Loraine Guéguen Plateforme ABiMS (Analyses and Bioinformatics for Marine Sciences) Service Informatique et Bioinformatique Station Biologique de Roscoff FR 2424 CNRS UPMC Place Georges Teissier CS 90074 29688 Roscoff cedex tel: ++33 (0)2 98 29 56 46 (interne : 415) http://abims.sb-roscoff.fr/ http://www.sb-roscoff.fr/

2 5

Tests not being run on toolsheds?
by Peter Briggs 09 Apr '15

09 Apr '15

Hello Apologies as this seems to be quite a frequent question (and I may have missed something) however: are the tool tests being executed on the main or test toolsheds at the moment? The most recent tests results that I can find for any of the tools I've uploaded is the 29th January. I've also looked at a couple of tools from other users and the situation appears to be the same (at least in those cases). Thanks for your help! Best wishes Peter -- Peter Briggs peter.briggs(a)manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482

3 9

DRMAA chmod to world-writable if to user chown fails should not be default?
by David Trudgian 08 Apr '15

08 Apr '15

Hi, Looking through the DRMAA job runner code I noticed that in lib/galaxy/jobs/__init__.py:1582 that if chown to a user in change_ownership_for_run fails then there is chmod 0777 instead. I'm thinking that this probably shouldn't happen by default, or there needs to be a big warning on the wiki? I understand the world-readable security concerns posted there - but world-writable, for whatever reason, on a shared cluster seems like a disaster waiting to happen. Maybe either warn on the wiki, make the chmod 777 a non-default option, or an option to disable it? DT ________________________________ UT Southwestern Medical Center The future of medicine, today.

2 2

Cloning state when installing tools.
by Carlos Lijeron 08 Apr '15

08 Apr '15

Everyone, I've been encountering problems when installing particular tools in a local Galaxy instance. One such tool is Cummerbund as you can see in link below: https://www.dropbox.com/s/bmg4qcci97ayq2k/Cloning%20State.JPG?dl=0 Would you have a suggestion on how to troubleshoot failed tool installations that do not give an output error message? Thank you all ! Carlos.

2 1

Workflow bug when *.loc entries are missing
by Peter Cock 06 Apr '15

06 Apr '15

Hi all, I was testing a new workflow which I developed on the current stable release, Galaxy 15.03, and uploaded to the Tool Shed yesterday: https://github.com/peterjc/galaxy_blast/tree/master/workflows/blast_top_hit… http://toolshed.g2.bx.psu.edu/view/peterjc/blast_top_hit_species I then switched to my development instance running the dev branch from github, 1. Log in as an Admin 2. Install http://toolshed.g2.bx.psu.edu/view/peterjc/blast_top_hit_species 3. Upload suggested test dataset (in workflow README file): http://nematode.net/Data/nacobbus_aberrans_transcript_assembly/N.abberans_r… 4. Click on "All Workflows" (bottom of left pane tool list) 5. Click on "Species of top BLAST hits (imported from uploaded file)" 6. On preview screen, examine step 3, blastx 7. Notice it will run against Protein BLAST database "nr" 8. Click "Run workflow" As noted in the README file, this workflow assumes the local blastdb_p.loc file includes an entry "nr" for a current/recent mirror of the NCBI "non-redundant" protein BLAST database. Initially my development galaxy had an empty blastdb_p.loc file, so I expected the workflow to spot this problem at the preview step (before ever clicking "run workflow"). Instead it proceeds and attempted to run BLASTX without no database (which of course failed): blastx -query "/mnt/galaxy/repositories/galaxy/database/files/000/dataset_81.dat" -db "" -query_gencode 1 -evalue 0.001 -out "/mnt/galaxy/repositories/galaxy/database/files/000/dataset_82.dat" -outfmt "6 qseqid sseqid pident length mismatch gapopen qstart qend sstart send evalue bitscore staxids sscinames scomnames sblastnames sskingdoms" -num_threads "${GALAXY_SLOTS:-8}" -strand both -matrix BLOSUM62 -seg yes -max_target_seqs 1 That's the bad news: It fails, but it fails with a reasonably clear error: Fatal error: Exit code 2 () BLAST Database error: Database name is required. Good news: If I update blastdb_p.loc to add the "nr" database it works. Very bad news: If I update blastdb_p.loc to add other databases (but not an entry for "nr"), the workflow defaults to the first database in blastdb_p.loc Summary: Workflow requesting key entry "xxx" from example.loc (a) example.loc empty, gets empty path, fails cleanly. (b) example.loc contains xxx, gets correct path, works. (c) example.loc non empty but lacks xxx, runs with WRONG path. Peter

4 10

test/base/twilltestcase.py - Converting local (test-data) bam to sam failed
by Peter Cock 02 Apr '15

02 Apr '15

Hi all, I have a query about a failing tool test on the Tool Shed, where it seems Galaxy is trying to convert both the expected BAM output and the tool's BAM output into SAM for comparison - but this fails. https://toolshed.g2.bx.psu.edu/view/peterjc/samtools_depad 2015-01-27 02:13:51 ... Tool id: samtools_depad Tool version: samtools_depad Test: test_tool_000000 (functional.test_toolbox.TestForTool_toolshed.g2.bx.psu.edu/repos/peterjc/sa…) Stderr: Traceback: Traceback (most recent call last): File "/tmp/buildslave/buildbot-install-test-main-tool-shed-py27/build/test/functional/test_toolbox.py", line 122, in test_tool self.do_it( td ) File "/tmp/buildslave/buildbot-install-test-main-tool-shed-py27/build/test/functional/test_toolbox.py", line 40, in do_it self._verify_outputs( testdef, test_history, jobs, shed_tool_id, data_list, galaxy_interactor ) File "/tmp/buildslave/buildbot-install-test-main-tool-shed-py27/build/test/functional/test_toolbox.py", line 80, in _verify_outputs galaxy_interactor.verify_output( history, jobs, output_data, output_testdef=output_testdef, shed_tool_id=shed_tool_id, maxseconds=maxseconds ) File "/tmp/buildslave/buildbot-install-test-main-tool-shed-py27/build/test/base/interactor.py", line 66, in verify_output self.twill_test_case.verify_hid( outfile, hda_id=hid, attributes=attributes, dataset_fetcher=fetcher, shed_tool_id=shed_tool_id ) File "/tmp/buildslave/buildbot-install-test-main-tool-shed-py27/build/test/base/twilltestcase.py", line 2341, in verify_hid raise AssertionError( errmsg ) AssertionError: History item different than expected, difference (using diff): ( /tmp/shed_tools/toolshed.g2.bx.psu.edu/repos/peterjc/samtools_depad/2d303f2e09e0/samtools_depad/test-data/sam_spec_padded.depad.bam v. /tmp/buildslave/buildbot-install-test-main-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/repositories_with_tools/tmp/tmpExQWRD/tmpSknKqLsam_spec_padded.depad.bam ) Converting local (test-data) bam to sam failed (and same error on the second test) The same occurs on the Test Tool Shed (with a very slightly updated version of this tool), although with a more concise traceback: https://testtoolshed.g2.bx.psu.edu/view/peterjc/samtools_depad 2015-01-25 08:21:59 ... Tool id: samtools_depad Tool version: samtools_depad Test: test_tool_000000 (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterj…) Stderr: Traceback: Traceback (most recent call last): File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 268, in test_tool self.do_it( td ) File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 67, in do_it raise e JobOutputsError: History item different than expected, difference (using diff): ( /tmp/shed_tools/testtoolshed.g2.bx.psu.edu/repos/peterjc/samtools_depad/8e1e3a1ecad7/samtools_depad/test-data/sam_spec_padded.depad.bam v. /tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/repositories_with_tools/tmp/tmpNvCGJi/tmphX3A_Fsam_spec_padded.depad.bam ) Converting local (test-data) bam to sam failed Both tests should produce a BAM file matching the provided test-data/sam_spec_padded.depad.bam example. The error comes from test/base/twilltestcase.py, p = subprocess.Popen( args='samtools view -h -o "%s" "%s"' % ( temp_local.name, local_name ), shell=True ) assert not p.wait(), 'Converting local (test-data) bam to sam failed' Running this command locally at the terminal seems to work fine, both with samtools 0.1.19 and 1.1. $ ~/bin/samtools_0.1.19 view -h -o temp.sam sam_spec_padded.depad.bam $ more temp.sam @HD VN:1.5 SO:coordinate @SQ SN:ref LN:45 ref 516 ref 1 0 45M * 0 0 AGCATGTTAGATAAGATAGCTGTGCTAGTAGGCAGTCAGCGCCAT * r001 99 ref 7 30 8M2I4M1D3M = 37 41 TTAGATAAAGGATACTG * * 768 ref 8 30 1M * 0 0 * * CT:Z:.;Warning;Note=Ref wrong? r002 0 ref 9 30 3S6M1P1I4M * 0 0 AAAAGATAAGGATA * PT:Z:1;4;+;homopolymer r003 0 ref 9 30 5H6M * 0 0 AGCTAA * NM:i:1 r004 0 ref 16 30 6M14D5M * 0 0 ATAGCTTCAGC * r003 2064 ref 29 30 6H5M * 0 0 TAGGC * NM:i:0 r001 147 ref 37 30 9M = 7 -41 CAGCGGCAT * NM:i:1 $ ~/bin/samtools_1.1 view -h -o temp.sam sam_spec_padded.depad.bam [galaxy@ppserver test-data]$ more temp.sam @HD VN:1.5 SO:coordinate @SQ SN:ref LN:45 ref 516 ref 1 0 45M * 0 0 AGCATGTTAGATAAGATAGCTGTGCTAGTAGGCAGTCAGCGCCAT * r001 99 ref 7 30 8M2I4M1D3M = 37 41 TTAGATAAAGGATACTG * * 768 ref 8 30 1M * 0 0 * * CT:Z:.;Warning;Note=Ref wrong? r002 0 ref 9 30 3S6M1P1I4M * 0 0 AAAAGATAAGGATA * PT:Z:1;4;+;homopolymer r003 0 ref 9 30 5H6M * 0 0 AGCTAA * NM:i:1 r004 0 ref 16 30 6M14D5M * 0 0 ATAGCTTCAGC * r003 2064 ref 29 30 6H5M * 0 0 TAGGC * NM:i:0 r001 147 ref 37 30 9M = 7 -41 CAGCGGCAT * NM:i:1 Which version of samtools is on the $PATH on the test system? Could the test system have run out of disk space again? The tests are passing on my local development instance (before and after updating to latest galaxy-central), and on TravisCI using the latest Galaxy: https://travis-ci.org/peterjc/pico_galaxy/builds/48485673 Thanks, Peter

2 5

value_from_basic - KeyError: 'base_name', was: Tests not being run on toolsheds?
by Peter Cock 02 Apr '15

02 Apr '15

Hi Dave, The following looks like a regression on the Test Tool Shed, the tests pass locally (using a recent revision), and on TravisCI using the current galaxy dev branch on GitHub: https://travis-ci.org/peterjc/pico_galaxy/builds/54870500 Problem tool: https://testtoolshed.g2.bx.psu.edu/view/peterjc/sample_seqs This may be unrelated, but by chance this is the first time I have uploaded a tool to the Tool Shed which uses the new functionality to test the stdout/stderr strings, and more importantly it includes a test expected to fail via <test expect_failure="true" expect_exit_code="1"> Revision on my development repository: https://github.com/peterjc/pico_galaxy/commit/55ebb308b911b4acef912cc3b03f4… Test Tool Shed output from last night: Automated test environment Time tested: 2015-03-18 02:46:55 System: Linux 3.13.0-36-generic Architecture: x86_64 Python version: 2.7.6 Galaxy revision: 17050:6395e7035143 Galaxy database version: 128 Tool shed revision: 16867:0468d285f89c Tool shed database version: 25 Tool shed mercurial version: 3.2.4 Tests that failed Tool id: sample_seqs Tool version: sample_seqs Test: test_tool_000000 (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterj…) Stderr: Traceback: Traceback (most recent call last): File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 268, in test_tool self.do_it( td ) File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 37, in do_it stage_data_in_history( galaxy_interactor, testdef.test_data(), test_history, shed_tool_id ) File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/base/interactor.py", line 38, in stage_data_in_history upload_wait() File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/base/interactor.py", line 279, in wait while not self.__history_ready( history_id ): File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/base/interactor.py", line 297, in __history_ready return self._state_ready( state, error_msg="History in error state." ) File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/base/interactor.py", line 356, in _state_ready raise Exception( error_msg ) Exception: History in error state. Traceback (most recent call last): File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/jobs/runners/__init__.py", line 158, in prepare_job job_wrapper.prepare() File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/jobs/__init__.py", line 828, in prepare tool_evaluator.set_compute_environment( compute_environment, get_special=get_special ) File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/tools/evaluation.py", line 53, in set_compute_environment incoming = self.tool.params_from_strings( incoming, self.app ) File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/tools/__init__.py", line 1873, in params_from_strings return params_from_strings( self.inputs, params, app, ignore_errors ) File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/tools/parameters/__init__.py", line 103, in params_from_strings value = params[key].value_from_basic( value, app, ignore_errors ) File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/tools/parameters/grouping.py", line 512, in value_from_basic raise e KeyError: 'base_name' requests.packages.urllib3.connectionpool: DEBUG: "GET /api/histories/c33ffb67c9176a6c?key=ae52a59b996b793c4c06a11fc3c33481 HTTP/1.1" 200 None requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 127.0.0.1 galaxy.datatypes.metadata: DEBUG: Cleaning up external metadata files requests.packages.urllib3.connectionpool: DEBUG: "GET /api/histories/c33ffb67c9176a6c?key=ae52a59b996b793c4c06a11fc3c33481 HTTP/1.1" 200 None requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 127.0.0.1 requests.packages.urllib3.connectionpool: DEBUG: "GET /api/histories/c33ffb67c9176a6c/contents?key=ae52a59b996b793c4c06a11fc3c33481 HTTP/1.1" 200 None requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 127.0.0.1 requests.packages.urllib3.connectionpool: DEBUG: "GET /api/histories/c33ffb67c9176a6c/contents/d80a923b055ee8e9?key=ae52a59b996b793c4c06a11fc3c33481 HTTP/1.1" 200 None requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 127.0.0.1 requests.packages.urllib3.connectionpool: DEBUG: "GET /api/histories/c33ffb67c9176a6c/contents/d80a923b055ee8e9/provenance?key=ae52a59b996b793c4c06a11fc3c33481 HTTP/1.1" 200 None (All the tests for this tool seem to have failed the same way) Regards, Peter On Wed, Mar 18, 2015 at 1:46 PM, Dave Bouvier <dave(a)bx.psu.edu> wrote: > Gentlemen, > > The issue with the nightly testing was due to a stalled test run blocking > subsequent tests. I've cleared out that blockage and a manual test run > appears to have completed successfully, as should future automated test > runs. As always, feel free to let us know if you encounter any additional > inexplicable behavior. > > --Dave B. > > On 03/18/2015 07:00 AM, Peter Cock wrote: >> >> ... I can also confirm that the Test Tool Shed example I >> gave was tested overnight, although it looks like a novel failure: >> https://testtoolshed.g2.bx.psu.edu/view/peterjc/sample_seqs >>

2 4

dat quota
by Williams, Charles Houston 01 Apr '15

01 Apr '15

After reading threads. I have deleted and purged all but 33gb of my datasets and a single history. However, my quota is still 99%. What can I do to fix this issue. My user address is Charles.h.williams@vanderbilt. Edu Thanks, Chuck Charles H Williams III Graduate Student Hong Lab Department of Cell and Developmental Biology Vanderbilt University Light Hall 1155a

2 1