- galaxy-dev - lists.galaxyproject.org

Installing workflows from the Tool Shed is very confusing
by Peter Cock 07 Apr '15

07 Apr '15

Hello all, I've been playing with the Tool Shed lately for publishing Galaxy workflows - and realised the download process is still very complicated, especially the final hurdle - importing the workflow in order to actually use it. For the following example I am starting from a clean state - an almost fresh Galaxy installation with NOTHING as yet installed from the Tool Shed (see end of email), and I want to install a single workflow: https://toolshed.g2.bx.psu.edu/view/peterjc/blast_top_hit_species 1. Goto Galaxy in browser, eg http://localhost/galaxy-dev/ 2. Goto "User", "Register" 3. Create account using address added to admin_users earlier in config/galaxy.ini 4. Click "Return to the home page" 5. Since logged in as admin, can goto "Admin" settings 6. Select "Search and browse tool sheds" 7. Select "Galaxy Main Tool Shed" 8. Enter "species" in the search box 9. Select "blast_top_hit_species" 10. Note the list of dependencies 11. Click "Install into Galaxy" 12. Leave the dependency handling ticked. 13. Ponder that the section selection will apply to? The current repository has no tools but some of the dependencies do... I don't want to bundle the BLAST tools in the same section as the text manipulation tools. Go with the default (don't pick a section). 14. Click "install" 15. Wait as the following are installed: Name Description Owner Revision blast_top_hit_species Workflow to count species of top nr BLASTX hits of a transcriptome peterjc db0c1bb92308 ncbi_blast_plus NCBI BLAST+ devteam 2fe07f50a41e blast_datatypes BlastXml and other blast datatypes devteam 5482a8cd0f36 package_blast_plus_2_2_29 NCBI BLAST+ 2.2.29 (binaries only) iuc a2ec897aac2c package_biopython_1_65 Downloads and compiles version 1.65 of the Biopython package. biopython dc595937617c package_numpy_1_9 Contains a tool dependency definition that downloads and compiles version 1.9 of the the python numpy package. iuc 84e97f5cd3ab package_atlas_3_10 Contains a tool dependency definition that downloads and compiles version 3.10 of the the Automatically Tuned Linear Algebra Software. iuc 37ff27ea37ed fasta_to_tabular FASTA-to-Tabular converter devteam 9d189d08f2ad sample_seqs Sub-sample sequences files (e.g. to reduce coverage) peterjc 02c13ef1a669 unique Advanced unique lines program using GNU sort bgruening 7ce75adb93be 16. Get bored, especially if the page appears to have stalled. Try reloading the page which goes back to the "Admin" welcome. Now how do I get back to the installation status... 17. Click on "Manage installed tool shed repositories" 18. If at this point some of the dependency requirements are in error state, try a round of repairing repositories (and watch at the terminal, e.g. compiling numpy is slow). 19. Admire all the nice green "tick" icons and "Installed" entries. 20. Notice the tiny grey "cog" icon against the workflow repository, "blast_top_hit_species". 21. Read the legend, "This repository contains exported workflows", which is encouraging but there is nothing on the page hinting at *how* to actually use the new workflow. 22. Try the context menu to see if that lets you use the workflow. No. 23. At this point I fully explored the workflow menus looking for an import option, and gave up and returned to "Manage installed tool shed repositories". 23. Ignore the context menu, click on "blast_top_hit_species". 24. Click on the top right "repository actions" menu 25. Find nothing about importing a workflow. 26. Click on the "Species of top BLAST hits" entry under "Contents of this repository" 27. Observe pretty SVG image of workflow. 24. Click on the top right "repository actions" menu 25. Notice the menu now offers "import workflow to galaxy". 26. Click on "import workflow to galaxy". 26. Back at SVG with green message "Workflow Species of top BLAST hits imported successfully. " 27. Return to "Analyze Data" (or "Workflows") and finally the workflow is available (although probably only for the Galaxy administrator's personal account). 28. Cry because all the new tools are dumped in the top level tool menu (see step 13). Possible suggestions to make this easier: At step 21, include a clue in the legend? At step 22 and 25, include "Import all these workflows" in menu? At step 24, change the button to "repository/workflow actions" since it includes some actions specific to the viewed workflow? How about the common use case where the Galaxy Admin was asked to import the workflow for a specific user? Can we have an action "Import workflow for specific user"? Also I think the tool section behaviour for dependencies needs clarification (step 13/28). From a tool author perspective, I would like to be able to set the default section for each tool (see also past requests to set the order they are added to the tool panel). Regards, Peter -- This was how I setup my current galaxy development instance tracking the default dev branch on github: $ cd ~/repositories $ git clone https://github.com/galaxyproject/galaxy.git $ cd galaxy I made minor config changes, including adding myself as an admin user via the admin_users setting: $ emacs config/galaxy.ini ... I also setup Apache to serve this instance at http://localhost/galaxy-dev/ and set ToolShed path to be ~/repositories/shed_tools/ This was working fine. To hopefully reset this back to a what is almost a clean slate (for this bug report), I next removed the auto-generated XML files, data files and SQLite database, and tool shed files: $ rm -f ~/repositories/galaxy/*.xml $ rm -f ~/repositories/galaxy/config/*.xml $ rm -f ~/repositories/galaxy/database/universe.sqlite $ rm -rf ~/repositories/galaxy/database/files/* $ rm -rf ~/repositories/shed_tools/ (Dangerous - which is why I explicitly used the full paths!) I then restart Galaxy, $ cd ~/repositories/galaxy $ ./run.sh ... Note it creates a new empty SQLite database etc.

3 3

Galaxy ToolShed Update Problem
by 孙兆朕 07 Apr '15

07 Apr '15

Hi all, I want to update some of my tool plug-in and some problem shows, even if I updated my Galaxy instance to the latest. I suppose it is not related to the version of Galaxy. The problem occured when I used the administrator's panel and update my plug-in using the "Get updates" option. The next page reports that an internal Server Error has occured and the update was not successful. All the plug-in updates seems had the same problem. I checked the logs where I get the following message: CGI Variables ------------- CONTENT_LENGTH: '0' HTTP_ACCEPT: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' HTTP_ACCEPT_ENCODING: 'gzip,deflate,sdch' HTTP_ACCEPT_LANGUAGE: 'zh-CN,zh;q=0.8' HTTP_CONNECTION: 'close' HTTP_COOKIE: 'galaxysession=c6ca0ddb55be603a23b93349895e875da395c1164de6cfc693d5dd79dc3d9d2edcd8d527c7310424' HTTP_HOST: '222.28.163.110' HTTP_REFERER: 'http://222.28.163.110/admin_toolshed/browse_repositories?status=done&messag….' HTTP_USER_AGENT: 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36' HTTP_X_FORWARDED_FOR: '222.28.170.89' HTTP_X_FORWARDED_HOST: '222.28.163.110' ORGINAL_HTTP_HOST: '127.0.0.1:8080' ORGINAL_REMOTE_ADDR: '127.0.0.1' PATH_INFO: '/admin_toolshed/update_to_changeset_revision' QUERY_STRING: 'tool_shed_url=https://toolshed.g2.bx.psu.edu/&name=blast_datatypes&owner=de…' REMOTE_ADDR: '222.28.170.89' REQUEST_METHOD: 'GET' SERVER_NAME: '127.0.0.1' SERVER_PORT: '8080' SERVER_PROTOCOL: 'HTTP/1.0' WSGI Variables -------------- application: <paste.recursive.RecursiveMiddleware object at 0xd888510> is_api_request: False paste.cookies: (<SimpleCookie: galaxysession='c6ca0ddb55be603a23b93349895e875da395c1164de6cfc693d5dd79dc3d9d2edcd8d527c7310424'>, 'galaxysession=c6ca0ddb55be603a23b93349895e875da395c1164de6cfc693d5dd79dc3d9d2edcd8d527c7310424') paste.expected_exceptions: [<class 'paste.httpexceptions.HTTPException'>] paste.httpexceptions: <paste.httpexceptions.HTTPExceptionHandler object at 0xd888fd0> paste.httpserver.thread_pool: <paste.httpserver.ThreadPool object at 0xdcf8650> paste.parsed_querystring: ([('tool_shed_url', 'https://toolshed.g2.bx.psu.edu/'), ('name', 'blast_datatypes'), ('owner', 'devteam'), ('changeset_revision', 'de11e1a921c4'), ('latest_changeset_revision', '5482a8cd0f36'), ('latest_ctx_rev', '10')], 'tool_shed_url=https://toolshed.g2.bx.psu.edu/&name=blast_datatypes&owner=de…') paste.recursive.forward: <paste.recursive.Forwarder from /> paste.recursive.include: <paste.recursive.Includer from /> paste.recursive.include_app_iter: <paste.recursive.IncluderAppIter from /> paste.recursive.script_name: '' paste.throw_errors: True request_id: 'a12a5604dce911e4a05000215e543aa8' webob._parsed_query_vars: (MultiDict([('tool_shed_url', 'https://toolshed.g2.bx.psu.edu/'), ('name', 'blast_datatypes'), ('owner', 'devteam'), ('changeset_revision', 'de11e1a921c4'), ('latest_changeset_revision', '5482a8cd0f36'), ('latest_ctx_rev', '10')]), 'tool_shed_url=https://toolshed.g2.bx.psu.edu/&name=blast_datatypes&owner=de…') wsgi process: 'Multithreaded' ------------------------------------------------------------ ---------------------------------------- Exception happened during processing of request from ('127.0.0.1', 39438) Traceback (most recent call last): File "/share/apps/galaxy/Galaxy_Test2015/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 1068, in process_request_in_thread self.finish_request(request, client_address) File "/usr/lib64/python2.6/SocketServer.py", line 322, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib64/python2.6/SocketServer.py", line 618, in __init__ self.finish() File "/usr/lib64/python2.6/SocketServer.py", line 661, in finish self.wfile.flush() File "/usr/lib64/python2.6/socket.py", line 303, in flush self._sock.sendall(buffer(data, write_offset, buffer_size)) error: [Errno 32] Broken pipe ---------------------------------------- 222.28.170.89 - - [07/Apr/2015:13:47:59 +0800] "GET /admin_toolshed/update_to_changeset_revision?tool_shed_url=https://toolshed.g2.bx.psu.edu/&name=blast_datatypes&owner=devteam&changeset_revision=de11e1a921c4&latest_changeset_revision=5482a8cd0f36&latest_ctx_rev=10 HTTP/1.0" 500 - "http://222.28.163.110/admin_toolshed/browse_repositories?status=done&messag…." "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36" Error - <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'get' URL: http://222.28.163.110/admin_toolshed/update_to_changeset_revision?tool_shed… File '/share/apps/galaxy/Galaxy_Test2015/galaxy-dist/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/share/apps/galaxy/Galaxy_Test2015/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/share/apps/galaxy/Galaxy_Test2015/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/share/apps/galaxy/Galaxy_Test2015/galaxy-dist/lib/galaxy/web/framework/base.py', line 133 in __call__ return self.handle_request( environ, start_response ) File '/share/apps/galaxy/Galaxy_Test2015/galaxy-dist/lib/galaxy/web/framework/base.py', line 191 in handle_request body = method( trans, **kwargs ) File '/share/apps/galaxy/Galaxy_Test2015/galaxy-dist/lib/galaxy/web/framework/decorators.py', line 87 in decorator return func( self, trans, *args, **kwargs ) File '/share/apps/galaxy/Galaxy_Test2015/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py', line 1954 in update_to_changeset_revision persist=True ) File '/share/apps/galaxy/Galaxy_Test2015/galaxy-dist/lib/tool_shed/galaxy_install/metadata/installed_repository_metadata_manager.py', line 31 in __init__ metadata_dict=metadata_dict, user=None ) File '/share/apps/galaxy/Galaxy_Test2015/galaxy-dist/lib/tool_shed/metadata/metadata_generator.py', line 60 in __init__ self.metadata_dict = { 'shed_config_filename' : self.shed_config_dict.get( 'config_filename', None ) } AttributeError: 'NoneType' object has no attribute 'get' CGI Variables ------------- CONTENT_LENGTH: '0' HTTP_ACCEPT: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' HTTP_ACCEPT_ENCODING: 'gzip,deflate,sdch' HTTP_ACCEPT_LANGUAGE: 'zh-CN,zh;q=0.8' HTTP_CONNECTION: 'close' HTTP_COOKIE: 'galaxysession=c6ca0ddb55be603a23b93349895e875da395c1164de6cfc693d5dd79dc3d9d2edcd8d527c7310424' HTTP_HOST: '222.28.163.110' HTTP_REFERER: 'http://222.28.163.110/admin_toolshed/browse_repositories?status=done&messag….' HTTP_USER_AGENT: 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36' HTTP_X_FORWARDED_FOR: '222.28.170.89' HTTP_X_FORWARDED_HOST: '222.28.163.110' ORGINAL_HTTP_HOST: '127.0.0.1:8080' ORGINAL_REMOTE_ADDR: '127.0.0.1' PATH_INFO: '/admin_toolshed/update_to_changeset_revision' QUERY_STRING: 'tool_shed_url=https://toolshed.g2.bx.psu.edu/&name=blast_datatypes&owner=de…' REMOTE_ADDR: '222.28.170.89' REQUEST_METHOD: 'GET' SERVER_NAME: '127.0.0.1' SERVER_PORT: '8080' SERVER_PROTOCOL: 'HTTP/1.0' WSGI Variables -------------- application: <paste.recursive.RecursiveMiddleware object at 0xd888510> is_api_request: False paste.cookies: (<SimpleCookie: galaxysession='c6ca0ddb55be603a23b93349895e875da395c1164de6cfc693d5dd79dc3d9d2edcd8d527c7310424'>, 'galaxysession=c6ca0ddb55be603a23b93349895e875da395c1164de6cfc693d5dd79dc3d9d2edcd8d527c7310424') paste.expected_exceptions: [<class 'paste.httpexceptions.HTTPException'>] paste.httpexceptions: <paste.httpexceptions.HTTPExceptionHandler object at 0xd888fd0> paste.httpserver.thread_pool: <paste.httpserver.ThreadPool object at 0xdcf8650> paste.parsed_querystring: ([('tool_shed_url', 'https://toolshed.g2.bx.psu.edu/'), ('name', 'blast_datatypes'), ('owner', 'devteam'), ('changeset_revision', 'de11e1a921c4'), ('latest_changeset_revision', '5482a8cd0f36'), ('latest_ctx_rev', '10')], 'tool_shed_url=https://toolshed.g2.bx.psu.edu/&name=blast_datatypes&owner=de…') paste.recursive.forward: <paste.recursive.Forwarder from /> paste.recursive.include: <paste.recursive.Includer from /> paste.recursive.include_app_iter: <paste.recursive.IncluderAppIter from /> paste.recursive.script_name: '' paste.throw_errors: True request_id: 'a58bb404dce911e481c700215e543aa8' webob._parsed_query_vars: (MultiDict([('tool_shed_url', 'https://toolshed.g2.bx.psu.edu/'), ('name', 'blast_datatypes'), ('owner', 'devteam'), ('changeset_revision', 'de11e1a921c4'), ('latest_changeset_revision', '5482a8cd0f36'), ('latest_ctx_rev', '10')]), 'tool_shed_url=https://toolshed.g2.bx.psu.edu/&name=blast_datatypes&owner=de…') wsgi process: 'Multithreaded' ------------------------------------------------------------ Could you please give me some advise to figure out where the problem is and how to fix it? Thanks, Zhaozhen

2 1

a question on data manager and data table
by Beginner TI 07 Apr '15

07 Apr '15

Hi All, I just made a new local instance, installed the fetch_all_fasta data manager, downloaded mm9 fasta. Then I noticed the following: in tool_data_table_conf.xml, it has: <table name="all_fasta" comment_char="#"> <columns>value, dbkey, name, path</columns> <file path="tool-data/all_fasta.loc" /> </table> however in shed_tool_data_table_conf.xml, it has: <table comment_char="#" name="all_fasta"> <columns>value, dbkey, name, path</columns> <file path="/auto/rcf-proj/yc1/galaxy-suite/galaxy-dist/tool-data/ toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_all_fasta/cc…" /> <tool_shed_repository><tool_shed>toolshed.g2.bx.psu.edu </tool_shed><repository_name>data_manager_fetch_genome_all_fasta</repository_name><repository_owner>devteam</repository_owner><installed_changeset_revision>cca219f2b212</installed_changeset_revision></tool_shed_repository></table> in tool-data, all_fasta.loc is empty, but in the all_fasta.loc of the shed_tool entry, it shows: mm9 mm9 Mouse July 2007 (NCBI37/mm9) (mm9) /auto/rcf-proj/yc1/galaxy-suite/galaxy-dist/tool-data/mm9/seq/mm9.fa So if I try "Extract Genomic DNA function", I could see the parameter passed to the command line is -g "/auto/rcf-proj/yc1/galaxy-suite/galaxy-dist/tool-data" which does not host the data. It should be at least -g "/auto/rcf-proj/yc1/galaxy-suite/galaxy-dist/tool-data/mm9/seq" I thought that data manager will automatically populate these loc files? Am I missing something obvious? I could manually modify tool_data_table_conf.xml to let it point to /auto/rcf-proj/yc1/galaxy-suite/galaxy-dist/tool-data/ toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_all_fasta/cc… This would work, but it is ugly. Could someone please give me a hand to fix this? Thanks, Rui

2 1

fml_gff3togtf: GTF-to-BED failes on numpy in Galaxy
by Curtis Hendrickson (Campus) 06 Apr '15

06 Apr '15

Folks I installed this very useful looking tool repository (fml_gff3togtf<https://toolshed.g2.bx.psu.edu/view/vipints/fml_gff3togtf>) to convert GFF3->BED12 based on this BioStars. https://www.biostars.org/p/85869 It installed fine, but when I run it on ftp://ftp.ncbi.nih.gov/genomes/Viruses/Human_herpesvirus_5_uid14559/NC_0062… I get Traceback (most recent call last): File "/share/apps/galaxy/test-shed-tools/toolshed.g2.bx.psu.edu/repos/vipints/fml_gff3togtf/6e589f267c14/fml_gff3togtf/gff_to_bed.py", line 18, in <module> import GFFParser File "/share/apps/galaxy/test-shed-tools/toolshed.g2.bx.psu.edu/repos/vipints/fml_gff3togtf/6e589f267c14/fml_gff3togtf/GFFParser.py", line 20, in <module> import scipy.io as sio ImportError: No module named scipy.io Anyone know the trick to get the python libraries set up correctly? (also note that this failed nightly tests on 2015-04-06 - but that looks like a lack of test data/configuration) I see some other packages (like htseq_count<https://toolshed.g2.bx.psu.edu/view/lparsons/htseq_count>) have numpy as a dependency.... Using the devteam's numpy 1.7 package: https://toolshed.g2.bx.psu.edu/view/devteam/package_numpy_1_7 Regards, Curtis Research Associate, Informatics Unit Center for Clinical and Translational Science http://bioinformatics.uab.edu University of Alabama at Birmingham

3 2

Workflow bug when *.loc entries are missing
by Peter Cock 06 Apr '15

06 Apr '15

Hi all, I was testing a new workflow which I developed on the current stable release, Galaxy 15.03, and uploaded to the Tool Shed yesterday: https://github.com/peterjc/galaxy_blast/tree/master/workflows/blast_top_hit… http://toolshed.g2.bx.psu.edu/view/peterjc/blast_top_hit_species I then switched to my development instance running the dev branch from github, 1. Log in as an Admin 2. Install http://toolshed.g2.bx.psu.edu/view/peterjc/blast_top_hit_species 3. Upload suggested test dataset (in workflow README file): http://nematode.net/Data/nacobbus_aberrans_transcript_assembly/N.abberans_r… 4. Click on "All Workflows" (bottom of left pane tool list) 5. Click on "Species of top BLAST hits (imported from uploaded file)" 6. On preview screen, examine step 3, blastx 7. Notice it will run against Protein BLAST database "nr" 8. Click "Run workflow" As noted in the README file, this workflow assumes the local blastdb_p.loc file includes an entry "nr" for a current/recent mirror of the NCBI "non-redundant" protein BLAST database. Initially my development galaxy had an empty blastdb_p.loc file, so I expected the workflow to spot this problem at the preview step (before ever clicking "run workflow"). Instead it proceeds and attempted to run BLASTX without no database (which of course failed): blastx -query "/mnt/galaxy/repositories/galaxy/database/files/000/dataset_81.dat" -db "" -query_gencode 1 -evalue 0.001 -out "/mnt/galaxy/repositories/galaxy/database/files/000/dataset_82.dat" -outfmt "6 qseqid sseqid pident length mismatch gapopen qstart qend sstart send evalue bitscore staxids sscinames scomnames sblastnames sskingdoms" -num_threads "${GALAXY_SLOTS:-8}" -strand both -matrix BLOSUM62 -seg yes -max_target_seqs 1 That's the bad news: It fails, but it fails with a reasonably clear error: Fatal error: Exit code 2 () BLAST Database error: Database name is required. Good news: If I update blastdb_p.loc to add the "nr" database it works. Very bad news: If I update blastdb_p.loc to add other databases (but not an entry for "nr"), the workflow defaults to the first database in blastdb_p.loc Summary: Workflow requesting key entry "xxx" from example.loc (a) example.loc empty, gets empty path, fails cleanly. (b) example.loc contains xxx, gets correct path, works. (c) example.loc non empty but lacks xxx, runs with WRONG path. Peter

4 10

changing usage indicator
by Nikhil Joshi 04 Apr '15

04 Apr '15

Hi all, Is there a way to change the "Using" indicator at the top right to be a Using/Total for the whole disk (not just Galaxy usage)? - Nik. -- Nikhil Joshi Bioinformatics Analyst/Programmer UC Davis Bioinformatics Core http://bioinformatics.ucdavis.edu/ najoshi -at- ucdavis -dot- edu 530.752.2698 (w)

2 3

NGINX uWSGI with require_login needs uwsgi_param SCRIPT_NAME '';
by David Trudgian 02 Apr '15

02 Apr '15

Hi, When I enabled 'require_login = True' on a galaxy instance running with NGINX+uWSGI I receive internal server error pages due to a Key error for File 'lib/galaxy/web/framework/base.py', line 356 in path return self.environ['SCRIPT_NAME'] + self.environ['PATH_INFO'] KeyError: 'SCRIPT_NAME' Works fine with 'require_login = False' - I'm not sure why require_login makes it take a code path which needs SCRIPT_NAME. Anyway - the addition of... uwsgi_param SCRIPT_NAME ''; ... to the nginx.conf prevents this error. Should I try and add this to the uWSGI config snippets on the wiki? Thanks, -- David Trudgian Ph.D., Computational Scientist UT Southwestern BioHPC -- David Trudgian Ph.D., Computational Scientist UT Southwestern BioHPC ________________________________ UT Southwestern Medical Center The future of medicine, today.

2 2

April! Galactic News!
by Dave Clements 02 Apr '15

02 Apr '15

1 0

using DataManager to add custom genomes
by Nikhil Joshi 02 Apr '15

02 Apr '15

Hi all, Is it possible to use a DataManager (or anything) to allow a user (or admin) to upload their genome of choice and have it be indexed and be used as a built-in/cached genome *without* having to go on the command-line? I.e., only using the Galaxy interface? - Nik. -- Nikhil Joshi Bioinformatics Analyst/Programmer UC Davis Bioinformatics Core http://bioinformatics.ucdavis.edu/ najoshi -at- ucdavis -dot- edu 530.752.2698 (w)

2 2

test/base/twilltestcase.py - Converting local (test-data) bam to sam failed
by Peter Cock 02 Apr '15

02 Apr '15

Hi all, I have a query about a failing tool test on the Tool Shed, where it seems Galaxy is trying to convert both the expected BAM output and the tool's BAM output into SAM for comparison - but this fails. https://toolshed.g2.bx.psu.edu/view/peterjc/samtools_depad 2015-01-27 02:13:51 ... Tool id: samtools_depad Tool version: samtools_depad Test: test_tool_000000 (functional.test_toolbox.TestForTool_toolshed.g2.bx.psu.edu/repos/peterjc/sa…) Stderr: Traceback: Traceback (most recent call last): File "/tmp/buildslave/buildbot-install-test-main-tool-shed-py27/build/test/functional/test_toolbox.py", line 122, in test_tool self.do_it( td ) File "/tmp/buildslave/buildbot-install-test-main-tool-shed-py27/build/test/functional/test_toolbox.py", line 40, in do_it self._verify_outputs( testdef, test_history, jobs, shed_tool_id, data_list, galaxy_interactor ) File "/tmp/buildslave/buildbot-install-test-main-tool-shed-py27/build/test/functional/test_toolbox.py", line 80, in _verify_outputs galaxy_interactor.verify_output( history, jobs, output_data, output_testdef=output_testdef, shed_tool_id=shed_tool_id, maxseconds=maxseconds ) File "/tmp/buildslave/buildbot-install-test-main-tool-shed-py27/build/test/base/interactor.py", line 66, in verify_output self.twill_test_case.verify_hid( outfile, hda_id=hid, attributes=attributes, dataset_fetcher=fetcher, shed_tool_id=shed_tool_id ) File "/tmp/buildslave/buildbot-install-test-main-tool-shed-py27/build/test/base/twilltestcase.py", line 2341, in verify_hid raise AssertionError( errmsg ) AssertionError: History item different than expected, difference (using diff): ( /tmp/shed_tools/toolshed.g2.bx.psu.edu/repos/peterjc/samtools_depad/2d303f2e09e0/samtools_depad/test-data/sam_spec_padded.depad.bam v. /tmp/buildslave/buildbot-install-test-main-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/repositories_with_tools/tmp/tmpExQWRD/tmpSknKqLsam_spec_padded.depad.bam ) Converting local (test-data) bam to sam failed (and same error on the second test) The same occurs on the Test Tool Shed (with a very slightly updated version of this tool), although with a more concise traceback: https://testtoolshed.g2.bx.psu.edu/view/peterjc/samtools_depad 2015-01-25 08:21:59 ... Tool id: samtools_depad Tool version: samtools_depad Test: test_tool_000000 (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterj…) Stderr: Traceback: Traceback (most recent call last): File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 268, in test_tool self.do_it( td ) File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 67, in do_it raise e JobOutputsError: History item different than expected, difference (using diff): ( /tmp/shed_tools/testtoolshed.g2.bx.psu.edu/repos/peterjc/samtools_depad/8e1e3a1ecad7/samtools_depad/test-data/sam_spec_padded.depad.bam v. /tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/repositories_with_tools/tmp/tmpNvCGJi/tmphX3A_Fsam_spec_padded.depad.bam ) Converting local (test-data) bam to sam failed Both tests should produce a BAM file matching the provided test-data/sam_spec_padded.depad.bam example. The error comes from test/base/twilltestcase.py, p = subprocess.Popen( args='samtools view -h -o "%s" "%s"' % ( temp_local.name, local_name ), shell=True ) assert not p.wait(), 'Converting local (test-data) bam to sam failed' Running this command locally at the terminal seems to work fine, both with samtools 0.1.19 and 1.1. $ ~/bin/samtools_0.1.19 view -h -o temp.sam sam_spec_padded.depad.bam $ more temp.sam @HD VN:1.5 SO:coordinate @SQ SN:ref LN:45 ref 516 ref 1 0 45M * 0 0 AGCATGTTAGATAAGATAGCTGTGCTAGTAGGCAGTCAGCGCCAT * r001 99 ref 7 30 8M2I4M1D3M = 37 41 TTAGATAAAGGATACTG * * 768 ref 8 30 1M * 0 0 * * CT:Z:.;Warning;Note=Ref wrong? r002 0 ref 9 30 3S6M1P1I4M * 0 0 AAAAGATAAGGATA * PT:Z:1;4;+;homopolymer r003 0 ref 9 30 5H6M * 0 0 AGCTAA * NM:i:1 r004 0 ref 16 30 6M14D5M * 0 0 ATAGCTTCAGC * r003 2064 ref 29 30 6H5M * 0 0 TAGGC * NM:i:0 r001 147 ref 37 30 9M = 7 -41 CAGCGGCAT * NM:i:1 $ ~/bin/samtools_1.1 view -h -o temp.sam sam_spec_padded.depad.bam [galaxy@ppserver test-data]$ more temp.sam @HD VN:1.5 SO:coordinate @SQ SN:ref LN:45 ref 516 ref 1 0 45M * 0 0 AGCATGTTAGATAAGATAGCTGTGCTAGTAGGCAGTCAGCGCCAT * r001 99 ref 7 30 8M2I4M1D3M = 37 41 TTAGATAAAGGATACTG * * 768 ref 8 30 1M * 0 0 * * CT:Z:.;Warning;Note=Ref wrong? r002 0 ref 9 30 3S6M1P1I4M * 0 0 AAAAGATAAGGATA * PT:Z:1;4;+;homopolymer r003 0 ref 9 30 5H6M * 0 0 AGCTAA * NM:i:1 r004 0 ref 16 30 6M14D5M * 0 0 ATAGCTTCAGC * r003 2064 ref 29 30 6H5M * 0 0 TAGGC * NM:i:0 r001 147 ref 37 30 9M = 7 -41 CAGCGGCAT * NM:i:1 Which version of samtools is on the $PATH on the test system? Could the test system have run out of disk space again? The tests are passing on my local development instance (before and after updating to latest galaxy-central), and on TravisCI using the latest Galaxy: https://travis-ci.org/peterjc/pico_galaxy/builds/48485673 Thanks, Peter

2 5