February 2015 - galaxy-dev - lists.galaxyproject.org

test/base/twilltestcase.py - Converting local (test-data) bam to sam failed
by Peter Cock 02 Apr '15

02 Apr '15

Hi all, I have a query about a failing tool test on the Tool Shed, where it seems Galaxy is trying to convert both the expected BAM output and the tool's BAM output into SAM for comparison - but this fails. https://toolshed.g2.bx.psu.edu/view/peterjc/samtools_depad 2015-01-27 02:13:51 ... Tool id: samtools_depad Tool version: samtools_depad Test: test_tool_000000 (functional.test_toolbox.TestForTool_toolshed.g2.bx.psu.edu/repos/peterjc/sa…) Stderr: Traceback: Traceback (most recent call last): File "/tmp/buildslave/buildbot-install-test-main-tool-shed-py27/build/test/functional/test_toolbox.py", line 122, in test_tool self.do_it( td ) File "/tmp/buildslave/buildbot-install-test-main-tool-shed-py27/build/test/functional/test_toolbox.py", line 40, in do_it self._verify_outputs( testdef, test_history, jobs, shed_tool_id, data_list, galaxy_interactor ) File "/tmp/buildslave/buildbot-install-test-main-tool-shed-py27/build/test/functional/test_toolbox.py", line 80, in _verify_outputs galaxy_interactor.verify_output( history, jobs, output_data, output_testdef=output_testdef, shed_tool_id=shed_tool_id, maxseconds=maxseconds ) File "/tmp/buildslave/buildbot-install-test-main-tool-shed-py27/build/test/base/interactor.py", line 66, in verify_output self.twill_test_case.verify_hid( outfile, hda_id=hid, attributes=attributes, dataset_fetcher=fetcher, shed_tool_id=shed_tool_id ) File "/tmp/buildslave/buildbot-install-test-main-tool-shed-py27/build/test/base/twilltestcase.py", line 2341, in verify_hid raise AssertionError( errmsg ) AssertionError: History item different than expected, difference (using diff): ( /tmp/shed_tools/toolshed.g2.bx.psu.edu/repos/peterjc/samtools_depad/2d303f2e09e0/samtools_depad/test-data/sam_spec_padded.depad.bam v. /tmp/buildslave/buildbot-install-test-main-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/repositories_with_tools/tmp/tmpExQWRD/tmpSknKqLsam_spec_padded.depad.bam ) Converting local (test-data) bam to sam failed (and same error on the second test) The same occurs on the Test Tool Shed (with a very slightly updated version of this tool), although with a more concise traceback: https://testtoolshed.g2.bx.psu.edu/view/peterjc/samtools_depad 2015-01-25 08:21:59 ... Tool id: samtools_depad Tool version: samtools_depad Test: test_tool_000000 (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterj…) Stderr: Traceback: Traceback (most recent call last): File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 268, in test_tool self.do_it( td ) File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 67, in do_it raise e JobOutputsError: History item different than expected, difference (using diff): ( /tmp/shed_tools/testtoolshed.g2.bx.psu.edu/repos/peterjc/samtools_depad/8e1e3a1ecad7/samtools_depad/test-data/sam_spec_padded.depad.bam v. /tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/repositories_with_tools/tmp/tmpNvCGJi/tmphX3A_Fsam_spec_padded.depad.bam ) Converting local (test-data) bam to sam failed Both tests should produce a BAM file matching the provided test-data/sam_spec_padded.depad.bam example. The error comes from test/base/twilltestcase.py, p = subprocess.Popen( args='samtools view -h -o "%s" "%s"' % ( temp_local.name, local_name ), shell=True ) assert not p.wait(), 'Converting local (test-data) bam to sam failed' Running this command locally at the terminal seems to work fine, both with samtools 0.1.19 and 1.1. $ ~/bin/samtools_0.1.19 view -h -o temp.sam sam_spec_padded.depad.bam $ more temp.sam @HD VN:1.5 SO:coordinate @SQ SN:ref LN:45 ref 516 ref 1 0 45M * 0 0 AGCATGTTAGATAAGATAGCTGTGCTAGTAGGCAGTCAGCGCCAT * r001 99 ref 7 30 8M2I4M1D3M = 37 41 TTAGATAAAGGATACTG * * 768 ref 8 30 1M * 0 0 * * CT:Z:.;Warning;Note=Ref wrong? r002 0 ref 9 30 3S6M1P1I4M * 0 0 AAAAGATAAGGATA * PT:Z:1;4;+;homopolymer r003 0 ref 9 30 5H6M * 0 0 AGCTAA * NM:i:1 r004 0 ref 16 30 6M14D5M * 0 0 ATAGCTTCAGC * r003 2064 ref 29 30 6H5M * 0 0 TAGGC * NM:i:0 r001 147 ref 37 30 9M = 7 -41 CAGCGGCAT * NM:i:1 $ ~/bin/samtools_1.1 view -h -o temp.sam sam_spec_padded.depad.bam [galaxy@ppserver test-data]$ more temp.sam @HD VN:1.5 SO:coordinate @SQ SN:ref LN:45 ref 516 ref 1 0 45M * 0 0 AGCATGTTAGATAAGATAGCTGTGCTAGTAGGCAGTCAGCGCCAT * r001 99 ref 7 30 8M2I4M1D3M = 37 41 TTAGATAAAGGATACTG * * 768 ref 8 30 1M * 0 0 * * CT:Z:.;Warning;Note=Ref wrong? r002 0 ref 9 30 3S6M1P1I4M * 0 0 AAAAGATAAGGATA * PT:Z:1;4;+;homopolymer r003 0 ref 9 30 5H6M * 0 0 AGCTAA * NM:i:1 r004 0 ref 16 30 6M14D5M * 0 0 ATAGCTTCAGC * r003 2064 ref 29 30 6H5M * 0 0 TAGGC * NM:i:0 r001 147 ref 37 30 9M = 7 -41 CAGCGGCAT * NM:i:1 Which version of samtools is on the $PATH on the test system? Could the test system have run out of disk space again? The tests are passing on my local development instance (before and after updating to latest galaxy-central), and on TravisCI using the latest Galaxy: https://travis-ci.org/peterjc/pico_galaxy/builds/48485673 Thanks, Peter

2 5

GalaxyAdmins Online Meetup on 19 February
by Dave Clements 31 Mar '15

31 Mar '15

Hello all, The first GalaxyAdmins <https://wiki.galaxyproject.org/Community/GalaxyAdmins> meetup of 2015 <https://wiki.galaxyproject.org/Community/GalaxyAdmins/Meetups/2015_02_19> will happen online on Thursday, 19 February <http://bit.ly/1yZJSkN>. GalaxyAdmins <https://wiki.galaxyproject.org/Community/GalaxyAdmins> is a special interest group for Galaxy community members who are responsible for Galaxy installations. A summary of both the user and admin/developer 2014 Galaxy Community Questionnaires <https://wiki.galaxyproject.org/News/2014Questionnaire> will be presented by Dave Clements <https://wiki.galaxyproject.org/DaveClements>, a *Galaxy Project Update* will be offered by a Galaxy Team member <https://wiki.galaxyproject.org/GalaxyTeam>, and Hans-Rudolf Hotz <https://wiki.galaxyproject.org/HansrudolfHotz> will lead a short discussion on *GalaxyAdmins direction*. Thanks to everyone for letting us know what dates and times worked best for you. This time and day of the week worked for a remarkable 91% of respondents. We are looking forward to starting this group up again, Hans-Rudolf Hotz and Dave Clements -- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ https://wiki.galaxyproject.org/

3 4

Re: [galaxy-dev] data collections - workflow - bug?
by John Chilton 30 Mar '15

30 Mar '15

Sorry again about this mixup (bringing conversation back on channel). Does this happen everytime you run a workflow with collections? Do you know how many job handlers Galaxy is configured with? -John On Mon, Jan 26, 2015 at 11:24 AM, John Chilton <jmchilton(a)gmail.com> wrote: > Ugh.... misread sqlalchemy as sqlite. I will work on this. > > -John > > On Mon, Jan 26, 2015 at 11:16 AM, Torsten Houwaart > <houwaart(a)informatik.uni-freiburg.de> wrote: >> Hi John, >> >> i ran this on the Galaxy server in Freiburg: galaxy.uni-freiburg.de >> Björn (who sits immediately opposite of my desk :) ) told me it's a Postres >> database and that shouldn't be the problem. >> >> Best, >> Torsten >> >> >> On 26.01.2015 16:48, John Chilton wrote: >>> >>> Hey, >>> >>> Really intensive database operations including datasest collections >>> but other things too like multi-running tools or workflows over many >>> individual datasets for instance - can very easily overwhelm the >>> default sqlite database. This is frustrating and shouldn't happen - >>> but it does unfortunately. I would recommend using a postgres database >>> when testing out dataset collections. The good news is that it is >>> easier than ever to get a fully fledged production-quality server >>> thanks to Bjoern's docker server >>> (https://github.com/bgruening/docker-galaxy-stable) - it comes bundled >>> with Postgres and Slurm so it should be able to handle the collection >>> operations. If you need to run Galaxy on a non-containerized server >>> (for instance because that is where the software is) more information >>> on setting up Galaxy can be found here >>> https://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer. >>> >>> Here is a Trello card to track progress on the database optimization >>> efforts if you are interested https://trello.com/c/UPLsMKQI. >>> >>> Very sorry. >>> >>> -John >>> >>> >>> >>> -John >>> >>> On Mon, Jan 26, 2015 at 9:35 AM, Torsten Houwaart >>> <houwaart(a)informatik.uni-freiburg.de> wrote: >>>> >>>> Hello Galaxy Devs, >>>> >>>> I was using data collections (for the first time) for a new workflow of >>>> ours >>>> and I ran into this problem. There was no complaint by the >>>> workflow-editor >>>> and I could start the workflow but then <see below> happened. >>>> If you need more information about the workflow or otherwise let me know. >>>> >>>> Best, >>>> Torsten H. >>>> >>>> >>>> job traceback: >>>> Traceback (most recent call last): >>>> File >>>> "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", >>>> line 565, in finish_job >>>> job_state.job_wrapper.finish( stdout, stderr, exit_code ) >>>> File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line >>>> 1250, in finish >>>> self.sa_session.flush() >>>> File >>>> >>>> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/scoping.py", >>>> line 114, in do >>>> return getattr(self.registry(), name)(*args, **kwargs) >>>> File >>>> >>>> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/session.py", >>>> line 1718, in flush >>>> self._flush(objects) >>>> File >>>> >>>> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/session.py", >>>> line 1789, in _flush >>>> flush_context.execute() >>>> File >>>> >>>> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/unitofwork.py", >>>> line 331, in execute >>>> rec.execute(self) >>>> File >>>> >>>> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/unitofwork.py", >>>> line 475, in execute >>>> uow >>>> File >>>> >>>> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/persistence.py", >>>> line 59, in save_obj >>>> mapper, table, update) >>>> File >>>> >>>> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/persistence.py", >>>> line 485, in _emit_update_statements >>>> execute(statement, params) >>>> File >>>> >>>> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py", >>>> line 1449, in execute >>>> params) >>>> File >>>> >>>> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py", >>>> line 1584, in _execute_clauseelement >>>> compiled_sql, distilled_params >>>> File >>>> >>>> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py", >>>> line 1698, in _execute_context >>>> context) >>>> File >>>> >>>> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py", >>>> line 1691, in _execute_context >>>> context) >>>> File >>>> >>>> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/default.py", >>>> line 331, in do_execute >>>> cursor.execute(statement, parameters) >>>> DBAPIError: (TransactionRollbackError) deadlock detected >>>> DETAIL: Process 3144 waits for ShareLock on transaction 2517124; blocked >>>> by >>>> process 3143. >>>> Process 3143 waits for ShareLock on transaction 2517123; blocked by >>>> process >>>> 3144. >>>> HINT: See server log for query details. >>>> 'UPDATE workflow_invocation SET update_time=%(update_time)s WHERE >>>> workflow_invocation.id = %(workflow_invocation_id)s' {'update_time': >>>> datetime.datetime(2015, 1, 26, 14, 20, 4, 155440), >>>> 'workflow_invocation_id': >>>> 5454} >>>> >>>> >>>> ___________________________________________________________ >>>> Please keep all replies on the list by using "reply all" >>>> in your mail client. To manage your subscriptions to this >>>> and other Galaxy lists, please use the interface at: >>>> https://lists.galaxyproject.org/ >>>> >>>> To search Galaxy mailing lists use the unified search at: >>>> http://galaxyproject.org/search/mailinglists/ >> >>

2 2

liftOver functional tests failed
by Anne Pajon 09 Mar '15

09 Mar '15

Dear All, I am still trying to migrate my production server to new hardware and would like to have all functional tests pass before turning off my old server. I am facing an issue with liftOver functional tests, they failed when running 'sh run_tests.sh -id liftOver1' on the command line with these errors: ====================================================================== ERROR: test_tool_000000 (functional.test_toolbox.TestForTool_liftOver1) Convert genome coordinates ( liftOver1 ) > Test-1 ---------------------------------------------------------------------- Traceback (most recent call last): File "/mnt/nfs/data/galaxy/galaxy-dist/test/functional/test_toolbox.py", line 122, in test_tool self.do_it( td ) File "/mnt/nfs/data/galaxy/galaxy-dist/test/functional/test_toolbox.py", line 37, in do_it data_list, jobs = galaxy_interactor.run_tool( testdef, test_history ) File "/mnt/nfs/data/galaxy/galaxy-dist/test/base/interactor.py", line 218, in run_tool raise Exception( datasets_object[ 'message' ] ) Exception: {u'type': u'error', u'data': {u'to_dbkey': u'An invalid option was selected, please verify'}} -------------------- >> begin captured logging << -------------------- ====================================================================== ERROR: test_tool_000001 (functional.test_toolbox.TestForTool_liftOver1) Convert genome coordinates ( liftOver1 ) > Test-2 ---------------------------------------------------------------------- Traceback (most recent call last): File "/mnt/nfs/data/galaxy/galaxy-dist/test/functional/test_toolbox.py", line 122, in test_tool self.do_it( td ) File "/mnt/nfs/data/galaxy/galaxy-dist/test/functional/test_toolbox.py", line 37, in do_it data_list, jobs = galaxy_interactor.run_tool( testdef, test_history ) File "/mnt/nfs/data/galaxy/galaxy-dist/test/base/interactor.py", line 218, in run_tool raise Exception( datasets_object[ 'message' ] ) Exception: {u'type': u'error', u'data': {u'to_dbkey': u'An invalid option was selected, please verify'}} -------------------- >> begin captured logging << -------------------- ====================================================================== ERROR: test_tool_000002 (functional.test_toolbox.TestForTool_liftOver1) Convert genome coordinates ( liftOver1 ) > Test-3 ---------------------------------------------------------------------- Traceback (most recent call last): File "/mnt/nfs/data/galaxy/galaxy-dist/test/functional/test_toolbox.py", line 122, in test_tool self.do_it( td ) File "/mnt/nfs/data/galaxy/galaxy-dist/test/functional/test_toolbox.py", line 37, in do_it data_list, jobs = galaxy_interactor.run_tool( testdef, test_history ) File "/mnt/nfs/data/galaxy/galaxy-dist/test/base/interactor.py", line 218, in run_tool raise Exception( datasets_object[ 'message' ] ) Exception: {u'type': u'error', u'data': {u'to_dbkey': u'An invalid option was selected, please verify'}} -------------------- >> begin captured logging << -------------------- ====================================================================== ERROR: test_tool_000003 (functional.test_toolbox.TestForTool_liftOver1) Convert genome coordinates ( liftOver1 ) > Test-4 ---------------------------------------------------------------------- Traceback (most recent call last): File "/mnt/nfs/data/galaxy/galaxy-dist/test/functional/test_toolbox.py", line 122, in test_tool self.do_it( td ) File "/mnt/nfs/data/galaxy/galaxy-dist/test/functional/test_toolbox.py", line 37, in do_it data_list, jobs = galaxy_interactor.run_tool( testdef, test_history ) File "/mnt/nfs/data/galaxy/galaxy-dist/test/base/interactor.py", line 218, in run_tool raise Exception( datasets_object[ 'message' ] ) Exception: {u'type': u'error', u'data': {u'to_dbkey': u'An invalid option was selected, please verify'}} -------------------- >> begin captured logging << -------------------- But when tested using the GUI, they did pass. I have configured the alignseq.loc and liftOver.loc, they both contained TAB delimited data and file locations have been checked. I’ve downloaded 5.bed into the GUI, used for testing liftOver, and configure the parameters according to those set in the test: <test> <param name="input" value="5.bed" dbkey="hg18" ftype="bed" /> <param name="to_dbkey" value="panTro2" /> <param name="minMatch" value="0.95" /> <param name="choice" value="0" /> <output name="out_file1" file="5_liftover_mapped.bed"/> <output name="out_file2" file="5_liftover_unmapped.bed"/> </test> [cid:5D5398C5-3D95-4074-9A86-81486A8CD20F@cri.camres.org] [cid:45B5289B-C7A8-46CE-A5B2-C879D26B1D18@cri.camres.org] And here are the results: [cid:D4910E74-8A0E-4C58-9D17-46C37DEAF99F@cri.camres.org] Any ideas of what could be that the functional tests on the command line failed? Any help to where to look would be greatly appreciated. Many thanks in advance. Best wishes, Anne. -- Dr Anne Pajon - Bioinformatics Core Cancer Research UK - Cambridge Institute Li Ka Shing Centre, Robinson Way, Cambridge CB2 0RE anne.pajon(a)cruk.cam.ac.uk<mailto:anne.pajon@cruk.cam.ac.uk> | +44 (0)1223 769 631

2 2

RFC: new param attribute for Galaxy tool XML
by Björn Grüning 04 Mar '15

04 Mar '15

Hi all, we are planning to work on a project to implement a Galaxy fuse based shell. Probably starting with the work from Clare [1]. Next to our Galaxy IPython integration it should attract more bioinformaticians and should offer a new way to interact with Galaxy. This includes moving, deleting datasets, but also executing tools and workflows. For the latter I would like to have some sort of bash auto-completion. Type in your tool/workflow and you will see all the parameters you can/should modify, in addition to your normal help page. Currently the parameter identifiers (<param name="foo">) do not need to be meaningful. In fact many tools invent their own unique names, to identify a parameter in the cheetah section. This name is always unique but it's hard to guess it's meaning from just the name and are also not mappable to the original parameter name. This makes tool execution from the API sometimes hard. I would like to propose a new attribute for all <param> tags. This should specify a unique value that 100% matches the original parameter name of the underlying tool. This attribute could be used to: * automatically enhance the help text of each parameter. Currently best practise it to include this parameter in brackets at the end of each help text. We can do this now automatically, or only show it by mouse over etc ... * In a Galaxy shell, the user could just type in a normal command (-i history1/foo.bam -o history1/bar.sam) and Galaxy shell would be able to map this parameter to the correct <param> tag in Galaxy I greatly appreciate any comments! Thanks, Bjoern [1] https://github.com/claresloggett/gvl_commandline_utilities/blob/master/gala…

3 3

Interested in GSOC idea "Docker Swarm and Compose" & ask for advices.
by afe.young＠gmail.com 02 Mar '15

02 Mar '15

Hi I'm a first-year student for my Master Degree in China. I'm really interested in GSOC and viewed a lot of 2015 ideas recently. "Docker Swarm and Compose" is the most attractive one for me. I've some experience in opensource and have developed a lot of projects with python. I read some books about docker recently and have deployed galaxy with https://github.com/bgruening/docker-galaxy-stable/ . I wondering what else I can do to prepare for the idea "Docker Swarm and Compose". It will be awesome to me to work with you. So I'm here asking for advices. Thanks in advance.

3 2

Tool Testing: field inputs that occur inside repeat tag
by Dooley, Damion 28 Feb '15

28 Feb '15

Helop aI've been testing (using planemo) some tools scheduled for toolshed publication. My first test works, but Galaxy reports that the second one fails with: "Parameter %s requires a value, but has no legal values defined" % self.name AssertionError: Parameter filter_column requires a value, but has no legal values defined requests.packages.urllib3.connectionpool: DEBUG: "POST /api/tools HTTP/1.1" 500 None I presume its because of extra stuff needed to name variables have to be referred-to when they are inside a repeat section? Or is it just something pecular to do with data_table_sources? <test> <param name="blastxml_file" value="blast_reporting_1.blastxml"/> <output name="tabular_file" file="blast_reporting_1a.tabular"/> <param name="out_format" value="std"/> <param name="column_labels" value="" /> <param name="drop_redundant_hits" value="true"/> </test> <test> <param name="blastxml_file" value="blast_reporting_1.blastxml"/> <output name="tabular_file" file="blast_reporting_1b.tabular"/> <param name="out_format" value="std"/> <param name="column_labels" value="" /> <param name="drop_redundant_hits" value="true"/> <param name="filter_column" value="pident"/> <param name="filter_comparison" value="gte"/> <param name="filter_value" value="97"/> </test> The form repeat area: <repeat name="filter_num" title="Numeric Filter" min="0" max="4"> <param name="filter_column" type="select" label="Col"> <options from_data_table="blast_reporting_fields"> <filter type="static_value" value="numeric" column="type" /> <filter type="sort_by" column="name"/> </options> </param> <repeat name="constraint" title="Constraint" min="1" max="3"> <param name="filter_comparison" type="select" label="Comparison"> <option value="gte">>= </option> <option value="gt">></option> <option value="lt">< </option> <option value="lte"><= </option> <option value="==">equal to </option> <option value="!=">not equal to </option> </param> <param name="filter_value" type="text" value="" label="Value"> <validator type="regex" message="Please input a number">([0-9]+|[0-9]*\.[0-9]+)</validator> </param> </repeat> </repeat> The working command section that sends the filter field info is: #for $i, $my_repeat in enumerate( $filter_num ) $my_repeat.filter_column: #for $i, $my_repeat2 in enumerate( $my_repeat.constraint ) $my_repeat2.filter_comparison $my_repeat2.filter_value, #end for ; #end for Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC Centre for Disease Control 655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada

3 7

problems installing tool dependency
by Sajdak, Doris 27 Feb '15

27 Feb '15

Hi all, I am trying to install the Genome Diversity tools from the Miller Lab that are in the Galaxy toolshed. I've been able to get all dependencies installed except Quicktree. Whenever I try to install it, I get the error: Error downloading from URL ftp://ftp.sanger.ac.uk/pub4/resources/software/quicktree/quicktree.tar.gz: <urlopen error ftp error: 550 pub4: No such file or directory> This is the tool_dependencies.xml file for installing quicktree: <?xml version="1.0"?> <tool_dependency> <package name="quicktree" version="1.1"> <install version="1.0"> <actions>  <action type="download_by_url" target_filename="quicktree_1.1.tar.gz">ftp://ftp.sanger.ac.uk/pub4/resources/software/quicktree/quicktree.tar.gz</action>  <action type="shell_command">make quicktree</action>  <action type="shell_command">cp -R bin $INSTALL_DIR</action>  <action type="set_environment"> <environment_variable name="PATH" action="prepend_to">$INSTALL_DIR/bin</environment_variable> </action> </actions> </install> </package> </tool_dependency> I know this is not their file I'm downloading but the thing is, if I copy and paste the URL, I can download it with no problem. I'm not sure why it's reporting that it can't be found but the xml looks right to me. I've tried contacting the Miller Lab about this but haven't heard from anyone so I'm hoping someone here can tell me how to move past this problem. Is there a way to install this tool manually and have the main Genome Diversity tools recognize it on the Galaxy server? Also, I noticed when looking at the Tool Dependency Definitions at https://toolshed.g2.bx.psu.edu/ there are errors going back many versions for this tool (see Test runs - Installation errors - Tool dependencies). Thank you for any help you can provide. Dori ********************************** Dori Sajdak Senior Systems Administrator State University of NY at Buffalo Center for Computational Research 701 Ellicott St Buffalo, NY 14203 Phone: (716) 881-8934 Fax: (716) 849-6656 Web: http://ccr.buffalo.edu **********************************

3 7

Database deadlock with large workflows + dataset collections
by Aaron Petkau 27 Feb '15

27 Feb '15

Hey, I wanted to know if anyone else has had experience with database deadlock when using dataset collections and running a large number of samples through a workflow. Traceback (most recent call last): File "/Warehouse/Applications/irida/galaxy/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 565, in finish_job job_state.job_wrapper.finish( stdout, stderr, exit_code ) File "/Warehouse/Applications/irida/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 1250, in finish self.sa_session.flush() File "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/scoping.py", line 114, in do return getattr(self.registry(), name)(*args, **kwargs) File "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 1718, in flush self._flush(objects) File "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 1789, in _flush flush_context.execute() File "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/unitofwork.py", line 331, in execute rec.execute(self) File "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/unitofwork.py", line 475, in execute uow File "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/persistence.py", line 59, in save_obj mapper, table, update) File "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/persistence.py", line 485, in _emit_update_statements execute(statement, params) File "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1449, in execute params) File "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement compiled_sql, distilled_params File "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1698, in _execute_context context) File "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1691, in _execute_context context) File "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/default.py", line 331, in do_execute cursor.execute(statement, parameters) DBAPIError: (TransactionRollbackError) deadlock detected DETAIL: Process 25859 waits for ShareLock on transaction 144373; blocked by process 25858. Process 25858 waits for ShareLock on transaction 144372; blocked by process 25859. HINT: See server log for query details. 'UPDATE workflow_invocation SET update_time=%(update_time)s WHERE workflow_invocation.id = %(workflow_invocation_id)s' {'update_time': datetime.datetime(2015, 2, 27, 3, 51, 57, 81403), 'workflow_invocation_id': 48} I saw this post http://dev.list.galaxyproject.org/data-collections-workflow-bug-td4666496.h… with a similar issue and the solution was to make sure not to use a sqlite database, but I'm using a postgres database and still encountered this issue. This was after running a very large number of samples (~200) using dataset collections. Just wondering if anyone else was running into this issue? Thanks, Aaron

2 2

$GALAXY_TEST_DB_TEMPLATE failing with KeyError: <VerNum(127)>
by Peter Cock 27 Feb '15

27 Feb '15

Hi all, I've been using the SQLite template database feature to speed up functional testing. Rather than starting for the initial schema and applying all the incremental changes (as done when installing a fresh Galaxy instance), this lets the test suite start with a recent version of the database, by setting the environment variable $GALAXY_TEST_DB_TEMPLATE to a URL to fetch a template database, e.g. from: https://github.com/jmchilton/galaxy-downloads Database schema version 120 was working, changing to 127 broke: https://github.com/peterjc/pico_galaxy/commit/412d3fe88b8011f0b690956d129fd… TravisCI using v120, worked: https://travis-ci.org/peterjc/pico_galaxy/builds/52308064 TravisCI using v127, broke: https://travis-ci.org/peterjc/pico_galaxy/builds/52398778 galaxy.model.migrate.check DEBUG 2015-02-27 10:05:33,274 pysqlite>=2 egg successfully loaded for sqlite dialect Traceback (most recent call last): File "./scripts/functional_tests.py", line 602, in <module> sys.exit( main() ) File "./scripts/functional_tests.py", line 409, in main app = UniverseApplication( **kwargs ) File "/home/travis/build/peterjc/pico_galaxy/galaxy-master/lib/galaxy/app.py", line 49, in __init__ self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file ) File "/home/travis/build/peterjc/pico_galaxy/galaxy-master/lib/galaxy/config.py", line 779, in _configure_models create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self ) File "/home/travis/build/peterjc/pico_galaxy/galaxy-master/lib/galaxy/model/migrate/check.py", line 61, in create_or_verify_database migrate() File "/home/travis/build/peterjc/pico_galaxy/galaxy-master/lib/galaxy/model/migrate/check.py", line 57, in migrate migrate_to_current_version( engine, db_schema ) File "/home/travis/build/peterjc/pico_galaxy/galaxy-master/lib/galaxy/model/migrate/check.py", line 117, in migrate_to_current_version changeset = schema.changeset( None ) File "/home/travis/build/peterjc/pico_galaxy/galaxy-master/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/schema.py", line 80, in changeset changeset = self.repository.changeset(database, start_ver, version) File "/home/travis/build/peterjc/pico_galaxy/galaxy-master/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/repository.py", line 225, in changeset changes = [self.version(v).script(database, op) for v in versions] File "/home/travis/build/peterjc/pico_galaxy/galaxy-master/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/repository.py", line 189, in version return self.versions.version(*p, **k) File "/home/travis/build/peterjc/pico_galaxy/galaxy-master/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/version.py", line 140, in version return self.versions[VerNum(vernum)] KeyError: <VerNum(127)> This is now testing against the (beta) Galaxy on GitHub: https://github.com/galaxyproject/galaxy Is it possible this is out of synch with the SQLite templates John posts here? https://github.com/jmchilton/galaxy-downloads Peter

2 2