- galaxy-dev - lists.galaxyproject.org

Running functional tests is too slow
by Peter Cock 31 Oct '13

31 Oct '13

Hi all, I'd like to echo Florent's concern that run_functional_tests.sh is too slow, and that this discourages Tool Authors from adding more tests to their tools: https://trello.com/c/wL21d2do/1017-functional-tests-take-too-long (And encourage people to vote up that issue ;) ) Florent has identified one clear bottleneck is the creation of a fresh SQLite database each time, upon which the growing number of schema migration calls must be performed. Could we not cache an empty but up to date copy of this database? i.e. Something like this: 1. run_functional_tests.sh starts 2. If it exists, copy empty_test_sqlite_database.db, otherwise create new test SQLite database as now. 3. Check the schema version. 4. If the temporary SQlite database is out of date, run the migration, and then save a copy of this now up to date database as empty_test_sqlite_database.db, 5. run the tests Perhaps the empty SQLite database could even be cached in BitBucket too, for a faster first test run with a clean checkout? Separately, running the tests themselves seems overly slow - can anything be tweaked here? For example, is there any point executing the external set_meta script in the test environment? Regards, Peter P.S. On a related note, my achievement last Friday was to get TravisCI doing continuous integration testing of a GitHub repository of Galaxy tools: https://travis-ci.org/peterjc/galaxy_blast https://github.com/peterjc/galaxy_blast For those not familiar with this system, the idea is that via a special .travis.yml configuration file, each time new commits are pushed to GitHub, the latest code is checked out and tested on a virtual machine. i.e. Continuous in the sense of running the tests after each change, rather than just once a night. Right now the tests for the BLAST+ suite and associated tool wrappers like Blast2GO takes from 15 to 20 minutes, which I feel could be much improved.

2 5

Filtering number of outputs with a conditional
by Alexandre Loywick 31 Oct '13

31 Oct '13

Hello Galaxians, I am currently working on a tool to format reports but I have a problem with the <filter> tag. A little example is easier to figure than a long explanation so... My xml looks like this: <tool name="My tool" id="my tool"> <command interpreter="python">$SCRIPTTYPE.SCRIPT $REPORT $SECONDPARAM $OUTFILE $MAP</command> <inputs> <param type="data" label="My input file" name="REPORT" help="Just for the example"/> <conditional name="SCRIPTTYPE"> <param type="select" label="Script to use" name="SCRIPT"> <option value="script_x.py">X</option> <option value="script_y.py">Y</option> <option value="script_z.py">Z</option> </param> <when value="script_x.py"> <param type="select" name="SECONDPARAM" label="My parameter"> <option value="foo">foo</option> <option value="bar">bar</option> </param> </when> <when value="script_y.py"> <param type="select" name="SECONDPARAM" label="My parameter"> <option value="A">A</option> <option value="B">B</option> </param> </when> <when value="script_z.py"> <param type="select" name="SECONDPARAM" label="My parameter"> <option value="1">1</option> <option value="2">2</option> </param> </when> </conditional> </inputs> <outputs> <data type="data" format="tabular" name="OUTFILE" label="My outfile"> <filter> SCRIPTTYPE['SCRIPT'] == script_x.py</filter> </data> <data type="data" name="OUTFILE" label="My_outfile.tgz"> <filter> SCRIPTTYPE['SCRIPT'] == script_y.py</filter> </data> <data type="data" format="tabular" name="OUTFILE" label="My other outfile"> <filter> SCRIPTTYPE['SCRIPT'] == script_z.py</filter> </data> <data type="data" format="tabular" name="MAPFILE" label="My map"> <filter> SCRIPTTYPE['SCRIPT'] == script_z.py</filter> </data> </outputs> </tool> ===================== I try to make my xml work that way: When I select my script x, there is only one output labeled "my outfile". When I select my script y, there is only one output labeled "My_outfile.tgz" When I select my script z, there are two outputs. I had it working properly with just x or y, but when I add z to my xml, the thing gets all buggy. Instead of having one output when I select x or y, I always have two outputs "my other outfile" and "my map" as if I had selected z. Do you have any idea how I could solve that? Thank you, best regards, Alexandre

2 3

Blastdb and blastdbp datatypes not enforced
by Ganote, Carrie L 30 Oct '13

30 Oct '13

Hi List, It seems that the blastdb and blastdbp data format is not respected on my local install; is it just not generally defined, or am I missing something in /lib/galaxy/datatypes or in datatypes_conf? The most recent blasp wrapper includes this for the input parameter: type="data" format="blastdbp" When I choose the BLAST Database from history option for db, I get every valid dataset in my history in the list. Is this because it's a composite set? Thanks for any tips, Carrie Ganote

2 2

What to do with toolshed tools that get stuck "Installing"
by Ira Cooke 30 Oct '13

30 Oct '13

Hi All, The short version of my problem is that I often seem to find myself in a situation where my tool gets stuck in the "Installing" phase ... and I don't know how to get feedback on it to see where it is failing. What I guess I'm after is just the stdout of the tool installation .. but since the tool install never completes I can never see this. What I usually do is try to recreate the environment that galaxy sees during tool install in a normal shell .. and then run it manually .. which sometimes tells me the problem ... but not always (see below). The longer version of my problem is below. In my case I am struggling with the following tool http://testtoolshed.g2.bx.psu.edu/view/iracooke/package_protk_1_2_5 Which uses the new "setup_ruby_environment" tag ... so the tool should do the following (1) Install the dependency .. Ruby 2.0 ... which it does successfully (2) Install my protk rubygem from rubygems.org .. which it gets stuck doing. The last few lines from my galaxy server log during the failed install are; 10.0.2.2 - - [28/Oct/2013:21:54:26 +0000] "POST /admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 - "http://localhost:8088/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36" [localhost] local: touch /home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/env.sh [localhost] local: echo 'RUBYLIB=/home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/lib/:$RUBYLIB; export RUBYLIB' >> /home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/env.sh 10.0.2.2 - - [28/Oct/2013:21:54:30 +0000] "POST /admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 - "http://localhost:8088/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36" [localhost] local: echo 'PATH=/home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/bin:$PATH; export PATH' >> /home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/env.sh [localhost] local: echo 'GALAXY_RUBY_HOME=/home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/bin; export GALAXY_RUBY_HOME' >> /home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/env.sh [localhost] local: rm -rf ./database/tmp/tmp-toolshed-mtdQaLI7O ruby version 2.0 installed in /home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63 [localhost] local: touch /home/vagrant/tool_dependencies/ruby/2.0/iracooke/package_protk_1_2_5/876e44dd4609/env.sh [localhost] local: echo 'RUBYLIB=/home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/lib/:$RUBYLIB; export RUBYLIB' >> /home/vagrant/tool_dependencies/ruby/2.0/iracooke/package_protk_1_2_5/876e44dd4609/env.sh [localhost] local: echo 'PATH=/home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/bin:$PATH; export PATH' >> /home/vagrant/tool_dependencies/ruby/2.0/iracooke/package_protk_1_2_5/876e44dd4609/env.sh [localhost] local: echo 'GALAXY_RUBY_HOME=/home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/bin; export GALAXY_RUBY_HOME' >> /home/vagrant/tool_dependencies/ruby/2.0/iracooke/package_protk_1_2_5/876e44dd4609/env.sh [localhost] local: rm -rf ./database/tmp/tmp-toolshed-mtdJYyhO0 10.0.2.2 - - [28/Oct/2013:21:54:34 +0000] "POST /admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 - "http://localhost:8088/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36" 10.0.2.2 - - [28/Oct/2013:21:47:22 +0000] "POST /admin_toolshed/manage_repositories HTTP/1.1" 302 - "http://localhost:8088/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36" 10.0.2.2 - - [28/Oct/2013:21:54:37 +0000] "GET /admin_toolshed/monitor_repository_installation?tool_shed_repository_ids=f597429621d6eb2b&tool_shed_repository_ids=f2db41e1fa331b3e HTTP/1.1" 200 - "http://localhost:8088/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36" With a normal tool install I would see the last line of this output repeat lots of times as installation is monitored ... but for some reason in this case monitoring just shuts down and I don't see any further updates. If I recreate the situation on the command-line (eg by manually setting the RUBYLIB and GEM_HOME environment variables appropriately I can install my gem without any errors. More details on my setup: I'm using a vagrant VM for testing which is based on Ubuntu precise64 ... with a minimal number of packages installed ... (My VM is documented in full here: https://bitbucket.org/iracooke/protk_vagrant/src/0ef8d86cc3dcec4ee3a9a83edf… ) I'm testing based on the current default branch of galaxy-central .... which I regularly tear down and re-clone from scratch ... always with the same result. Any help or pointers to how to resolve this much appreciated. Cheers Ira

4 10

Workflows with <param type="data" multiple="true" ...>
by Peter Cock 30 Oct '13

30 Oct '13

Hi John, I've just tried a workflow using my MIRA4 de novo wrapper: http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira4_assembler To keep it simple, in this example I just had two input FASTQ files, the forward and reverse files for paired end MiSeq data. This seemed to work fine for re-running the workflow :) However, on loading the workflow to edit it, only the first input file was drawn linked to the MIRA4 tool. On making some minor workflow changes (renaming some output), and trying to run it, the second input file seemed not to be used. Is this something you are aware of? The good news is that I could "re-draw" the missing connection from the second FASTQ file in the workflow editor, and that appears to be working... Peter -- This was using the current stable release, $ hg branch default $ hg heads changeset: 10421:a477486bf18e branch: stable tag: tip user: Nate Coraor <nate(a)bx.psu.edu> date: Thu Sep 26 11:02:58 2013 -0400 summary: Bugfix for tool-to-destination mapping, tool ids are lowercased but the mapping id was not lowercased. changeset: 10411:c42567f43aa7 user: greg date: Mon Aug 19 13:19:56 2013 -0400 summary: Filter invalid objects when generating the list of repository_dependencies objects that are associated with a tool shed repository installed into Galaxy.

2 2

Tool refuses to appear in Toolshed package contents
by Lukasse, Pieter 30 Oct '13

30 Oct '13

Hi , I have a toolshed repository where I expect 5 tools to appear in the "Contents of this repository" section but one of them fails to appear in the list (see screenshot). Where can I find out what is going wrong? Thanks, Pieter. [cid:image001.png@01CED0BA.3733D000] Pieter Lukasse Wageningen UR, Plant Research International Departments of Bioscience and Bioinformatics Wageningen Campus, Building 107, Droevendaalsesteeg 1, 6708 PB, Wageningen, the Netherlands +31-317480891; skype: pieter.lukasse.wur http://www.pri.wur.nl<http://www.pri.wur.nl/>

4 16

How we found the filenames of all of the Galaxy datasets
by Yaron de Leeuw 30 Oct '13

30 Oct '13

Hi. During IT housekeeping, we have decided that we must know the filenames of all of the galaxy datasets in our local deployment. Searching the docs and the web yielded no reasonable answer, so we figured we'll just write our own tool. At first, we thought about a Galaxy tool, but found no way to extract all the datasets of all histories. So we went deeper. To the SQL itself. Without further ado, I give you the query: SELECT substring(to_char(history_dataset_association.dataset_id, '000000') from 2 for 3) || '/dataset_' || history_dataset_association.dataset_id || '.dat' , history_dataset_association.create_time, history_dataset_association.name, history_dataset_association.extension, history.name AS history_name,galaxy_user.username FROM history_dataset_association JOIN history ON history_dataset_association.history_id = history.id JOIN galaxy_user ON history.user_id = galaxy_user.id WHERE history_dataset_association.deleted = false AND history_dataset_association.purged = false ORDER BY username, history_name, create_time; We use PostgreSQL, so to run this you save the query as `get_filenames.sql`, and run: psql -U galaxy galaxy_prod -f get_filenames.sql -A > galaxy_export_filenames.csv This prints out a line structured as: 003/dataset_3748.dat|2012-10-09 12:07:37.786751| index.fastq|fastq|nan|sean Which means: folder/filename | time of creation | name as appears in history | format | history name | history owner Only files that were not deleted in galaxy are printed (you can remove the "WHERE" line if you want them all). By the way, one of the main struggles was finding the folder/filename. We have searched the database repeatedly for a filename field to no avail, until someone recognised the fact that they can be derived from the dataset_id. If you zero pad the id to six chars, the first three chars would be your folder name. The whole id is the file name. e.g.: dataset_id = 12345 folder = 012 filename = dataset_12345.dat Yaron. Search engine candies: export migrate files datasets filenames on disk

2 1

Tabular data not displaying in main panel
by Peter Cock 30 Oct '13

30 Oct '13

Hi all, Multiple users of our internal Galaxy server have reported with multiple browsers that tabular data is not displayed in the main central panel. The preview in the history works, but clicking on the "eye" icon gives a blank page. I can confirm this in Firefox 24.0 and Safari 6.0.5, the HTML for the central frame looks blank but is actually a script when viewed via the frame source. The tabular file data is there in the script's first_data_chunk entry - but does not get shown. An except is shown below. Has anyone else noticed this problem? I believe it began when I last updated this Galaxy instance, but was not noticed immediately. Thanks, Peter ----------------------- I believe we are running the current stable release, or at least the latest commit on the default branch from galaxy-dist $ hg pull warning: bitbucket.org certificate with fingerprint 24:9c:45:8b:9c:aa:ba:55:4e:01:6d:58:ff:e4:28:7d:2a:14:ae:3b not verified (check hostfingerprints or web.cacerts config setting) real URL is https://bitbucket.org/galaxy/galaxy-dist pulling from http://bitbucket.org/galaxy/galaxy-dist searching for changes no changes found $ hg update 0 files updated, 0 files merged, 0 files removed, 0 files unresolved $ hg branch default [galaxy@ppserver galaxy-dist]$ hg log -b default | head changeset: 10411:c42567f43aa7 user: greg date: Mon Aug 19 13:19:56 2013 -0400 summary: Filter invalid objects when generating the list of repository_dependencies objects that are associated with a tool shed repository installed into Galaxy. changeset: 10410:e9ee9c5d30ae user: Dave Bouvier <dave(a)bx.psu.edu> date: Mon Aug 19 13:08:45 2013 -0400 summary: Remove duplicate method from the install and test's functional_tests.py. ... ----------------------- <!DOCTYPE HTML> <html>  <head> <title>Dataset Display</title> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <link href="/galaxy/static/style/base.css?v=1380731103" media="screen" rel="Stylesheet" type="text/css" /> <script src="/galaxy/static/scripts/libs/jquery/jquery.js?v=1380731103" type="text/javascript"></script> <script src="/galaxy/static/scripts/libs/jquery/jquery.migrate.js?v=1380731103" type="text/javascript"></script> <script src="/galaxy/static/scripts/libs/jquery/select2.js?v=1380731103" type="text/javascript"></script> <script src="/galaxy/static/scripts/libs/bootstrap.js?v=1380731103" type="text/javascript"></script> <script src="/galaxy/static/scripts/libs/underscore.js?v=1380731103" type="text/javascript"></script> <script src="/galaxy/static/scripts/libs/backbone/backbone.js?v=1380731103" type="text/javascript"></script> <script src="/galaxy/static/scripts/libs/backbone/backbone-relational.js?v=1380731103" type="text/javascript"></script> <script src="/galaxy/static/scripts/libs/handlebars.runtime.js?v=1380731103" type="text/javascript"></script> <script src="/galaxy/static/scripts/galaxy.base.js?v=1380731103" type="text/javascript"></script> <script src="/galaxy/static/scripts/templates/compiled/template-popupmenu-menu.js?v=1380731103" type="text/javascript"></script> <script src="/galaxy/static/scripts/mvc/ui.js?v=1380731103" type="text/javascript"></script> <script type="text/javascript"> // console protection window.console = window.console || { log : function(){}, debug : function(){}, info : function(){}, warn : function(){}, error : function(){}, assert : function(){} }; // Set up needed paths. var galaxy_paths = new GalaxyPaths({ root_path: '/galaxy/', image_path: '/galaxy/static/images' }); </script> <script src="/galaxy/static/scripts/libs/require.js?v=1380731103" type="text/javascript"></script> <script src="/galaxy/static/scripts/templates/compiled/helpers-common-templates.js?v=1380731103" type="text/javascript"></script> <script type="text/javascript"> require.config({ baseUrl: "/galaxy/static/scripts", shim: { "libs/backbone/backbone": { exports: "Backbone" }, "libs/backbone/backbone-relational": ["libs/backbone/backbone"] } }); require(['mvc/data'], function(data) { data.createTabularDatasetChunkedView( _.extend( {"misc_blurb": "7,891 lines", "peek": "<table cellspacing=\"0\" cellpadding=\"3\"><tr><th>1</th><th>2</th><th>3</th></tr><tr><td>Gpa_EST_02_04___C05_022</td><td>GO:0005737</td><td>...</td></tr></table>", "data_type": "tabular", "deleted": false, "history_id": "cdbf837ba6516a14", "metadata_column_names": null, "visible": true, "genome_build": "?", "hid": 10, "file_size": 509095, "metadata_data_lines": 7891, "file_ext": "tabular", "id": "0ddcd2cf2375e24a", "misc_info": "\nBLASTX output identified\nInput has 4507 queries\nDone\nStandard out:\nBlast2GO Pipeline Version 2.5.0\n--------------------------------------\n\nHash for resource ec2go_19072011.txt of type ec created. 4003 associations loaded.\nBlast2GO build: 23092011\nChange ", "hda_ldda": "hda", "name": "Blast2GO Local database (May 2011)", "metadata_columns": 3, "uuid": null, "state": "ok", "metadata_comment_lines": null, "model_class": "HistoryDatasetAssociation", "metadata_dbkey": "?", "purged": false, "metadata_column_types": ["str", "str", "str"]}, { url_viz: "/galaxy/visualization", chunk_url: "/galaxy/dataset/display?dataset_id=0ddcd2cf2375e24a", first_data_chunk: {"ck_index": 1, "ck_data": "Gpa_EST_02_04___C05_022\tGO:0005737\tn-dimethylarginine dimethylaminohydrolase\n...\n"} } ), // Append view to body. $('body') ); }); </script> </head> <body> </body> </html> --

5 17

Problem With Galaxy
by Caitlin Grube 29 Oct '13

29 Oct '13

I am having an issue trying to do anything. I click on a tool, the screen pops up in the middle but then when I move my mouse over to that screen, the drop down menus all pull down in a large black screen so that I cannot get to anything. The screen also does not go away if I try to click away from it, but only if I choose one of the options from the list. I would click analyze data which would take back to where I started. This happens for every tool that I try. I did not have this problem this morning or afternoon, but I've been trying to get it to work for almost an hour. As soon as my mouse hits that center area (top, bottom, middle), the screen comes down, but only after I select a tool. Please help. Caitlin Grube

2 1

cloning Galaxy cluster on cloud Amazon EC2
by Manisha Sapre 29 Oct '13

29 Oct '13

Hello Team Galaxy trying to launch a clone cluster of galaxy on cloud amazon EC2 (Scr.1),launching instance on US East region getting error unable to console (Scr.2), insights of team will be very helpful. [cid:DC3AB594-6DEF-4B12-B6C1-6D5ACEC22EEB] [cid:CD19EA5C-961A-4777-A811-DC0FFEBAD796] Thanks Manisha

2 7