October 2013 - galaxy-dev - lists.galaxyproject.org

Workflows with multiple editors
by Eric Rasche 01 Nov '13

01 Nov '13

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm working with a number of people in my department to develop a single workflow that will be used for a course we teach. So far I've found that "sharing" a workflow with individual users/roles isn't very optimal. They email me with changes they'd like to happen, or they create their own copies which immediately branch off and don't help everyone else who might need them. As such, I'd like to propose changes and would like community feedback on my proposed changes so when I create the trello card/maybe try my hand at implementing them, they aren't completely specific to my use case. In my mind, the 'optimal' version of managing workflows would be something along the following lines: Top priorities: - - workflows can be shared as "write" or "read only" with multiple users/roles. - This would allow there to be a single copy of a workflow in cases where multiple people might make changes to a single workflow - For "read only", this allows the previous/traditional model of sharing a workflow and the receiver being able to run it as is, or to copy and modify it. Note: For my case at least, realtime multiple person editing is not needed. If a banner shows up which reads "someone started editing before you, either ask them to save + leave or save as a copy" would work in my case. Wishlist features: - - workflows are treated as VCS repositories - Note that full merge/branching probably not really necessary for this case (that would be really nice, but probably too much of a nightmare to code...) - - You can see previous iterations of workflows. - - Workflows can be "owned" by either a user or a role - users could create a workflow and transfer ownership to a role - role owned workflows are delete-able (or maybe require some X% of users in that role to confirmation deletion?) - This would sort of work like a group owned VCS repository, where multiple users can create workflows of interest to a group of people (role) and have them, by default, be available. Anyone have input on this proposal? Cheers, Eric - -- Eric Rasche Programmer II Center for Phage Technology Texas A&M University College Station, TX 77843 404-692-2048 esr(a)tamu.edu rasche.eric(a)yandex.ru -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJScWXTAAoJEMqDXdrsMcpVFH4QAMO0V4tSJGzM3PGGIwdonY0v nyuHTCJJ5qliWuapUJUK9E0UQBblnaJYooLjh2l1vgo95JaXdHEAC9NVecQLH8J2 ey5cu1xODf6JmBGlicRGxQWlLZmo9U3R4T/OhHco40I239aTpuSDPcwcBJUm1cAE czz+/2pgdCpkJqYQJ9EE956TAYdyPBYFvSrIO7ebWkSvC5He40qca0x06tgq14uR Dta+8QH8brVYX1G+TAgdh3hGlLUwIlPMiZsWOEMcwSUtyXP3I7OKx/XIOy3ZYVTu Py/XlvNlZ87yiaiwDSxjxR9qaDeGQ3J8CRCG9mqWu8O5Y9QKkyrC410I+hZH5lE+ my28z7l5rK0i05pR6o8euASJ7D1eFHEDYmBmN6j3qRcOgzzdup2oQpm6TA7ue1sd NAS4rnW7QGmfNj+9xNSQhDWSj7CeIUeHXbDe55suBX3JofCNg8Qq6uYzPd7KGj7n jvWp8tGnpwbgTmvDPlR03gJiXMZld+gGwKrlCyn1kdBkfI1r5l0UfoaIUE+8HDm3 wXviQ8GU7seYU5bJFK798owVNawjY1H3JenzIYt1XvF1z53Uc39x58Q9lxXbVfSe KgiqCfF0YuvVtOczGTAQ57qCFwlpKg2ra+X+HtAZyMDNMsb3ANgfogTqsB+7HYvF czq2QJHq9mBC5N9JLhRo =conD -----END PGP SIGNATURE-----

3 3

browse directories on galaxy server
by David Hoover 01 Nov '13

01 Nov '13

Is there any desire to create a way in galaxy to browse local directories on the galaxy server itself? Not the client machine running the web browwser, but the machine on which galaxy is running. I've created a tool that allows a user to upload a single file from their own personal directory on the galaxy server into galaxy directly, without going through Apache. However, this is very limiting. It would be very helpful to be able to browse directories that are deemed accessible to the galaxy user, and before I begin trying to develop anything I'd like to know if there is any support for such a feature. David Hoover Helix Systems Staff National Institutes of Health

3 2

User activations
by jordi＠biociphers.org 31 Oct '13

31 Oct '13

Hi all, Recently I have updated our test galaxy server in order to check some new features like user email activation using emails. I saw that in the admin page http://wiki.galaxyproject.org/Admin/UserAccounts . But I download the last version and the conf values that are supose to be in the universe_wsgi.ini are not present ( at least in the sample file). Do you know if this is a future feature or there is anything I need to do for testing it? thanks everyone for your help Jordi

2 1

November 2013 Galaxy Update Newsletter
by Dave Clements 31 Oct '13

31 Oct '13

Hello all, The November 2013 Galaxy Update is out<http://wiki.galaxyproject.org/GalaxyUpdates/2013_11> : *Highlights:* - Two new public Galaxy servers<http://wiki.galaxyproject.org/GalaxyUpdates/2013_11#New_Public_Servers> : CoSSci: Complex Social Science Gateway<http://wiki.galaxyproject.org/GalaxyUpdates/2013_11#CoSSci:_Complex_Social_…> (which has *nothing* to do with biology), and BioCiphers Lab Galaxy<http://wiki.galaxyproject.org/GalaxyUpdates/2013_11#BioCiphers_Lab_Galaxy> . - 53 new papers<http://wiki.galaxyproject.org/GalaxyUpdates/2013_11#New_Papers>, including "Expanding roles in a library-based bioinformatics service program: a case study," "DDBJ Read Annotation Pipeline: A Cloud Computing-Based Pipeline for High-Throughput Analysis of Next-Generation Sequencing Data," and "Ten Simple Rules for Reproducible Computational Research" - Who's hiring<http://wiki.galaxyproject.org/GalaxyUpdates/2013_11#Who.27s_Hiring> - Upcoming Events<http://wiki.galaxyproject.org/GalaxyUpdates/2013_11#Other_Events>, including - Save these dates! GCC2014: June 30 - July 2, Baltimore<http://wiki.galaxyproject.org/GalaxyUpdates/2013_11#GCC2014:_June_30_-_July…> - Galaxy Day, December 4, Paris<http://wiki.galaxyproject.org/GalaxyUpdates/2013_11#Galaxy_Day.2C_December_…> - UC Davis Bioinformatics Boot Camps<http://wiki.galaxyproject.org/GalaxyUpdates/2013_11#UC_Davis_Bioinformatics…> - Lifeportal launched at the University of Oslo<http://wiki.galaxyproject.org/GalaxyUpdates/2013_11#Lifeportal_at_the_Unive…> - Tool Shed contributions<http://wiki.galaxyproject.org/GalaxyUpdates/2013_11#Tool_Shed_Contributions> If you have anything you would like to see in the next *Galaxy Update<http://wiki.galaxyproject.org/GalaxyUpdates> *, please let us know. Dave Clements and the Galaxy Team <http://wiki.galaxyproject.org/GalaxyTeam> -- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ http://wiki.galaxyproject.org/

1 0

Running functional tests is too slow
by Peter Cock 31 Oct '13

31 Oct '13

Hi all, I'd like to echo Florent's concern that run_functional_tests.sh is too slow, and that this discourages Tool Authors from adding more tests to their tools: https://trello.com/c/wL21d2do/1017-functional-tests-take-too-long (And encourage people to vote up that issue ;) ) Florent has identified one clear bottleneck is the creation of a fresh SQLite database each time, upon which the growing number of schema migration calls must be performed. Could we not cache an empty but up to date copy of this database? i.e. Something like this: 1. run_functional_tests.sh starts 2. If it exists, copy empty_test_sqlite_database.db, otherwise create new test SQLite database as now. 3. Check the schema version. 4. If the temporary SQlite database is out of date, run the migration, and then save a copy of this now up to date database as empty_test_sqlite_database.db, 5. run the tests Perhaps the empty SQLite database could even be cached in BitBucket too, for a faster first test run with a clean checkout? Separately, running the tests themselves seems overly slow - can anything be tweaked here? For example, is there any point executing the external set_meta script in the test environment? Regards, Peter P.S. On a related note, my achievement last Friday was to get TravisCI doing continuous integration testing of a GitHub repository of Galaxy tools: https://travis-ci.org/peterjc/galaxy_blast https://github.com/peterjc/galaxy_blast For those not familiar with this system, the idea is that via a special .travis.yml configuration file, each time new commits are pushed to GitHub, the latest code is checked out and tested on a virtual machine. i.e. Continuous in the sense of running the tests after each change, rather than just once a night. Right now the tests for the BLAST+ suite and associated tool wrappers like Blast2GO takes from 15 to 20 minutes, which I feel could be much improved.

2 5

Filtering number of outputs with a conditional
by Alexandre Loywick 31 Oct '13

31 Oct '13

Hello Galaxians, I am currently working on a tool to format reports but I have a problem with the <filter> tag. A little example is easier to figure than a long explanation so... My xml looks like this: <tool name="My tool" id="my tool"> <command interpreter="python">$SCRIPTTYPE.SCRIPT $REPORT $SECONDPARAM $OUTFILE $MAP</command> <inputs> <param type="data" label="My input file" name="REPORT" help="Just for the example"/> <conditional name="SCRIPTTYPE"> <param type="select" label="Script to use" name="SCRIPT"> <option value="script_x.py">X</option> <option value="script_y.py">Y</option> <option value="script_z.py">Z</option> </param> <when value="script_x.py"> <param type="select" name="SECONDPARAM" label="My parameter"> <option value="foo">foo</option> <option value="bar">bar</option> </param> </when> <when value="script_y.py"> <param type="select" name="SECONDPARAM" label="My parameter"> <option value="A">A</option> <option value="B">B</option> </param> </when> <when value="script_z.py"> <param type="select" name="SECONDPARAM" label="My parameter"> <option value="1">1</option> <option value="2">2</option> </param> </when> </conditional> </inputs> <outputs> <data type="data" format="tabular" name="OUTFILE" label="My outfile"> <filter> SCRIPTTYPE['SCRIPT'] == script_x.py</filter> </data> <data type="data" name="OUTFILE" label="My_outfile.tgz"> <filter> SCRIPTTYPE['SCRIPT'] == script_y.py</filter> </data> <data type="data" format="tabular" name="OUTFILE" label="My other outfile"> <filter> SCRIPTTYPE['SCRIPT'] == script_z.py</filter> </data> <data type="data" format="tabular" name="MAPFILE" label="My map"> <filter> SCRIPTTYPE['SCRIPT'] == script_z.py</filter> </data> </outputs> </tool> ===================== I try to make my xml work that way: When I select my script x, there is only one output labeled "my outfile". When I select my script y, there is only one output labeled "My_outfile.tgz" When I select my script z, there are two outputs. I had it working properly with just x or y, but when I add z to my xml, the thing gets all buggy. Instead of having one output when I select x or y, I always have two outputs "my other outfile" and "my map" as if I had selected z. Do you have any idea how I could solve that? Thank you, best regards, Alexandre

2 3

Blastdb and blastdbp datatypes not enforced
by Ganote, Carrie L 30 Oct '13

30 Oct '13

Hi List, It seems that the blastdb and blastdbp data format is not respected on my local install; is it just not generally defined, or am I missing something in /lib/galaxy/datatypes or in datatypes_conf? The most recent blasp wrapper includes this for the input parameter: type="data" format="blastdbp" When I choose the BLAST Database from history option for db, I get every valid dataset in my history in the list. Is this because it's a composite set? Thanks for any tips, Carrie Ganote

2 2

What to do with toolshed tools that get stuck "Installing"
by Ira Cooke 30 Oct '13

30 Oct '13

Hi All, The short version of my problem is that I often seem to find myself in a situation where my tool gets stuck in the "Installing" phase ... and I don't know how to get feedback on it to see where it is failing. What I guess I'm after is just the stdout of the tool installation .. but since the tool install never completes I can never see this. What I usually do is try to recreate the environment that galaxy sees during tool install in a normal shell .. and then run it manually .. which sometimes tells me the problem ... but not always (see below). The longer version of my problem is below. In my case I am struggling with the following tool http://testtoolshed.g2.bx.psu.edu/view/iracooke/package_protk_1_2_5 Which uses the new "setup_ruby_environment" tag ... so the tool should do the following (1) Install the dependency .. Ruby 2.0 ... which it does successfully (2) Install my protk rubygem from rubygems.org .. which it gets stuck doing. The last few lines from my galaxy server log during the failed install are; 10.0.2.2 - - [28/Oct/2013:21:54:26 +0000] "POST /admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 - "http://localhost:8088/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36" [localhost] local: touch /home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/env.sh [localhost] local: echo 'RUBYLIB=/home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/lib/:$RUBYLIB; export RUBYLIB' >> /home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/env.sh 10.0.2.2 - - [28/Oct/2013:21:54:30 +0000] "POST /admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 - "http://localhost:8088/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36" [localhost] local: echo 'PATH=/home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/bin:$PATH; export PATH' >> /home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/env.sh [localhost] local: echo 'GALAXY_RUBY_HOME=/home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/bin; export GALAXY_RUBY_HOME' >> /home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/env.sh [localhost] local: rm -rf ./database/tmp/tmp-toolshed-mtdQaLI7O ruby version 2.0 installed in /home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63 [localhost] local: touch /home/vagrant/tool_dependencies/ruby/2.0/iracooke/package_protk_1_2_5/876e44dd4609/env.sh [localhost] local: echo 'RUBYLIB=/home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/lib/:$RUBYLIB; export RUBYLIB' >> /home/vagrant/tool_dependencies/ruby/2.0/iracooke/package_protk_1_2_5/876e44dd4609/env.sh [localhost] local: echo 'PATH=/home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/bin:$PATH; export PATH' >> /home/vagrant/tool_dependencies/ruby/2.0/iracooke/package_protk_1_2_5/876e44dd4609/env.sh [localhost] local: echo 'GALAXY_RUBY_HOME=/home/vagrant/tool_dependencies/ruby/2.0/bgruening/package_ruby_2_0/a0494c6e1c63/bin; export GALAXY_RUBY_HOME' >> /home/vagrant/tool_dependencies/ruby/2.0/iracooke/package_protk_1_2_5/876e44dd4609/env.sh [localhost] local: rm -rf ./database/tmp/tmp-toolshed-mtdJYyhO0 10.0.2.2 - - [28/Oct/2013:21:54:34 +0000] "POST /admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 - "http://localhost:8088/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36" 10.0.2.2 - - [28/Oct/2013:21:47:22 +0000] "POST /admin_toolshed/manage_repositories HTTP/1.1" 302 - "http://localhost:8088/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36" 10.0.2.2 - - [28/Oct/2013:21:54:37 +0000] "GET /admin_toolshed/monitor_repository_installation?tool_shed_repository_ids=f597429621d6eb2b&tool_shed_repository_ids=f2db41e1fa331b3e HTTP/1.1" 200 - "http://localhost:8088/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36" With a normal tool install I would see the last line of this output repeat lots of times as installation is monitored ... but for some reason in this case monitoring just shuts down and I don't see any further updates. If I recreate the situation on the command-line (eg by manually setting the RUBYLIB and GEM_HOME environment variables appropriately I can install my gem without any errors. More details on my setup: I'm using a vagrant VM for testing which is based on Ubuntu precise64 ... with a minimal number of packages installed ... (My VM is documented in full here: https://bitbucket.org/iracooke/protk_vagrant/src/0ef8d86cc3dcec4ee3a9a83edf… ) I'm testing based on the current default branch of galaxy-central .... which I regularly tear down and re-clone from scratch ... always with the same result. Any help or pointers to how to resolve this much appreciated. Cheers Ira

4 10

Workflows with <param type="data" multiple="true" ...>
by Peter Cock 30 Oct '13

30 Oct '13

Hi John, I've just tried a workflow using my MIRA4 de novo wrapper: http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira4_assembler To keep it simple, in this example I just had two input FASTQ files, the forward and reverse files for paired end MiSeq data. This seemed to work fine for re-running the workflow :) However, on loading the workflow to edit it, only the first input file was drawn linked to the MIRA4 tool. On making some minor workflow changes (renaming some output), and trying to run it, the second input file seemed not to be used. Is this something you are aware of? The good news is that I could "re-draw" the missing connection from the second FASTQ file in the workflow editor, and that appears to be working... Peter -- This was using the current stable release, $ hg branch default $ hg heads changeset: 10421:a477486bf18e branch: stable tag: tip user: Nate Coraor <nate(a)bx.psu.edu> date: Thu Sep 26 11:02:58 2013 -0400 summary: Bugfix for tool-to-destination mapping, tool ids are lowercased but the mapping id was not lowercased. changeset: 10411:c42567f43aa7 user: greg date: Mon Aug 19 13:19:56 2013 -0400 summary: Filter invalid objects when generating the list of repository_dependencies objects that are associated with a tool shed repository installed into Galaxy.

2 2

Tool refuses to appear in Toolshed package contents
by Lukasse, Pieter 30 Oct '13

30 Oct '13

Hi , I have a toolshed repository where I expect 5 tools to appear in the "Contents of this repository" section but one of them fails to appear in the list (see screenshot). Where can I find out what is going wrong? Thanks, Pieter. [cid:image001.png@01CED0BA.3733D000] Pieter Lukasse Wageningen UR, Plant Research International Departments of Bioscience and Bioinformatics Wageningen Campus, Building 107, Droevendaalsesteeg 1, 6708 PB, Wageningen, the Netherlands +31-317480891; skype: pieter.lukasse.wur http://www.pri.wur.nl<http://www.pri.wur.nl/>

4 16