September 2014 - galaxy-dev - lists.galaxyproject.org

New version of vcflib wrappers
by Anton Nekrutenko 15 Sep '14

15 Sep '14

The new version is based on vcflib 386afcaafd. GitHub -> https://github.com/nekrut/vcflib-tools ToolShed -> https://toolshed.g2.bx.psu.edu/view/anton/suite_vcflib_tools_3_0 These will be on test and main instances shortly. Any feedback is appreciated. anton -- Anton Nekrutenko Professor of Biochemistry and Molecular Biology http://nekrut.bx.psu.edu (814) 826-3051

1 0

This looks like a Galaxy bug in installation of tools with a complex repository dependency.
by Melissa Cline 15 Sep '14

15 Sep '14

Hi Folks, I've done a bit more work on this problem that I posted about on Friday. I'm working on streamlining the installation of two tools, one of which has a complex repository dependency on the other. FWIW, both tools are in the test tool shed, under visualization: xena_import has a complex repository dependency on start_xena. I know that it should not be necessary to specify the toolshed and changeset revision in tool_dependencies.xml when indicating the complex repository dependency. But here is what I observed. When I specified the changeset_revision and toolshed, the installation worked fine. When I removed the changeset_revision tag and reinstalled the tool, I experienced the bug detailed below, where a Google Chrome pop-up window told me that the repository installation failed, and after I clicked OK on the pop-up window, Galaxy seemed to go into an infinite state of indicating that the tool was "Cloning" and showing the same repeated POST commands in paster.log. When I put the changeset_revision tag back again, the tool installation worked just fine. Just to make sure that this wasn't all from some cruft hiding somewhere in my Galaxy installation, I started with a clean install today (including an 'hg update stable') and a clean shed_tools subdirectory. I hope this helps. In the meantime, thanks, everyone for all of your help! Melissa On Fri, Sep 12, 2014 at 12:02 PM, Melissa Cline <cline(a)soe.ucsc.edu> wrote: > Thanks, folks! Everything you've said has clarified many things, and I'm > working on putting them into practice. But in the meantime, I've got a new > error condition, with the same repositories (and complex repository > dependencies), and could use some guidance. > > I started to see this error condition when I removed the toolshed and > changeset_revision tags in the tool_dependencies.xml for the dependent > repository xena_install. It now reads: > > <?xml version="1.0"?> > > <tool_dependency> > > <package name="installXena" version="1.0"> > > <repository name="start_xena" owner="melissacline"/> > > </package> > > </tool_dependency> > > Short and simple. But lately, when I try to install this repository from > the test toolshed, I get a pop-up window from Chrome (my web browser) that > says "The page at 127.0.0.1:8080 says:", "Initializing repository > installation failed", with an OK button. There is no error message visible > in paster.log. Here is what I do see: > > 127.0.0.1 - - [12/Sep/2014:11:45:57 -0700] "GET > /admin_toolshed/prepare_for_ins\ > > tall?tool_shed_url= > https://testtoolshed.g2.bx.psu.edu/&repository_ids=4ab1d3705\ > > ad0de2d&changeset_revisions=fe95cf1ae864 HTTP/1.1" 200 - "-" "Mozilla/5.0 > (Maci\ > > ntosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) > Chrome/37.\ > > 0.2062.94 Safari/537.36" > > tool_shed.galaxy_install.repository_dependencies.repository_dependency_manager > \ > > DEBUG 2014-09-12 11:46:05,937 Creating repository dependency objects... > > tool_shed.util.shed_util_common DEBUG 2014-09-12 11:46:07,214 Updating an > exist\ > > ing row for repository 'xena_import' in the tool_shed_repository table, > status \ > > set to 'New'. > > tool_shed.galaxy_install.repository_dependencies.repository_dependency_manager > \ > > DEBUG 2014-09-12 11:46:07,241 Skipping installation of revision > 75c7d80df9c1 of\ > > repository 'start_xena' because it was installed with the (possibly > updated) r\ > > evision 75c7d80df9c1 and its current installation status is 'Installed'. > > tool_shed.galaxy_install.repository_dependencies.repository_dependency_manager > \ > > DEBUG 2014-09-12 11:46:07,241 Building repository dependency > relationships... > > 127.0.0.1 - - [12/Sep/2014:11:46:05 -0700] "POST > /admin_toolshed/prepare_for_in\ > > stall HTTP/1.1" 200 - " > http://127.0.0.1:8080/admin_toolshed/prepare_for_install\ > > ?tool_shed_url= > https://testtoolshed.g2.bx.psu.edu/&repository_ids=4ab1d3705ad0d\ > > e2d&changeset_revisions=fe95cf1ae864" "Mozilla/5.0 (Macintosh; Intel Mac > OS X 1\ > > 0_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.94 > Safari/537.36\ > > " > > warning: testtoolshed.g2.bx.psu.edu certificate with fingerprint > 79:94:82:1f:74\ > > :e6:f8:5c:1c:b8:3e:8d:89:f0:0e:88:9a:ac:2b:61 not verified (check > hostfingerpri\ > > nts or web.cacerts config setting) > > 127.0.0.1 - - [12/Sep/2014:11:46:07 -0700] "POST > /admin_toolshed/manage_reposit\ > > ories HTTP/1.1" 500 - " > http://127.0.0.1:8080/admin_toolshed/prepare_for_install\ > > " "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 > (KHTML, li\ > > ke Gecko) Chrome/37.0.2062.94 Safari/537.36" > > Debug at: http://127.0.0.1:8080/_debug/view/1410547505 > > > Then when I click OK, the status of the tool changes to "Cloning", and > stays in that state indefinitely while paster.log keeps logging a repeat of > this message: > > 127.0.0.1 - - [12/Sep/2014:11:55:28 -0700] "POST > /admin_toolshed/repository_ins\ > > tallation_status_updates HTTP/1.1" 200 - " > http://127.0.0.1:8080/admin_toolshed/\ > > prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) > AppleWebKi\ > > t/537.36 (KHTML, like Gecko) Chrome/37.0.2062.94 Safari/537.36" > > Just in case this is from some cruft from a previous installation, I tried > starting over with a brand new galaxy-dist directory. It doesn't look like > that changed anything. > > > Any ideas? Many thanks in advance! > > > Melissa > > > > > > On Fri, Sep 12, 2014 at 10:30 AM, Dave Bouvier <dave(a)bx.psu.edu> wrote: > >> Melissa, >> >> The revision recorded in the filesystem path can seem odd at first >> glance. My inline responses should hopefully clarify that. >> >> --Dave B. >> >> On 09/10/2014 05:49 PM, Melissa Cline wrote: >> >>> Hi Folks, >>> >>> Okay, I really need debugging ideas on this one. >>> >>> I have three repositories I'm developing under the test toolshed. >>> They're named start_xena, xena_import and xena_find_datasets (they're >>> all under visualization). start_xena contains a simple tool dependency >>> to a package named installXena. xena_import and xena_find_datasets >>> contain complex dependencies to installXena. I've installed these tools >>> on two computers. On one, everything works great - amazingly well, in >>> fact! Kudos, Development Team! On the other computer, the env.sh files >>> associated with the complex dependencies keep referencing the wrong >>> version of start_xena/installXena. >>> >>> Here are the gory details. >>> >>> The start_xena tool is installed in >>> testtoolshed.g2.bx.psu.edu/repos/melissacline/start_xena/ >>> 82755b0ee5a5/start_xena >>> <http://testtoolshed.g2.bx.psu.edu/repos/melissacline/ >>> start_xena/82755b0ee5a5/start_xena>. >>> I'm a bit confused about the revision part of the path, because the >>> latest revision is different (75c7d80df9c1), and I have uninstalled and >>> re-installed the tool since its last update. Its package, installXena, >>> is installed under tool_dependencies at >>> tool_dependencies/installXena/1.0/melissacline/start_xena/82755b0ee5a5/. >>> That directory contains all the files I'd expect to be there, and its >>> env.sh file contains all the environment variables I'd intended to set. >>> So far, so good. >>> >>> xena_import is installed on my computer at >>> testtoolshed.g2.bx.psu.edu/repos/melissacline/xena_ >>> import/dc42b6bbc22b/xena_import >>> <http://testtoolshed.g2.bx.psu.edu/repos/melissacline/ >>> xena_import/dc42b6bbc22b/xena_import>. >>> Its tool_dependencies.xml reference installXena under start_xena as: >>> >>> <?xml version="1.0"?> >>> >>> <tool_dependency> >>> >>> <package name="installXena" version="1.0"> >>> >>> <repository toolshed="http://testtoolshed.g2.bx.psu.edu" >>> name="start_xena" owner="melissacline" changeset_revision=" >>> 82755b0ee5a5"/> >>> >>> </package> >>> >>> </tool_dependency> >>> >>> (Aside: is it necessary to have the changeset_revision in there? I'm a >>> bit worried about the maintenance task of updating it for all of my >>> dependent tools every time I update start_xena, but as I'll explain, >>> it's not clear that this revision info is getting used). >>> >> >> It is not necessary to populate the changeset_revision or toolshed >> attributes. If you omit them, they will be automatically filled in based on >> the most recent matching repository. >> >> >>> The tool dependency dir for xena_import is at >>> tool_dependencies/installXena/1.0/melissacline/xena_import/dc42b6bbc22b/. >>> Its >>> env.sh reads: >>> >>> if [ -f >>> /Users/melissacline/src/galaxy-dist/tool_dependencies/ >>> installXena/1.0/melissacline/start_xena/0676a227dbc6/env.sh >>> ] ; then . >>> /Users/melissacline/src/galaxy-dist/tool_dependencies/ >>> installXena/1.0/melissacline/start_xena/0676a227dbc6/env.sh >>> ; fi >>> >>> Notice that the wrong revision number is in there. The 06 revision was >>> an older one. Needless to say, the files and environment variables I >>> need aren't in the 06 directory. >>> >>> Where does this incorrect revision number come from? I've deleted all >>> the 06 directories, repeatedly. I've deleted and reinstalled the tools, >>> and verified that after deletion, the directories in question were in >>> fact gone. I've tried resetting the metadata for these tools under the >>> Galaxy Admin menu. Now I need new ideas. As I mentioned, all of this >>> works beautifully on a second computer. It's as if the first Galaxy >>> installation, on the first computer, got stuck with some outdated >>> information, and I haven't figured out how to clear it out. >>> >> >> The 0676 revision number is based on the revision that was first >> installed, and should never change. When you update the repository, the >> database will record the new revision in a separate field, but the original >> revision id needs to be recorded in order to find the env.sh file on the >> filesystem. >> >> >> >>> Thanks! >>> >>> Melissa >>> >>> >>> >>> >>> ___________________________________________________________ >>> Please keep all replies on the list by using "reply all" >>> in your mail client. To manage your subscriptions to this >>> and other Galaxy lists, please use the interface at: >>> http://lists.bx.psu.edu/ >>> >>> To search Galaxy mailing lists use the unified search at: >>> http://galaxyproject.org/search/mailinglists/ >>> >>> >

1 0

Queueing system and job parameters
by Velayutham, Prakash (Prakash) 15 Sep '14

15 Sep '14

Hi, I have a local Galaxy instance. I have tied the server to a local HPC and we use LSF for scheduling jobs. I was wondering how a user can submit a job from Galaxy while specifying a wall time and memory requirements for a job… Is this doable? Thanks, Prakash

2 3

Workflow step IDs: out of order?
by Freek de Bruijn 15 Sep '14

15 Sep '14

Hi everyone, The Galaxy API is a great way to interact with a Galaxy server and I've been using it to run workflows (supported by the blend4j library, since I'm working on a Java project). Recently, I ran into a challenge with the workflow step IDs. When retrieving the metadata of a Galaxy workflow (using the API and blend4j), the order of the workflow step IDs appears to be random (or at least, not what I expected). This makes it complicated to specify parameters when you run a workflow via the API, because the step ID and parameter name normally are a nice way to pick the parameter you want to set (blend4j method: WorkflowInputs.setStepParameter). Is is possible to determine which step ID is assigned to - for example - the second workflow step? >From a few tests and looking at the Galaxy source code, it looks to me like the workflow steps are sorted according to their position in the workflow editor (in ascending order of sqrt(sqr(position.x) + sqr(position.y))). When storing the workflow in the database, the ID field for each step is assigned a unique unpredictable number (by SQLAlchemy or the database?). But in addition to the ID, there is also a nice order_index field, which contains the indices in the expected order! Would it be possible to add the order_index field to the API and would that solve the step order issue? Cheers, Freek ------ Freek de Bruijn Scientific programmer | VU University medical center

2 1

custom datatypes
by Calvin Morrison 13 Sep '14

13 Sep '14

Hi, I just want a simple data type for my 'custom' datatype (it's just a trained matrix in a specific format generated by my database building tool), so that my tools which use this database will only see ones with that format in the dropdown select list. in my datatypes.xml i have this: <datatype extension="gz" type="galaxy.datatypes.quikrdb" mimetype="application/octet-stream" display_in_upload="true" description="A database trained with quikr." /> then in my tool xml files i have <outputs> <data name="output" format="quikrdb" /> </outputs> which seems to be fine, but my tools which use it still show me all my data, not just trained db's <param name="dbname" type="data" format="quikrdb" label="custom trained database"/> Am i doing something wrong? Calvin Morrison

2 1

Incorrect revision getting referenced in a complex tool dependency - why?
by Melissa Cline 12 Sep '14

12 Sep '14

Hi Folks, Okay, I really need debugging ideas on this one. I have three repositories I'm developing under the test toolshed. They're named start_xena, xena_import and xena_find_datasets (they're all under visualization). start_xena contains a simple tool dependency to a package named installXena. xena_import and xena_find_datasets contain complex dependencies to installXena. I've installed these tools on two computers. On one, everything works great - amazingly well, in fact! Kudos, Development Team! On the other computer, the env.sh files associated with the complex dependencies keep referencing the wrong version of start_xena/installXena. Here are the gory details. The start_xena tool is installed in testtoolshed.g2.bx.psu.edu/repos/melissacline/start_xena/82755b0ee5a5/start…. I'm a bit confused about the revision part of the path, because the latest revision is different (75c7d80df9c1), and I have uninstalled and re-installed the tool since its last update. Its package, installXena, is installed under tool_dependencies at tool_dependencies/installXena/1.0/melissacline/start_xena/82755b0ee5a5/. That directory contains all the files I'd expect to be there, and its env.sh file contains all the environment variables I'd intended to set. So far, so good. xena_import is installed on my computer at testtoolshed.g2.bx.psu.edu/repos/melissacline/xena_import/dc42b6bbc22b/xena…. Its tool_dependencies.xml reference installXena under start_xena as: <?xml version="1.0"?> <tool_dependency> <package name="installXena" version="1.0"> <repository toolshed="http://testtoolshed.g2.bx.psu.edu" name="start_xena" owner="melissacline" changeset_revision="82755b0ee5a5"/> </package> </tool_dependency> (Aside: is it necessary to have the changeset_revision in there? I'm a bit worried about the maintenance task of updating it for all of my dependent tools every time I update start_xena, but as I'll explain, it's not clear that this revision info is getting used). The tool dependency dir for xena_import is at tool_dependencies/installXena/1.0/melissacline/xena_import/dc42b6bbc22b/. Its env.sh reads: if [ -f /Users/melissacline/src/galaxy-dist/tool_dependencies/installXena/1.0/melissacline/start_xena/0676a227dbc6/env.sh ] ; then . /Users/melissacline/src/galaxy-dist/tool_dependencies/installXena/1.0/melissacline/start_xena/0676a227dbc6/env.sh ; fi Notice that the wrong revision number is in there. The 06 revision was an older one. Needless to say, the files and environment variables I need aren't in the 06 directory. Where does this incorrect revision number come from? I've deleted all the 06 directories, repeatedly. I've deleted and reinstalled the tools, and verified that after deletion, the directories in question were in fact gone. I've tried resetting the metadata for these tools under the Galaxy Admin menu. Now I need new ideas. As I mentioned, all of this works beautifully on a second computer. It's as if the first Galaxy installation, on the first computer, got stuck with some outdated information, and I haven't figured out how to clear it out. Thanks! Melissa

3 3

Tool Errors
by Calvin Morrison 12 Sep '14

12 Sep '14

Hi guys, I am trying to write a tool with a conditional and for whatever reason the results, well aren't any results. It just says 'no seek', which totally bewildering. Does anyone know what this means? Thank you, Calvin my xml file: <tool id="quikr" name="Quikr"> <description>Classify with Quikr</description> <command> # if $qdb.dbtype == "user" quikr -v -k 0 -s ${qdb.dbname} -i ${input} -o ${output} # else quikr -v -k 0 -s "${qdb.dbname}${qdb.dbsize}".mat.gz -i ${input} -o ${output} # end if </command> <inputs> <param name="input" type="data" format="fasta" label="input fasta file"/> <conditional name="qdb"> <param name="dbtype" type="select" label="Genomes to score against"> <option value="default" selected="true">Pretrained databases</option> <option value="user">Custom trained quikr database</option> </param> <when value="user"> <param name="dbname" type="data" format="quikrdb" label="custom trained database"/> <param name="dbsize" type="hidden" label="mer size"> <option value="5">5</option> </param> </when> <when value="default"> <param name="dbname" type="select" label="Choose database"> <option value="/data/galaxy/data/gg91">Green Genes 91%</option> <option value="/data/galaxy/data/gg94">Green Genes 94%</option> <option value="/data/galaxy/data/rdp">RDP Trained Set</option> </param> <param name="dbsize" type="select" label="mer size"> <option value="5">5</option> <option value="6">6</option> <option value="7">7</option> <option value="8">8</option> </param> </when> </conditional> <param name="lambda" type="integer" size="6" value="10000" label="lambda" help="default 10000"/> </inputs> <outputs> <data name="output" format="tabular" /> </outputs> <help> </help> </tool>

4 7

Amazon cloud setup for galaxy training
by Anne Pajon 12 Sep '14

12 Sep '14

Dear, I am setting up with my colleague Jing an Introductory Galaxy training course at the University of Cambridge due to on the 24th September. I am looking for help on how to setup the cloud to have a smooth user experience during the workshop. Would you have any links with information to recommend or any advices on how to set it up? Many thanks in advance. Kindest regards, Anne. -- Dr Anne Pajon - Bioinformatics Core Cancer Research UK - Cambridge Institute Li Ka Shing Centre, Robinson Way, Cambridge CB2 0RE anne.pajon(a)cruk.cam.ac.uk | +44 (0)1223 769 631

2 1

Installing Galaxy on Freebsd 10.0
by ashish damania 11 Sep '14

11 Sep '14

# sh run.shSome eggs are out of date, attempting to fetch...Warning: setuptools (a dependent egg of sqlalchemy-migrate) cannot be fetchedWarning: simplejson (a dependent egg of bioblend) cannot be fetchedsimplejson 2.1.1 couldn't be downloaded automatically. You can trybuilding it by hand with: python scripts/scramble.py -e simplejsonFetch failed.Traceback (most recent call last): File "/usr/home/ec2-user/galaxy-dist/lib/galaxy/webapps/galaxy/buildapp.py", line 38, in app_factory from galaxy.app import UniverseApplication File "/usr/home/ec2-user/galaxy-dist/lib/galaxy/app.py", line 12, in <module> from galaxy.visualization.data_providers.registry import DataProviderRegistry File "/usr/home/ec2-user/galaxy-dist/lib/galaxy/visualization/data_providers/registry.py", line 2, in <module> from galaxy.visualization.data_providers import genome File "/usr/home/ec2-user/galaxy-dist/lib/galaxy/visualization/data_providers/genome.py", line 17, in <module> from pysam import !csamtools, ctabix File "/usr/home/ec2-user/galaxy-dist/eggs/pysam-0.4.2_kanwei_b10f6e722e9a-py2.7-freebsd-10.0-RELEASE-p7-amd64-ucs2.egg/pysam/__init__.py", line 1, in <module> from csamtools import *ImportError: /usr/home/ec2-user/galaxy-dist/eggs/pysam-0.4.2_kanwei_b10f6e722e9a-py2.7-freebsd-10.0-RELEASE-p7-amd64-ucs2.egg/csamtools.so: Undefined symbol "__ks_insertsort_heap"So I have tried running this scriptpython scripts/scramble.py -e simplejsonTraceback (most recent call last): File "scripts/scramble.py", line 50, in <module> egg.scramble() File "/usr/home/ec2-user/galaxy-dist/lib/galaxy/eggs/scramble.py", line 60, in scramble raise ScrambleFailure( self, "%s(): Egg build for %s did not appear to fail, but no egg found to copy from expected path:\n %s" % ( sys._getframe().f_code.co_name, self.name, new_egg ) )galaxy.eggs.scramble.ScrambleFailure: scramble(): Egg build for simplejson did not appear to fail, but no egg found to copy from expected path: /usr/home/ec2-user/galaxy-dist/scripts/scramble/build/py2.7/simplejson/dist/simplejson-2.1.1-py2.7.eggI found that I have file named: "simplejson-2.1.1-py2.7-freebsd-10.0-RELEASE-p7-amd64-ucs2.egg" in the "../scripts/scramble/build/py2.7/simplejson/dist/" folder. I renamed that file simplejson-2.1.1-py2.7.egg but it gives me that same error when I execute "sh run.sh".Is it ok to run Galaxy under Freebsd 10.0? Are there any compatibility issues?Thanks you.

2 3

no capsule export from a local tool shed
by Christophe Antoniewski 11 Sep '14

11 Sep '14

Hi, Here is a question maybe to Greg: Since the GCC2014 we started a local toolshed following the instructions in http://gregvonkuster.org/galaxy-tool-shed-framework-building-galaxy-tools/ It works nicely, except that today I tried to export repository capsules ("export this revision") from the local toolshed and it returns the following error (see below). I cannot remember whether I had ever tested this particular action since the local tool shed set up. Thus I even cannot say whether the bug is due to an original misconfiguration or it happened later on. I can add that the local toolshed is served by apache with rewriting rules (I am saying this because in addition it is not possible to download repositories as gz archives etc... and suspect that there is maybe a problem, in addition, with the apache config). Any help appreciated ! Regards Christophe --------------- URL: http://lbcd41.snv.jussieu.fr/toolshed/repository/export?repository_id=8b0be… Module galaxy.web.framework.middleware.error:*149* in __call__ >> <http://lbcd41.snv.jussieu.fr/toolshed/repository/export?repository_id=8b0be…> app_iter *=* self*.*application*(*environ*,* sr_checker*)* Module paste.debug.prints:*106* in __call__ >> <http://lbcd41.snv.jussieu.fr/toolshed/repository/export?repository_id=8b0be…> environ*,* self*.*app*)* Module paste.wsgilib:*543* in intercept_output >> <http://lbcd41.snv.jussieu.fr/toolshed/repository/export?repository_id=8b0be…> app_iter *=* application*(*environ*,* replacement_start_response*)* Module paste.recursive:*84* in __call__ >> <http://lbcd41.snv.jussieu.fr/toolshed/repository/export?repository_id=8b0be…> *return* self*.*application*(*environ*,* start_response*)* Module paste.httpexceptions:*633* in __call__ >> <http://lbcd41.snv.jussieu.fr/toolshed/repository/export?repository_id=8b0be…> *return* self*.*application*(*environ*,* start_response*)* Module galaxy.web.framework.base:*132* in __call__ >> <http://lbcd41.snv.jussieu.fr/toolshed/repository/export?repository_id=8b0be…> *return* self*.*handle_request*(* environ*,* start_response *)* Module galaxy.web.framework.base:*190* in handle_request >> <http://lbcd41.snv.jussieu.fr/toolshed/repository/export?repository_id=8b0be…> body *=* method*(* trans*,* ****kwargs *)* Module galaxy.webapps.tool_shed.controllers.repository:*1249* in export >> <http://lbcd41.snv.jussieu.fr/toolshed/repository/export?repository_id=8b0be…> repositories_archive_filename *=* os*.*path*.*basename*(* repositories_archive*.*name *)* *AttributeError: 'NoneType' object has no attribute 'name'* Christophe Antoniewski Drosophila Genetics and Epigenetics Laboratoire de Biologie du Développement 9, Quai St Bernard, Boîte courrier 24 75252 Paris Cedex 05 christophe.antoniewski(a)upmc.fr http://bio-dev.snv.jussieu.fr/ Tel +33 1 44 27 34 39 Fax +33 1 44 27 34 45 Mobile +33 6 68 60 51 50 http://drosophile.org http://www.sciencesenmarche.org/

2 3