March 2014 - galaxy-dev - lists.galaxyproject.org

Adding a tool with additional html/js interface to make input selection
by Saba Sehrish 20 Mar '14

20 Mar '14

Hi, We are trying to add a new tool to our local galaxy instance with the following requirements. When a user clicks on the tool in side panel from tools list, a table display should appear where the first element in each row is a clickable link. Users can click on any of the element, and then see only there selected row and another table with more options. Once user has made all the selections, then a form should appear describing that you have made the following selection, do you want to execute the tool now? And then users can submit the job. If we implement such a thing in galaxy, we need to load a page with aforementioned functionality, then map the users selection to the tool's inputs, and then run the tool. The question is, is there a way in galaxy to load an html with javascript and use it to prepare the inputs for the tool we want to run, using the xml wrapper. Thanks, Saba

5 14

Re: [galaxy-dev] dynamic select problem
by Saba Sehrish 19 Mar '14

19 Mar '14

I see the similar problem when I try to use first param_value as a filter on the second input selection and then second on the third input selection. The code works fine and makes right selections but displays the warning message “An invalid option was selected, please verify”. My select code is similar: <param name="inputmodel" type="select" size="100" label="Input Model" area="True"> <options from_file="coyote.loc"> <column name="name" index="2"/> <column name="value" index="2"/> <filter type="unique_value" column="2"/> </options> </param> <param name="inputSize" type="select" label="Box Size" selected="false"> <options from_file="coyote.loc"> <column name="name" index="3"/> <column name="value" index="3"/> <filter type="param_value" ref="inputmodel" column="2"/> <filter type="unique_value" column="3"/> </options> </param> <param name="inputRealization" type="select" label="Realization based on different seed values" selected="false"> <options from_file="coyote.loc"> <column name="name" index="4"/> <column name="value" index="4"/> <filter type="param_value" ref="inputmodel" column="2"/> <filter type="param_value" ref="inputSize" column="3"/> <filter type="unique_value" column="4"/> </options> </param> <param name="input" type="select" size="300" label="Input Snapshot" area="True" help="Choosing Input Snapshot files" selected="false"> <options from_file="coyote.loc"> <column name="name" index="6"/> <column name="value" index="6"/> <filter type="param_value" ref="inputmodel" column="2”/> <filter type="param_value" ref="inputSize" column=“3"/> <filter type="param_value" ref="inputRealization" column="4"/> </options> </param> Did you figure out where the problem is? Thanks, Saba

2 3

Tool Testing Enhancements
by John Chilton 19 Mar '14

19 Mar '14

Hello Tool Developers, Haven't known when to send this out, but I figure since you haven't received any e-mail from me today it might be a good time. tl;dr - Tool functional tests experienced a significant overhaul over the last release and will continue to change over the next couple releases, but in a backward compatible so hopefully you will not need to care unless you want to. The last release included a considerable overhaul to the framework used to functionally test tools (i.e. actually running those <test> tags in tool XML). This includes changes to how the tests are executed and an expansion in the expressiveness of these tests. As some background for why the changes are being made, currently tools are tested by starting up a test Galaxy instance and using a library called Twill to exercise the GUI and upload data and run tools through by exercising actual HTML pages. The last release added the optional ability to exercise these tool tests via Galaxy's tool API instead of via web forms, with the slightly longer term goal of eliminating the Twill testing option altogether. This will allow the Galaxy tool form API to evolve more quickly, eliminate much of the testing complexity and many of the limitations imposed by Twill while ensuring that Galaxy's tool running API is sufficiently expressive. I have put a significant amount of effort into backward compatibility - so all current tests SHOULD pass using the new method. I know some people have included significant hacks with their test definitions to sculpt them deal with Twill-isms. These may fail - I would like to catalog what these are and see if we can hack up the test framework to deal with them or transition these tests to use cleaner mechanisms allowed for by the more direct representation of tool parameters consumed by the API. If you are running tests locally with the run_functional_tests.sh script distributed with Galaxy, simply setting the environment variable GALAXY_TEST_DEFAULT_INTERACTOR to 'api' before running the script and all tests will be run through the API. Individual tests can be set to target the API (for instance if they use newer, API-only features outlined below) by adding the XML attribute interactor="api" to the test tag in the tool XML. This change works right now if you want to update test tool shed tools to target the new testing method and provide feedback. As part of this transition, many bugs were fixed that affect both styles of test and new features were added. The biggest improvement is the ability to specify parameters in a nested structure matching the inputs themselves. This allows disambiguation of repeat and conditional parameters for both styles of functional tests and in the case of API driven tests allows specification of any number of repeat block instances in tests (one still can only specify at most one instance for any repeat tag while using Twill driven tests). The following annotated tool XML examples concretely demonstrate this new expressiveness. https://bitbucket.org/galaxy/galaxy-central/src/tip/test/functional/tools/d… https://bitbucket.org/galaxy/galaxy-central/src/tip/test/functional/tools/d… Additional Enhancements: Enhancement (Twill and API): Allow unspecified conditional test parameters. Previously specifying a parameter in a conditional but not the value of the conditional test parameter itself (the select or boolean parameter the conditional changes based on) resulted in cryptic test errors. Now the default is assumed if unspecified. https://bitbucket.org/galaxy/galaxy-central/commits/8b99c7bc5e11cd7c452c2ad… https://bitbucket.org/galaxy/galaxy-central/commits/76bd49e658876f5cc90ec5b… Enhancement (Twill and API): Allow specifying tests for composite extra files without checking primary file. https://bitbucket.org/galaxy/galaxy-central/commits/45c1d11cfff552ce095c056… Enhancement (API-only): Allow testing metadata properties on test output datasets. https://bitbucket.org/galaxy/galaxy-central/commits/01f05ab8df7092b6e03f35f… Enhancement (API-only): Remove restriction of output order specification. https://bitbucket.org/galaxy/galaxy-central/commits/d05be33ad6b772c2a688d87… Enhancement: Allow testing repeat blocks with min="1" (twill) or any non-zero value (API) https://bitbucket.org/galaxy/galaxy-central/commits/4615f7424dcb4d2b61be5f3… Enhancement (API-only): Allow testing of data parameters with multiple="True". https://bitbucket.org/galaxy/galaxy-central/commits/211f30207d48101a1250983… Fuller break down of this is development is available on Trello https://trello.com/c/MsJctFem. Once Galaxy and the tool shed have been modified to target the API when testing tools by default the tool XML syntax wiki page will be updated with examples of this new functionality. -John

2 3

REFERENCE genome in Bowtie2
by Emir Islamovic 18 Mar '14

18 Mar '14

Dear Galaxy Representative, I'm trying to use Bowtie2 in Galaxy and I need to select reference genome. It says "If your genome of interest is not listed, contact the Galaxy team " and that is why I am contacting you. My genome of interest is Maize (corn, Zea mays) and information about it can be found at http://www.maizegdb.org/ Thank you. Emir -- Emir Islamovic Postdoctoral Research Assistant Plant and Microbial Biology Department 311 Koshland Hall University of California, Berkeley Berkeley, CA 94720-3102 Phone: 510-642-8058 Fax: 510-642-4995 Email: emirislamovic(a)berkeley.edu emirislamovic(a)yahoo.com

1 0

Tool Shed install best practise : Precompiled binaries vs local compile
by Peter Cock 18 Mar '14

18 Mar '14

Hi Greg, I've retitled the thread, previously about a ToolShed nightly test failure. A brief recap, we're talking about the Galaxy ToolShed XML installation recipes for the NCBI BLAST+ packages and my MIRA4 wrapper in their tool_dependencies.xml files: http://toolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_29 http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_29 http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira4_assembler These use the pattern of having os/arch specific <action> tags (which download and install the tool author's precompiled binaries) and a fall back default <action> which is to report an error with the os/arch combination and that there are no ready made binaries available. Greg is instead advocating the fall back action be to download the source code, and do a local compile. My reply is below... On Thu, Mar 6, 2014 at 5:24 PM, Peter Cock <p.j.a.cock(a)googlemail.com> wrote: > On Thu, Mar 6, 2014 at 4:53 PM, Greg Von Kuster <greg(a)bx.psu.edu> wrote: >> >> As we briefly discussed earlier, your mira4 recipe is not currently >> following best practices. Although you uncovered a problem in >> the framework which has now been corrected, your recipe's fall >> back <actions> tag set should be the recipe for installing mira4 >> from source ( http://sourceforge.net/projects/mira-assembler/ ) >> since there is no licensing issues for doing so. This would be a >> more ideal approach than echoing the error messages. >> >> Thanks very much for helping us discover this problem though! >> >> Greg Von Kuster > > Hi Greg, > > No problem - I'm good at discovering problems ;) > > If the download approach failed, it it most likely due to a > transient error (e.g. network issues with download). Here I > would much prefer Galaxy aborted and reported this as an > error (and does not attempt the default action). Is that what > you just fixed? > > As to best practice for the fall back action, I think that needs > a new thread. > > Regards, > > Peter As to best practice, I do not agree that in cases like this (MIRA4, NCBI BLAST+) where there are provided binaries for the major platforms that the fall back should be compiling from source. The NCBI BLAST+ provide binaries for 32 bit and 64 bit Linux and Mac OS X (which I believe covers all the mainstream platforms Galaxy runs on). Similarly, MIRA4 provides binaries for 64 bit Linux and Mac OS X. Note that 32 bit binaries are not provided, but would be very restricted in terms of the datasets they could be used on anyway - and I doubt many of the systems Galaxy runs on these days are 32 bits. If the os/arch combination is exotic enough that precompiled binaries are not available, then it is likely compilation will be tricky anyway - or not supported for that tool, or Galaxy itself. Essentially I am arguing that where the precompiled tool binaries cover any mainstream system Galaxy might be used on, a local compile fall back is not needed. Also, these are both complex tools which are relatively slow to compile, and have quite a large compile time dependency set (e.g. MIRA4 requires at least a quite recent GCC, BOOST, flex, expat, and strongly recommends TCmalloc). Here at least some of the dependencies have been packaged for the ToolShed (probably by Bjoern?) but in the case of MIRA4 and BLAST+ this is still a lot of effort for no practical gain. I also feel there is an argument that the Galaxy goal of reproducibility should favour using precompiled binaries if available: A locally compiled binary will generally mean a different compiler version, perhaps with different optimisation flags, and different library versions. It will not necessarily give the same results as the tool author's provided precompiled binary. (Wow, this ended up being a long email!) Regards, Peter

3 4

GATK 3.1
by Berner, Thomas 18 Mar '14

18 Mar '14

Hi, I've seen there is a new version (3.1-1) of GATK available at http://www.broadinstitute.org/gatk/download . Are there any plans of getting this version into Galaxy in the nearer future? Greetings, Thomas --- Thomas Berner Julius Kühn-Institut (JKI) - Federal Research Centre for Cultivated Plants - Erwin Baur-Straße 27 06484 Quedlinburg - Germany - Phone: ++49 ( 0 ) 3946 47 562 EMail: thomas.berner(a)jki.bund.de

3 2

UCSC Display problem
by Pete Schmitt 18 Mar '14

18 Mar '14

4 8

Persistent jobs in cluster queue even after canceling job in galaxy
by Ravi Alla 18 Mar '14

18 Mar '14

Hi All, I've been able to submit jobs to the cluster through galaxy, it works great. But when the job is in queue to run (it is gray in the galaxy history pane) and I cancel the job, it still remains in queue on the cluster. Why does this happen? How can I delete the jobs in queue as well? I tried qdel <job_id> as galaxy user but it says I am not authenticated to delete the job. Any help would be greatly appreciated. Thanks Ravi.

5 8

Verifying test output datatypes, was: Problem with <change_format> and conditional inputs?
by Peter Cock 18 Mar '14

18 Mar '14

Thanks John, I suggest making this test framework perform this check by default (the twill and API based frameworks) and seeing what - if anything - breaks as a result on the Test Tool Shed. Note that one area of fuzziness is subclasses, e.g. if the tool output was labelled "fastqsanger", but the <test> just said "fastq", I would say the test was broken. On the other hand, if the <test> used a specific datatype like "fastqsanger" but the tool produced a dataset tagged with a more generic datatype like "fastq" I think that is a again a real failure. Peter On Fri, Mar 14, 2014 at 3:15 PM, John Chilton <jmchilton(a)gmail.com> wrote: > Grepping around the code I think this is the only way a ftype > attribute on an output affects the evaluation of test data. > > if attributes.get( 'ftype', None ) == 'bam': > local_fh, temp_name = self._bam_to_sam( > local_name, temp_name ) > local_name = local_fh.name > > I am not sure it was ever meant as a strict test. > > I worry about breaking backward compatibility but it is easy enough to > implement this as an actual check when using newer API driven tests. I > have opened a pull request for this functionality here: > > https://bitbucket.org/galaxy/galaxy-central/pull-request/347/check-ftype-at… > > Thoughts? > > -John > > On Tue, Mar 11, 2014 at 10:30 AM, Peter Cock <p.j.a.cock(a)googlemail.com> wrote: >> >> That seems to work - thanks Björn :) >> >> This seems to have exposed a bug in the <test> framework, e.g. >> >> <test> >> <param name="query" value="rhodopsin_nucs.fasta" ftype="fasta" /> >> <param name="db_opts_selector" value="file" /> >> <param name="subject" value="three_human_mRNA.fasta" >> ftype="fasta" /> >> <param name="database" value="" /> >> <param name="evalue_cutoff" value="1e-40" /> >> <param name="out_format" value="5" /> >> <param name="adv_opts_selector" value="basic" /> >> <output name="output1" >> file="blastn_rhodopsin_vs_three_human.xml" ftype="blastxml" /> >> </test> >> >> This was saying the output should have been "blastxml", >> but until I just fixed it the output was being tagged as >> "tabular" (although run_functional_tests.sh did check >> the content it didn't check the datatype matched). >> >> Dave - do think this is a reasonable enhancement? >> >> Peter

2 4

BugFix for dynamic_options ParamValueFilter to run IUC SnpEff in a workflow
by Jim Johnson 18 Mar '14

18 Mar '14

The current code in: lib/galaxy/tools/parameters/dynamic_options.py only searches the top layer of the dict to find the dependency value. A fix is provide in pull request: #343: Need to traverse the other_value dict to find dependencies for ParamValueFilter in<https://bitbucket.org/galaxy/galaxy-central/pull-request/343/need-to-traver…> SnpEff tool_config <inputs> <param format="vcf,tabular,pileup,bed" name="input" type="data" label="Sequence changes (SNPs, MNPs, InDels)"/> ... <conditional name="snpDb"> <param name="genomeSrc" type="select" label="Genome source"> <option value="cached">Locally installed reference genome</option> <option value="history">Reference genome from your history</option> <option value="named">Named on demand</option> </param> <when value="cached"> <param name="genomeVersion" type="select" label="Genome">  <options from_data_table="snpeff_genomedb"> <filter type="unique_value" column="0" /> </options> </param> <param name="extra_annotations" type="select" display="checkboxes" multiple="true" label="Additional Annotations"> <help>These are available for only a few genomes</help> <options from_data_table="snpeff_annotations"> <filter type="param_value" ref="genomeVersion" key="genome" column="0" /> <filter type="unique_value" column="1" /> </options> </param> When running workflow: input.vcf -> SnpEff The values in ParamValueFilter filter_options function: |>>>| ||self*.*ref_name|| |'genomeVersion' ||>>>| ||other_values|| |{u'spliceSiteSize': '1', u'filterHomHet': 'no_filter', u'outputFormat': 'vcf', u'filterOut': None, u'inputFormat': 'vcf', u'filterIn': 'no_filter', u'udLength': '5000', u'generate_stats': True, u'noLog': True, u'chr': 'None', u'intervals': None, u'snpDb': {'extra_annotations': None, 'regulation': None, 'genomeVersion': 'GRCh37.71', 'genomeSrc': 'cached', '__current_case__': 0}, u'offset': '', u'input': <galaxy.tools.parameters.basic.DummyDataset object at 0x11451b8d0>, u'transcripts': None, u'annotations': ['-canon', '-lof', '-onlyReg']} || Since |||'genomeVersion' isn't in the keys of other_values, but rather in ||||||other_values['|||||||||snpDb'] this failed the assertion: |||||||||| assert self.ref_name in other_values, "Required dependency '%s' not found in incoming values" % self.ref_name Pull request |||||343: |||||<https://bitbucket.org/galaxy/galaxy-central/pull-request/343/need-to-traver…> |||||$ hg diff lib/galaxy/tools/parameters/dynamic_options.py diff -r 95517f976cca lib/galaxy/tools/parameters/dynamic_options.py --- a/lib/galaxy/tools/parameters/dynamic_options.py Thu Feb 27 16:56:25 2014 -0500 +++ b/lib/galaxy/tools/parameters/dynamic_options.py Fri Feb 28 11:37:04 2014 -0600 @@ -177,8 +177,27 @@ return self.ref_name def filter_options( self, options, trans, other_values ): if trans is not None and trans.workflow_building_mode: return [] - assert self.ref_name in other_values, "Required dependency '%s' not found in incoming values" % self.ref_name - ref = other_values.get( self.ref_name, None ) + ## Depth first traversal to find the value for a dependency + def get_dep_value(param_name, dep_name, cur_val, layer): + dep_val = cur_val + if isinstance(layer, dict ): + if dep_name in layer: + dep_val = layer[dep_name] + if param_name in layer: + return dep_val + else: + for l in layer.itervalues(): + dep_val = get_dep_value(param_name, dep_name, dep_val, l) + if dep_val: + break + elif isinstance( layer, list): + for l in layer: + dep_val = get_dep_value(param_name, dep_name, dep_val, l) + if dep_val: + break + return None + ref = get_dep_value(self.dynamic_option.tool_param.name, self.ref_name, None, other_values) + assert not ref, "Required dependency '%s' not found in incoming values" % self.ref_name for ref_attribute in self.ref_attribute: if not hasattr( ref, ref_attribute ): return [] #ref does not have attribute, so we cannot filter, return empty list -- James E. Johnson, Minnesota Supercomputing Institute, University of Minnesota

2 2