Thanks John,
that's quite the detailed response.

I agree with your reasoning, and I can see the progress in planemo -- the shed_test
command is great and I'm using it to see why some of my tool tests fail.
In this particular case there is nothing wrong with the framework though,
as the planemo test will also fail if samtools is not in the path.
I guess it's just because samtools is not installed on the machine that runs the testing framework.
In the light of the proposed changes I understand if you don't want to touch this aspect anymore,
and I'm happy to have samtools installed when doing the (planemo) testing!

Marius

On 12 June 2015 at 17:37, John Chilton <jmchilton@gmail.com> wrote:
tl;dr - The devteam is planning to remove the display of those test
results from the tool shed UI and replace them with something better
hopefully by the end of 2015.

Full Details:

As Peter has documented very well over the last year, there are large
number of problems with the install and test framework right now - it
has been very consistently broken. I think the samtools problem
specifically has been reported before for instance. At a recent
devteam meeting - I think the devteam reached a reluctant consensus
that it is very important (testing installs and tools) but until such
a time that we feel we can devote the the engineering time to keep it
up and running consistently or that the framework improves to the
point where this engineering effort is minimal - it would be better to
just remove the display of these tests in the tool shed interface -
since they are making perfectly functional tools look broken.

Planemo now implements a shed_test command
(http://planemo.readthedocs.org/en/latest/commands.html#shed-test-command)
that supports similar functionality to the install and test framework.
On the plus side it is much easier to run the tests through planemo
and the output is better and more flexible. It is not a complete
replacement right now however - since it doesn't hook into the tool
shed display at this point and isn't a service the Galaxy project is
providing. I think the team will do both of these - hopefully by
sometime this Fall - and then we will have a framework for installing
and testing tools that makes it trivial to run the tools from the
command-line or use within a tool shed display. I have created a
Planemo milestone to track some of these developments -
https://github.com/galaxyproject/planemo/milestones/Planemo-as-a-Service.

This may make people unhappy - it certainly doesn't make me happy -
but a year worth of brokenness does need to be addressed. So I would
encourage everyone to please respond with comments if you feel
strongly one way or the other - maybe we can make a new thread for
that and a more formal announcement of this change.

-John

On Thu, Jun 11, 2015 at 9:43 AM, Marius van den Beek
<m.vandenbeek@gmail.com> wrote:
> Hi list,
>
> I’ve been trying to write functional tests for a tool that produces BAM
> files as output. The tests run fine when using planemo locally,
> but not on the testtoolshed or on machines that do not have samtools
> installed.
> Apparently galaxy automatically tries to convert BAM to SAM to compare
> the output line by line, and so this fails. Surprisingly this is also true
> when
> comparing by size.
> This is the error:
>
> Tool id: bowtieForSmallRNA
> Tool version: bowtieForSmallRNA
> Test: test_tool_000000
> (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/drosofff/msp_sr_bowtie/bowtieForSmallRNA/1.1.0)
> Stderr:
>
> Traceback:
> Traceback (most recent call last):
>   File
> "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py",
> line 270, in test_tool
>     self.do_it( td )
>   File
> "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py",
> line 67, in do_it
>     raise e
> JobOutputsError: History item  different than expected, difference (using
> diff):
> (
> /tmp/shed_tools/testtoolshed.g2.bx.psu.edu/repos/drosofff/msp_sr_bowtie/316124e85b8d/msp_sr_bowtie/test-data/output.bam
> v.
> /tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/repositories_with_tools/tmp/tmpjxtteB/tmpQMbAfNoutput.bam
> )
> Converting local (test-data) bam to sam failed: (cmd=[samtools view -h -o
> "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/repositories_with_tools/tmp/tmpjxtteB/local_bam_converted_to_sam_8Ym0af.sam"
> "/tmp/shed_tools/testtoolshed.g2.bx.psu.edu/repos/drosofff/msp_sr_bowtie/316124e85b8d/msp_sr_bowtie/test-data/output.bam"],
> stdout=[], stderr=[/bin/sh: 1: samtools: not found
> ])
>
> Should samtools perhaps be added to the list of tools required for running
> galaxy?
>
> Best,
> Marius
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/