Functional test for BAM output
Hi list, I’ve been trying to write functional tests for a tool that produces BAM files as output. The tests run fine when using planemo locally, but not on the testtoolshed or on machines that do not have samtools installed. Apparently galaxy automatically tries to convert BAM to SAM to compare the output line by line, and so this fails. Surprisingly this is also true when comparing by size. This is the error: Tool id: bowtieForSmallRNA Tool version: bowtieForSmallRNA Test: test_tool_000000 (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/drosofff/msp_sr_bowtie/bowtieForSmallRNA/1.1.0) Stderr: Traceback: Traceback (most recent call last): File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 270, in test_tool self.do_it( td ) File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 67, in do_it raise e JobOutputsError: History item different than expected, difference (using diff): ( /tmp/shed_tools/testtoolshed.g2.bx.psu.edu/repos/drosofff/msp_sr_bowtie/316124e85b8d/msp_sr_bowtie/test-data/output.bam v. /tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/repositories_with_tools/tmp/tmpjxtteB/tmpQMbAfNoutput.bam ) Converting local (test-data) bam to sam failed: (cmd=[samtools view -h -o "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/repositories_with_tools/tmp/tmpjxtteB/local_bam_converted_to_sam_8Ym0af.sam" "/tmp/shed_tools/testtoolshed.g2.bx.psu.edu/repos/drosofff/msp_sr_bowtie/316124e85b8d/msp_sr_bowtie/test-data/output.bam"], stdout=[], stderr=[/bin/sh: 1: samtools: not found ]) Should samtools perhaps be added to the list of tools required for running galaxy? Best, Marius
On Thu, Jun 11, 2015 at 2:43 PM, Marius van den Beek <m.vandenbeek@gmail.com> wrote:
Hi list,
I’ve been trying to write functional tests for a tool that produces BAM files as output. The tests run fine when using planemo locally, but not on the testtoolshed or on machines that do not have samtools installed. Apparently galaxy automatically tries to convert BAM to SAM to compare the output line by line, and so this fails. ...
Should samtools perhaps be added to the list of tools required for running galaxy?
Hi Marius, I thought samtools was on the list of base dependencies for Galaxy - is it missing from somewhere you were reading (URL please)? Thanks, Peter
Hello Peter, I didn't read it in the docs -- I was just assuming that the instance that runs the functional tests would have all the dependencies installed, which appears not to be the case. Best, Marius On 12 June 2015 at 10:41, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Thu, Jun 11, 2015 at 2:43 PM, Marius van den Beek <m.vandenbeek@gmail.com> wrote:
Hi list,
I’ve been trying to write functional tests for a tool that produces BAM files as output. The tests run fine when using planemo locally, but not on the testtoolshed or on machines that do not have samtools installed. Apparently galaxy automatically tries to convert BAM to SAM to compare the output line by line, and so this fails. ...
Should samtools perhaps be added to the list of tools required for running galaxy?
Hi Marius,
I thought samtools was on the list of base dependencies for Galaxy - is it missing from somewhere you were reading (URL please)?
Thanks,
Peter
tl;dr - The devteam is planning to remove the display of those test results from the tool shed UI and replace them with something better hopefully by the end of 2015. Full Details: As Peter has documented very well over the last year, there are large number of problems with the install and test framework right now - it has been very consistently broken. I think the samtools problem specifically has been reported before for instance. At a recent devteam meeting - I think the devteam reached a reluctant consensus that it is very important (testing installs and tools) but until such a time that we feel we can devote the the engineering time to keep it up and running consistently or that the framework improves to the point where this engineering effort is minimal - it would be better to just remove the display of these tests in the tool shed interface - since they are making perfectly functional tools look broken. Planemo now implements a shed_test command (http://planemo.readthedocs.org/en/latest/commands.html#shed-test-command) that supports similar functionality to the install and test framework. On the plus side it is much easier to run the tests through planemo and the output is better and more flexible. It is not a complete replacement right now however - since it doesn't hook into the tool shed display at this point and isn't a service the Galaxy project is providing. I think the team will do both of these - hopefully by sometime this Fall - and then we will have a framework for installing and testing tools that makes it trivial to run the tools from the command-line or use within a tool shed display. I have created a Planemo milestone to track some of these developments - https://github.com/galaxyproject/planemo/milestones/Planemo-as-a-Service. This may make people unhappy - it certainly doesn't make me happy - but a year worth of brokenness does need to be addressed. So I would encourage everyone to please respond with comments if you feel strongly one way or the other - maybe we can make a new thread for that and a more formal announcement of this change. -John On Thu, Jun 11, 2015 at 9:43 AM, Marius van den Beek <m.vandenbeek@gmail.com> wrote:
Hi list,
I’ve been trying to write functional tests for a tool that produces BAM files as output. The tests run fine when using planemo locally, but not on the testtoolshed or on machines that do not have samtools installed. Apparently galaxy automatically tries to convert BAM to SAM to compare the output line by line, and so this fails. Surprisingly this is also true when comparing by size. This is the error:
Tool id: bowtieForSmallRNA Tool version: bowtieForSmallRNA Test: test_tool_000000 (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/drosofff/msp_sr_bowtie/bowtieForSmallRNA/1.1.0) Stderr:
Traceback: Traceback (most recent call last): File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 270, in test_tool self.do_it( td ) File "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 67, in do_it raise e JobOutputsError: History item different than expected, difference (using diff): ( /tmp/shed_tools/testtoolshed.g2.bx.psu.edu/repos/drosofff/msp_sr_bowtie/316124e85b8d/msp_sr_bowtie/test-data/output.bam v. /tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/repositories_with_tools/tmp/tmpjxtteB/tmpQMbAfNoutput.bam ) Converting local (test-data) bam to sam failed: (cmd=[samtools view -h -o "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/repositories_with_tools/tmp/tmpjxtteB/local_bam_converted_to_sam_8Ym0af.sam" "/tmp/shed_tools/testtoolshed.g2.bx.psu.edu/repos/drosofff/msp_sr_bowtie/316124e85b8d/msp_sr_bowtie/test-data/output.bam"], stdout=[], stderr=[/bin/sh: 1: samtools: not found ])
Should samtools perhaps be added to the list of tools required for running galaxy?
Best, Marius
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
On Fri, Jun 12, 2015 at 4:37 PM, John Chilton <jmchilton@gmail.com> wrote:
tl;dr - The devteam is planning to remove the display of those test results from the tool shed UI and replace them with something better hopefully by the end of 2015.
<snip>
This may make people unhappy - it certainly doesn't make me happy - but a year worth of brokenness does need to be addressed. So I would encourage everyone to please respond with comments if you feel strongly one way or the other - maybe we can make a new thread for that and a more formal announcement of this change.
-John
I'd agree with John that this is not ideal, but would still be an improvement on the current situation where the Tool Shed test results are not useful to anyone. A new thread formally announcing this proposed change would be a good idea. Peter
Thanks John, that's quite the detailed response. I agree with your reasoning, and I can see the progress in planemo -- the shed_test command is great and I'm using it to see why some of my tool tests fail. In this particular case there is nothing wrong with the framework though, as the planemo test will also fail if samtools is not in the path. I guess it's just because samtools is not installed on the machine that runs the testing framework. In the light of the proposed changes I understand if you don't want to touch this aspect anymore, and I'm happy to have samtools installed when doing the (planemo) testing! Marius On 12 June 2015 at 17:37, John Chilton <jmchilton@gmail.com> wrote:
tl;dr - The devteam is planning to remove the display of those test results from the tool shed UI and replace them with something better hopefully by the end of 2015.
Full Details:
As Peter has documented very well over the last year, there are large number of problems with the install and test framework right now - it has been very consistently broken. I think the samtools problem specifically has been reported before for instance. At a recent devteam meeting - I think the devteam reached a reluctant consensus that it is very important (testing installs and tools) but until such a time that we feel we can devote the the engineering time to keep it up and running consistently or that the framework improves to the point where this engineering effort is minimal - it would be better to just remove the display of these tests in the tool shed interface - since they are making perfectly functional tools look broken.
Planemo now implements a shed_test command (http://planemo.readthedocs.org/en/latest/commands.html#shed-test-command) that supports similar functionality to the install and test framework. On the plus side it is much easier to run the tests through planemo and the output is better and more flexible. It is not a complete replacement right now however - since it doesn't hook into the tool shed display at this point and isn't a service the Galaxy project is providing. I think the team will do both of these - hopefully by sometime this Fall - and then we will have a framework for installing and testing tools that makes it trivial to run the tools from the command-line or use within a tool shed display. I have created a Planemo milestone to track some of these developments - https://github.com/galaxyproject/planemo/milestones/Planemo-as-a-Service.
This may make people unhappy - it certainly doesn't make me happy - but a year worth of brokenness does need to be addressed. So I would encourage everyone to please respond with comments if you feel strongly one way or the other - maybe we can make a new thread for that and a more formal announcement of this change.
-John
Hi list,
I’ve been trying to write functional tests for a tool that produces BAM files as output. The tests run fine when using planemo locally, but not on the testtoolshed or on machines that do not have samtools installed. Apparently galaxy automatically tries to convert BAM to SAM to compare the output line by line, and so this fails. Surprisingly this is also
On Thu, Jun 11, 2015 at 9:43 AM, Marius van den Beek <m.vandenbeek@gmail.com> wrote: true
when comparing by size. This is the error:
Tool id: bowtieForSmallRNA Tool version: bowtieForSmallRNA Test: test_tool_000000 ( functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/drosofff/msp_sr_bowtie/bowtieForSmallRNA/1.1.0 ) Stderr:
Traceback: Traceback (most recent call last): File
"/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py",
line 270, in test_tool self.do_it( td ) File
"/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py",
line 67, in do_it raise e JobOutputsError: History item different than expected, difference (using diff): ( /tmp/shed_tools/ testtoolshed.g2.bx.psu.edu/repos/drosofff/msp_sr_bowtie/316124e85b8d/msp_sr_bowtie/test-data/output.bam v.
/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/repositories_with_tools/tmp/tmpjxtteB/tmpQMbAfNoutput.bam
) Converting local (test-data) bam to sam failed: (cmd=[samtools view -h -o
"/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/repositories_with_tools/tmp/tmpjxtteB/local_bam_converted_to_sam_8Ym0af.sam"
"/tmp/shed_tools/ testtoolshed.g2.bx.psu.edu/repos/drosofff/msp_sr_bowtie/316124e85b8d/msp_sr_bowtie/test-data/output.bam "], stdout=[], stderr=[/bin/sh: 1: samtools: not found ])
Should samtools perhaps be added to the list of tools required for running galaxy?
Best, Marius
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
participants (3)
-
John Chilton -
Marius van den Beek -
Peter Cock