On Wed, Mar 18, 2015 at 10:13 AM, Peter Briggs <peter.briggs@manchester.ac.uk> wrote:
Hello Peter C.
Thanks for your comments and advice Ironically after I sent that email the tests for the next tool I looked at in the toolshed had been run in the past couple of days.
That's good. I can also confirm that the Test Tool Shed example I gave was tested overnight, although it looks like a novel failure: https://testtoolshed.g2.bx.psu.edu/view/peterjc/sample_seqs
I'm interested because I've got a couple of tools that seem to work okay when I run the tests locally, but are reported as failing on the toolshed, and I wanted to see if I'd addressed the issue for one of them by updating the tool dependencies.
I suggest asking on this mailing list, posting the shareable Tool Shed URL and a copy of the error message. Hopefully it will be a simple issue (forgetting a test file is my most common mistake), or it may be that another tool author has seen the same error (which can happen due to problems in Galaxy itself).
(As an aside I'd also assumed that passing tests is one of the conditions for the tool to be marked as "verified", but maybe that's not the case?)
I think there is still a human involved, but passing tests is expected for a "verified" tool.
However as you say, more testing locally seems to be a good way to go - thanks for your suggestions. I looked at planemo briefly a while ago and it looked good. The other issue I've had with testing is actually testing tool installation (i.e. tool_dependencies.xml) - I recall that planemo didn't deal with that side of things, so I had to set up the environment for the tests manually.
There is some work on tool_dependencies.xml ongoing within planemo, John Chilton would be the person to ask (CC'd directly).
Thanks again for your advice and the links, I will investigate trying to set up a local solution (including some CI testing!).
This is something where using planemo ought to be very useful - for now I still do my local testing with a test Galaxy instance via: $ ./run_tests.sh -id my_tool_id Peter