Peter,

That is the intended behavior, but I've added a trello card (https://trello.com/c/O9YmzUT4) for revisiting that decision at some point. We are definitely willing to be flexible about the testing conditions, but the primary goal of the automated testing framework was to verify functional correctness of an entire repository, to simplify the approval process for the category of "tools contained within the repository".

--Dave B.

On 4/25/13 04:55:50.000, Peter Cock wrote:

On Wed, Apr 24, 2013 at 10:12 AM, Peter Cock p.j.a.cock@googlemail.com wrote:

...

..., I've still finding some other cases where despite having tests defined, nothing shows up on the Tool Shed test results page.

Here's an example where I see that two tools (promoter2 and wolf_psort) have no tests, but the results of all the other tools in this suite are not mentioned (some expected to fail/be skipped due to missing dependencies):

http://toolshed.g2.bx.psu.edu/view/peterjc/tmhmm_and_signalp Revision 13:dc958c2a963a

Equivalently:

http://testtoolshed.g2.bx.psu.edu/view/peterjc/tmhmm_and_signalp Revision: 14:0d8b1d20ce9c

---

Here's another example where based on what happens locally, I am expecting a failure:

http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename Revision: 1:6ce8634e2035

This is failing for me locally, but again I suspect this is due to an issue in the framework - much like the blastxml_to_top_descr test is failing due to the datatypes not being updated, here there seems to be something similar going wrong regarding the second uploaded input file (which is used for to provide the potential column parameters). See this thread:

http://lists.bx.psu.edu/pipermail/galaxy-dev/2010-November/003867.html http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-April/014261.html

Regards,

Peter

Hi Dave,

Yet another case,

http://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus Revision: 8:1f546099212f

and,

http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus Revision: 5:d42346e675c4

These list five or six invalid tools which do not define any unit tests, yet there is no mention of the tools present which do have tests (and if they passed or not).

Even if ncbi_blastdbcmd_wrapper doesn't (yet) have a test, I still want to know if the ncbi_blastp_wrapper tests pass ;)

Are this further symptoms of whatever was solved here? https://trello.com/card/toolshed-automated-functional-test-bug/506338ce32ae4...

Or does would a new Trello issue be best?

Thanks,

Peter

Peter,

Your point is well taken about the utility of the testing framework in the tool development process. The framework has been modified as of 9520:41d8cdde4729 to only flag a changeset revision not to be tested if no valid tests have been found in that revision. If one or more tools have functional tests, and there is test data for one or more of those tests, the repository will now be installed and tested.

I will be re-running the testing framework on the test tool shed shortly.

--Dave B.

On 4/25/13 09:58:39.000, Peter Cock wrote:

Hi Dave,

On Thu, Apr 25, 2013 at 2:27 PM, Dave Bouvier dave@bx.psu.edu wrote:

...

Peter,

That is the intended behavior, but I've added a trello card (https://trello.com/c/O9YmzUT4) for revisiting that decision at some point.

Is that the right Trello card? Is seems to be all about multiple tool shed repository revisions rather than what I'm asking about.

...

We are definitely willing to be flexible about the testing conditions, but the primary goal of the automated testing framework was to verify functional correctness of an entire repository, to simplify the approval process for the category of "tools contained within the repository".

--Dave B.

Well as you can tell, I disagree with that design choice: If there are tools with tests within a repository I think those tests should be run. It seems far more pragmatic than the current all-or-nothing approach (for which I can't see any real justification).

This is especially frustrating when there are still obstacles from the test framework itself which block writing tests for all my tools. e.g. http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-April/014280.html

Given "the primary goal of the automated testing framework was to verify functional correctness of an entire repository", not running any tests if just one tool is missing tests seems very bizarre. Right now I can run those tests locally, but I have no way to know if they would work on the Tool Shed tests - until I reach full test coverage for the repository. If all the tests provided were actually run, I can fix any failures now, rather than waiting until issues blocking adding any missing tests are resolved.

I hope this aspect of the design can be revisited sooner rather than later.

Thanks,

Peter