On Tue, May 7, 2013 at 7:02 PM, Greg Von Kuster <greg@bx.psu.edu> wrote:
Hi Peter,
Missing test components implies a tool config that does not define a test (i.e, a missing test definition) or a tool config that defines a test, but the test's input or output files are missing from the repository.
This seems to be our point of confusion: I don't understand combining these two categories - it seems unhelpful to me. Tools missing a test definition clearly can't be tested - but since we'd like every tool to have tests having this as an easily view listing is useful both for authors and reviewers. It highlights tools which need some work - or in some cases work on the Galaxy test framework itself. They are neither passing nor failing tests - and it makes sense to list them separately. Tools with a test definition should be tested - if they are missing an input or output file this is just a special case of a test failure (and can be spotted without actually attempting to run the tool). This is clearly a broken test and the tool author should be able to fix this easily (by uploading the missing test data file)
I don't see the benefit of the above where you place tools missing tests into a different category than tools with defined tests, but missing test data. If any of the test components (test definition or required input or output files) are missing, then the test cannot be executed, so defining it as a failing test in either case is a bit misleading. It is actually a tool that is missing test components that are required for execution which will result in a pass / fail status.
It is still a failing test (just for the trivial reason of missing a test data file).
It would be much simpler to change the filter for failing tests to include those that are missing test components so that the list of missing test components is a subset of the list of failing tests.
What I would like is three lists: Latest revision: missing tool tests - repositories where at least 1 tool has no test defined [The medium term TO-DO list for the Tool Author] Latest revision: failing tool tests - repositories where at least 1 tool has a failing test (where I include tests missing their input or output test data files) [The priority TO-DO list for the Tool Author] Latest revision: all tool tests pass - repositories where every tool has tests and they all pass [The good list, Tool Authors should aim to have everything here] Right now http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus would appear under both "missing tool tests" and "failing tool tests", but I hope to fix this and have this under "missing tool tests" only (until my current roadblocks with the Galaxy Test Framework are resolved). I hope I've managed a clearer explanation this time, Thanks, Peter