Hi Peter, On May 7, 2013, at 9:10 AM, Peter Cock wrote:
On Tue, May 7, 2013 at 10:59 AM, Greg Von Kuster <greg@bx.psu.edu> wrote:
Hi Peter,
This change would be simple to make, but it would result in the inclusion of all repositories with missing test components to be included in the list of failing tests. There are currently 2 filters that provide the information to categorize these repositories: missing test components ( True / False ) and tools functionally correct ( True / False ). I'm not sure it makes sense to add a 3rd filter, but if we can come up with a sensible one, I would be happy to add it. If not, do you want the list of failing tool tests to include repositories that are missing test components?
By missing test components do you mean things like missing a test input or output file? If so, that would be fine. If I've written a test but not included the sample data, then to me that is just a special case of a failing test that needs fixing (and it should be a trivial fix).
Missing test components implies a tool config that does not define a test (i.e, a missing test definition) or a tool config that defines a test, but the test's input or output files are missing from the repository.
Currently (according to the yellow notes on the Test Tool Shed),
Latest revision: missing tool tests: * you are authorized to update them * the latest installable revision contains at least 1 tool with no defined tests OR: * the latest installable revision contains at least 1 tool with a test that requires a missing test data file
Latest revision: failing tool tests: * you are authorized to update them * the latest installable revision contains at least 1 tool * the latest installable revision is not missing any tool test components * the latest installable revision has at least 1 tool test that fails
My suggestion would be to treat missing test data files as a failing test, something like this:
Latest revision: missing tool tests: * you are authorized to update them * the latest installable revision contains at least 1 tool with no defined tests
I don't see the benefit of the above where you place tools missing tests into a different category than tools with defined tests, but missing test data. If any of the test components (test definition or required input or output files) are missing, then the test cannot be executed, so defining it as a failing test in either case is a bit misleading. It is actually a tool that is missing test components that are required for execution which will result in a pass / fail status. It would be much simpler to change the filter for failing tests to include those that are missing test components so that the list of missing test components is a subset of the list of failing tests.
Latest revision: failing tool tests: * you are authorized to update them * the latest installable revision has at least 1 tool test that fails OR requires a missing test data file
Under this scheme, right now on the Test Tool Shed my ncbi_blast_plus repository would appear on both lists due to missing tests for tools like ncbi_rpsblast_wrapper, and failing tests like ncbi_blastn_wrapper (due to a glitch in the dependency installation changes I made).
Peter