Hello, The current tool testing options provided by Galaxy are excellent for verifying that Galaxy is functioning properly and that tools provide reproducible results. However, I believe there are two related shortcomings to the current Galaxy approach, and I would like to propose a solution for these shortcomings. The first is that the workflow for the tool developer is very clunky, especially if the developer is building up a tool incrementally with a test-driven development (TDD) approach. For each parameter addition a whole new output file must be created externally, manually verified by the developer, possibly converted to a regular expression, and placed in the test data directory. Put another way, it is easy to iteratively build up a tool, but not to iteratively build a test case. The second point is a bit pedantic. The current approach only verifies that the output is the same as the supplied output, not that the output is actually "correct". The typical workflow I described above relies on manual inspection of the expected output file to verify it is "correct". I believe it is better to programmatically state assertions about what makes an output correct than to rely on manual verification, this serves both to reduce human error and act as documentation about what makes an output correct. To address these two points, I propose an extensible addition to the Galaxy tool syntax (and provide an implementation) to declaratively make assertions about output files. The syntax change is to the output child element of test elements. With this change, the output element may have a child element named assert_contents, which in turn may have any number of child elements each of which describes an assertion about the contents of the referenced output file. The file attribute on the output element is still required if an assert_contents element is not found, but it is optional if an assert_contents element is found. The whole file check described by the file attribute will be executed in addition to the listed assertions if both are present. As an example, the following fragment assserts that an output file contains the text 'chr7', does not contain the text 'chr8', and has a line matching the regular expression '.*\s+127489808\s+127494553'. <test> <param name="input" value="maf_stats_interval_in.dat" /> <output name="out_file1"> <assert_contents> <has_text text="chr7" /> <not_has_text text="chr8" /> <has_line_matching expression=".*\s+127489808\s+127494553" /> </assert_contents> </output> </test> Each potential child element of assert_contents corresponds to a simple python function. These functions are broken out into modules which are dynamically (mostly) loaded at test time. The extensibility of this approach comes from how trivial it is to add new assertion functions to a module and whole new modules of such functions. I have started work on three modules of assertion functions, these are for text, tabular, and XML output files respectively. has_text, not_has_text, and has_line_matching above are examples of three such assertion functions from the text module. To see how it works, here is a function from the file test/base/asserts/text.py defining the has_line_matching element: def assert_has_line_matching(output, expression): """ Asserts the specified output contains a line matching the regular expression specified by the argument expression.""" match = re.search("^%s$" % expression, output, flags = re.MULTILINE) assert match != None, "No line matching expression '%s' was found in output file." % expression As demonstrated, the function name corresponding to the element element_name is just assert_element_name. The code that calls these assertion functions, automatically matches XML attributes with function arguments by names, and matches an argument named output with a string containing the contents of the output file resulting from the test run. Matching function arguments this way gracefully allows for multiple arguments and optional arguments. There is additional information about the implementation at the end of this e-mail. This approach should really aide iterative development of tools. Each new parameter you add to a tool is going to change the output in some way, hopefully you will be able to describe how it affects the output as an assertion. As you add new parameters, the previous parameters will hopefully affect the output in the same way and the old assertion will not need to change, you will just need to add new ones. Obviously this won't always be the case, but hopefully changes to previous assertions will be minimal over time. I believe this process will be faster over time than repeatedly producing output files or interactive GUI based testing, and the final product will be a richer test case. I have attached two patches. The first patch (implementation.patch) is the patch that I propose merging into galaxy-central. It modifies the tool parser to parse these new elements, modifies twilltestcase.py to perform the assertions, and includes the three modules of assertions described above. The second patch (examples.patch) adds a data files to the test-data directory and modifies tools/filters/headWrapper.xml to demonstrate each of the initial assertions elements I have defined. This patch merely proves it works and provides a sandbox to quickly play around with these assertions from working examples, this is not meant to be merged into galaxy-central. The first patch can be imported by executing the following command from an up-to-date galaxy-central pull: hg patch /path/to/implementation.patch To try it out, apply the second patch (examples.patch) and run the functional test Show_beginning1 hg patch /path/to/examples.patch ./run_functional_tests.sh -id Show_beginning1 To view the examples ran with the test see tools/filters/headWrapper.xml after applying the second patch. Let me know if you have any questions, concerns, or if there are any changes I can make to get this work included in galaxy-central. Thanks for your time and consideration, -John ------------------------------------------------ John Chilton Software Developer University of Minnesota Supercomputing Institute Office: 612-625-0917 Cell: 612-226-9223 E-Mail: chilton@msi.umn.edu Advanced Usage: In addition to the argument output defined above, there are two other argument names that when used have a special meaning - children and verify_assertions_function. children is a parsed dictionary-based python description of the child element of the assertion element. verify_assertions_function is a function that takes in a string and the same parsed dictionary-based python description of assertion XML elements and checks them. Used in conjunction these can be used to by assertion function authors to allow for the expression recursively defining assertion over some subset of the output. Here is an example: <output name="out_file1"> <assert_contents> <element_text path="BlastOutput_iterations/Iteration/Iteration_hits/Hit/Hit_def"> <not_has_text text="EDK72998.1" /> <has_text_matching expression="ABK[\d\.]+" /> </element_text> </assert_contents> </output> With corresponding Python function definition: def assert_element_text(output, path, verify_assertions_function, children): """ Recursively checks the specified assertions against the text of the first element matching the specified path.""" text = xml_find_text(output, path) verify_assertions_function(text, children) The children argument could also be used to define other assertion specific syntaxes not dependent on verify_assertions_function. A note on this example: This example is admittedly convoluted, but it is working and included in the patch described above. My real desire for this functionality is for some tools I am developing that produce zip files. Stock Galaxy doesn't really play well with zip file base datatypes so this code is not included in the implementation, but you can imagine why this would be useful. I hope to be able to define tests like: <zip_has_file name="subfile"> <has_text text="subfile text" /> </zip_has_file>