Hello,
The current tool testing options provided by Galaxy are excellent for
verifying that Galaxy is functioning properly and that tools provide
reproducible results.
However, I believe there are two related shortcomings to the current
Galaxy approach, and I would like to propose a solution for these
shortcomings. The first is that the workflow for the tool developer is
very clunky, especially if the developer is building up a tool
incrementally with a test-driven development (TDD) approach. For each
parameter addition a whole new output file must be created externally,
manually verified by the developer, possibly converted to a regular
expression, and placed in the test data directory. Put another way, it
is easy to iteratively build up a tool, but not to iteratively build a
test case.
The second point is a bit pedantic. The current approach only verifies
that the output is the same as the supplied output, not that the
output is actually "correct". The typical workflow I described above
relies on manual inspection of the expected output file to verify it
is "correct". I believe it is better to programmatically state
assertions about what makes an output correct than to rely on manual
verification, this serves both to reduce human error and act as
documentation about what makes an output correct.
To address these two points, I propose an extensible addition to the
Galaxy tool syntax (and provide an implementation) to declaratively
make assertions about output files.
The syntax change is to the output child element of test elements.
With this change, the output element may have a child element named
assert_contents, which in turn may have any number of child elements
each of which describes an assertion about the contents of the
referenced output file. The file attribute on the output element is
still required if an assert_contents element is not found, but it is
optional if an assert_contents element is found. The whole file check
described by the file attribute will be executed in addition to the
listed assertions if both are present.
As an example, the following fragment assserts that an output file
contains the text 'chr7', does not contain the text 'chr8', and has a
line matching the regular expression '.*\s+127489808\s+127494553'.
<test>
<param name="input" value="maf_stats_interval_in.dat" />
<output name="out_file1">
<assert_contents>
<has_text text="chr7" />
<not_has_text text="chr8" />
<has_line_matching expression=".*\s+127489808\s+127494553" />
</assert_contents>
</output>
</test>
Each potential child element of assert_contents corresponds to a
simple python function. These functions are broken out into modules
which are dynamically (mostly) loaded at test time. The extensibility
of this approach comes from how trivial it is to add new assertion
functions to a module and whole new modules of such functions.
I have started work on three modules of assertion functions, these are
for text, tabular, and XML output files respectively. has_text,
not_has_text, and has_line_matching above are examples of three such
assertion functions from the text module. To see how it works, here is
a function from the file test/base/asserts/text.py defining the
has_line_matching element:
def assert_has_line_matching(output, expression):
""" Asserts the specified output contains a line matching the
regular expression specified by the argument expression."""
match = re.search("^%s$" % expression, output, flags = re.MULTILINE)
assert match != None, "No line matching expression '%s' was found
in output file." % expression
As demonstrated, the function name corresponding to the element
element_name is just assert_element_name. The code that calls these
assertion functions, automatically matches XML attributes with
function arguments by names, and matches an argument named output with
a string containing the contents of the output file resulting from the
test run. Matching function arguments this way gracefully allows for
multiple arguments and optional arguments. There is additional
information about the implementation at the end of this e-mail.
This approach should really aide iterative development of tools. Each
new parameter you add to a tool is going to change the output in some
way, hopefully you will be able to describe how it affects the output
as an assertion. As you add new parameters, the previous parameters
will hopefully affect the output in the same way and the old assertion
will not need to change, you will just need to add new ones. Obviously
this won't always be the case, but hopefully changes to previous
assertions will be minimal over time. I believe this process will be
faster over time than repeatedly producing output files or interactive
GUI based testing, and the final product will be a richer test case.
I have attached two patches. The first patch (implementation.patch) is
the patch that I propose merging into galaxy-central. It modifies the
tool parser to parse these new elements, modifies twilltestcase.py to
perform the assertions, and includes the three modules of assertions
described above. The second patch (examples.patch) adds a data files
to the test-data directory and modifies tools/filters/headWrapper.xml
to demonstrate each of the initial assertions elements I have defined.
This patch merely proves it works and provides a sandbox to quickly
play around with these assertions from working examples, this is not
meant to be merged into galaxy-central.
The first patch can be imported by executing the following command
from an up-to-date galaxy-central pull:
hg patch /path/to/implementation.patch
To try it out, apply the second patch (examples.patch) and run the
functional test Show_beginning1
hg patch /path/to/examples.patch
./run_functional_tests.sh -id Show_beginning1
To view the examples ran with the test see
tools/filters/headWrapper.xml after applying the second patch.
Let me know if you have any questions, concerns, or if there are any
changes I can make to get this work included in galaxy-central.
Thanks for your time and consideration,
-John
------------------------------------------------
John Chilton
Software Developer
University of Minnesota Supercomputing Institute
Office: 612-625-0917
Cell: 612-226-9223
E-Mail: chilton(a)msi.umn.edu
Advanced Usage:
In addition to the argument output defined above, there are two other
argument names that when used have a special meaning - children and
verify_assertions_function. children is a parsed dictionary-based
python description of the child element of the assertion element.
verify_assertions_function is a function that takes in a string and
the same parsed dictionary-based python description of assertion XML
elements and checks them. Used in conjunction these can be used to by
assertion function authors to allow for the expression recursively
defining assertion over some subset of the output. Here is an example:
<output name="out_file1">
<assert_contents>
<element_text
path="BlastOutput_iterations/Iteration/Iteration_hits/Hit/Hit_def">
<not_has_text text="EDK72998.1" />
<has_text_matching expression="ABK[\d\.]+" />
</element_text>
</assert_contents>
</output>
With corresponding Python function definition:
def assert_element_text(output, path, verify_assertions_function, children):
""" Recursively checks the specified assertions against the text of
the first element matching the specified path."""
text = xml_find_text(output, path)
verify_assertions_function(text, children)
The children argument could also be used to define other assertion
specific syntaxes not dependent on verify_assertions_function.
A note on this example: This example is admittedly convoluted, but it
is working and included in the patch described above. My real desire
for this functionality is for some tools I am developing that produce
zip files. Stock Galaxy doesn't really play well with zip file base
datatypes so this code is not included in the implementation, but you
can imagine why this would be useful. I hope to be able to define
tests like:
<zip_has_file name="subfile">
<has_text text="subfile text" />
</zip_has_file>