Hello Tool Developers,
Haven't known when to send this out, but I figure since you haven't
received any e-mail from me today it might be a good time.
tl;dr - Tool functional tests experienced a significant overhaul over
the last release and will continue to change over the next couple
releases, but in a backward compatible so hopefully you will not need
to care unless you want to.
The last release included a considerable overhaul to the framework
used to functionally test tools (i.e. actually running those <test>
tags in tool XML). This includes changes to how the tests are executed
and an expansion in the expressiveness of these tests.
As some background for why the changes are being made, currently tools
are tested by starting up a test Galaxy instance and using a library
called Twill to exercise the GUI and upload data and run tools through
by exercising actual HTML pages.
The last release added the optional ability to exercise these tool
tests via Galaxy's tool API instead of via web forms, with the
slightly longer term goal of eliminating the Twill testing option
altogether. This will allow the Galaxy tool form API to evolve more
quickly, eliminate much of the testing complexity and many of the
limitations imposed by Twill while ensuring that Galaxy's tool running
API is sufficiently expressive.
I have put a significant amount of effort into backward compatibility
- so all current tests SHOULD pass using the new method. I know some
people have included significant hacks with their test definitions to
sculpt them deal with Twill-isms. These may fail - I would like to
catalog what these are and see if we can hack up the test framework to
deal with them or transition these tests to use cleaner mechanisms
allowed for by the more direct representation of tool parameters
consumed by the API.
If you are running tests locally with the run_functional_tests.sh
script distributed with Galaxy, simply setting the environment
variable GALAXY_TEST_DEFAULT_INTERACTOR to 'api' before running the
script and all tests will be run through the API. Individual tests can
be set to target the API (for instance if they use newer, API-only
features outlined below) by adding the XML attribute interactor="api"
to the test tag in the tool XML. This change works right now if you
want to update test tool shed tools to target the new testing method
and provide feedback.
As part of this transition, many bugs were fixed that affect both
styles of test and new features were added. The biggest improvement is
the ability to specify parameters in a nested structure matching the
inputs themselves. This allows disambiguation of repeat and
conditional parameters for both styles of functional tests and in the
case of API driven tests allows specification of any number of repeat
block instances in tests (one still can only specify at most one
instance for any repeat tag while using Twill driven tests).
The following annotated tool XML examples concretely demonstrate this
Enhancement (Twill and API): Allow unspecified conditional test
parameters. Previously specifying a parameter in a conditional but not
the value of the conditional test parameter itself (the select or
boolean parameter the conditional changes based on) resulted in
cryptic test errors. Now the default is assumed if unspecified.
Enhancement (Twill and API): Allow specifying tests for composite
extra files without checking primary file.
Enhancement (API-only): Allow testing metadata properties on test
Enhancement (API-only): Remove restriction of output order
Enhancement: Allow testing repeat blocks with min="1" (twill) or any
non-zero value (API)
Enhancement (API-only): Allow testing of data parameters with multiple="True".
Fuller break down of this is development is available on Trello
Once Galaxy and the tool shed have been modified to target the API
when testing tools by default the tool XML syntax wiki page will be
updated with examples of this new functionality.