Hello Tool Developers,
Haven't known when to send this out, but I figure since you haven't received any e-mail from me today it might be a good time.
tl;dr - Tool functional tests experienced a significant overhaul over the last release and will continue to change over the next couple releases, but in a backward compatible so hopefully you will not need to care unless you want to.
The last release included a considerable overhaul to the framework used to functionally test tools (i.e. actually running those <test> tags in tool XML). This includes changes to how the tests are executed and an expansion in the expressiveness of these tests.
As some background for why the changes are being made, currently tools are tested by starting up a test Galaxy instance and using a library called Twill to exercise the GUI and upload data and run tools through by exercising actual HTML pages.
The last release added the optional ability to exercise these tool tests via Galaxy's tool API instead of via web forms, with the slightly longer term goal of eliminating the Twill testing option altogether. This will allow the Galaxy tool form API to evolve more quickly, eliminate much of the testing complexity and many of the limitations imposed by Twill while ensuring that Galaxy's tool running API is sufficiently expressive.
I have put a significant amount of effort into backward compatibility - so all current tests SHOULD pass using the new method. I know some people have included significant hacks with their test definitions to sculpt them deal with Twill-isms. These may fail - I would like to catalog what these are and see if we can hack up the test framework to deal with them or transition these tests to use cleaner mechanisms allowed for by the more direct representation of tool parameters consumed by the API.
If you are running tests locally with the run_functional_tests.sh script distributed with Galaxy, simply setting the environment variable GALAXY_TEST_DEFAULT_INTERACTOR to 'api' before running the script and all tests will be run through the API. Individual tests can be set to target the API (for instance if they use newer, API-only features outlined below) by adding the XML attribute interactor="api" to the test tag in the tool XML. This change works right now if you want to update test tool shed tools to target the new testing method and provide feedback.
As part of this transition, many bugs were fixed that affect both styles of test and new features were added. The biggest improvement is the ability to specify parameters in a nested structure matching the inputs themselves. This allows disambiguation of repeat and conditional parameters for both styles of functional tests and in the case of API driven tests allows specification of any number of repeat block instances in tests (one still can only specify at most one instance for any repeat tag while using Twill driven tests).
The following annotated tool XML examples concretely demonstrate this new expressiveness.
https://bitbucket.org/galaxy/galaxy-central/src/tip/test/functional/tools/di... https://bitbucket.org/galaxy/galaxy-central/src/tip/test/functional/tools/di...
Additional Enhancements:
Enhancement (Twill and API): Allow unspecified conditional test parameters. Previously specifying a parameter in a conditional but not the value of the conditional test parameter itself (the select or boolean parameter the conditional changes based on) resulted in cryptic test errors. Now the default is assumed if unspecified. https://bitbucket.org/galaxy/galaxy-central/commits/8b99c7bc5e11cd7c452c2ad8... https://bitbucket.org/galaxy/galaxy-central/commits/76bd49e658876f5cc90ec5b8...
Enhancement (Twill and API): Allow specifying tests for composite extra files without checking primary file. https://bitbucket.org/galaxy/galaxy-central/commits/45c1d11cfff552ce095c0561...
Enhancement (API-only): Allow testing metadata properties on test output datasets. https://bitbucket.org/galaxy/galaxy-central/commits/01f05ab8df7092b6e03f35fa...
Enhancement (API-only): Remove restriction of output order specification. https://bitbucket.org/galaxy/galaxy-central/commits/d05be33ad6b772c2a688d875...
Enhancement: Allow testing repeat blocks with min="1" (twill) or any non-zero value (API) https://bitbucket.org/galaxy/galaxy-central/commits/4615f7424dcb4d2b61be5f38...
Enhancement (API-only): Allow testing of data parameters with multiple="True". https://bitbucket.org/galaxy/galaxy-central/commits/211f30207d48101a1250983d...
Fuller break down of this is development is available on Trello https://trello.com/c/MsJctFem.
Once Galaxy and the tool shed have been modified to target the API when testing tools by default the tool XML syntax wiki page will be updated with examples of this new functionality.
-John