Running functional tests is too slow
Hi all, I'd like to echo Florent's concern that run_functional_tests.sh is too slow, and that this discourages Tool Authors from adding more tests to their tools: https://trello.com/c/wL21d2do/1017-functional-tests-take-too-long (And encourage people to vote up that issue ;) ) Florent has identified one clear bottleneck is the creation of a fresh SQLite database each time, upon which the growing number of schema migration calls must be performed. Could we not cache an empty but up to date copy of this database? i.e. Something like this: 1. run_functional_tests.sh starts 2. If it exists, copy empty_test_sqlite_database.db, otherwise create new test SQLite database as now. 3. Check the schema version. 4. If the temporary SQlite database is out of date, run the migration, and then save a copy of this now up to date database as empty_test_sqlite_database.db, 5. run the tests Perhaps the empty SQLite database could even be cached in BitBucket too, for a faster first test run with a clean checkout? Separately, running the tests themselves seems overly slow - can anything be tweaked here? For example, is there any point executing the external set_meta script in the test environment? Regards, Peter P.S. On a related note, my achievement last Friday was to get TravisCI doing continuous integration testing of a GitHub repository of Galaxy tools: https://travis-ci.org/peterjc/galaxy_blast https://github.com/peterjc/galaxy_blast For those not familiar with this system, the idea is that via a special .travis.yml configuration file, each time new commits are pushed to GitHub, the latest code is checked out and tested on a virtual machine. i.e. Continuous in the sense of running the tests after each change, rather than just once a night. Right now the tests for the BLAST+ suite and associated tool wrappers like Blast2GO takes from 15 to 20 minutes, which I feel could be much improved.
Peter, Yes, the functional test suite is a bit on the slow side, and one of my long-term goals has been to improve the performance as best I can. --Dave B. On 09/16/2013 07:24 AM, Peter Cock wrote:
Hi all,
I'd like to echo Florent's concern that run_functional_tests.sh is too slow, and that this discourages Tool Authors from adding more tests to their tools:
https://trello.com/c/wL21d2do/1017-functional-tests-take-too-long
(And encourage people to vote up that issue ;) )
Florent has identified one clear bottleneck is the creation of a fresh SQLite database each time, upon which the growing number of schema migration calls must be performed. Could we not cache an empty but up to date copy of this database?
i.e. Something like this:
1. run_functional_tests.sh starts 2. If it exists, copy empty_test_sqlite_database.db, otherwise create new test SQLite database as now. 3. Check the schema version. 4. If the temporary SQlite database is out of date, run the migration, and then save a copy of this now up to date database as empty_test_sqlite_database.db, 5. run the tests
And that would be very easy to automate, with the environment variable GALAXY_TEST_DBURI set to the path to the empty, migrated database, and a few aliases or scripts to replace the test database file with the "template" file on each run.
Perhaps the empty SQLite database could even be cached in BitBucket too, for a faster first test run with a clean checkout?
I'm not sure of that idea, it would feel a bit like adding a non-essential file to the codebase which is generated programmatically by existing code.
Separately, running the tests themselves seems overly slow - can anything be tweaked here? For example, is there any point executing the external set_meta script in the test environment?
The switch to the external set_meta script was done because setting metadata internally was becoming computationally expensive. As I understand it, switching to internal metadata would actually lead to poorer performance in the functional tests.
Regards,
Peter
P.S. On a related note, my achievement last Friday was to get TravisCI doing continuous integration testing of a GitHub repository of Galaxy tools:
https://travis-ci.org/peterjc/galaxy_blast https://github.com/peterjc/galaxy_blast
For those not familiar with this system, the idea is that via a special .travis.yml configuration file, each time new commits are pushed to GitHub, the latest code is checked out and tested on a virtual machine. i.e. Continuous in the sense of running the tests after each change, rather than just once a night.
Right now the tests for the BLAST+ suite and associated tool wrappers like Blast2GO takes from 15 to 20 minutes, which I feel could be much improved. ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
On Mon, Sep 16, 2013 at 2:30 PM, Dave Bouvier <dave@bx.psu.edu> wrote:
Peter,
Yes, the functional test suite is a bit on the slow side, and one of my long-term goals has been to improve the performance as best I can.
Great - I appreciate there are lots of other high priority issues with the test framework (e.g. repeats and conditionals), so speed alone won't be top of the list.
Florent has identified one clear bottleneck is the creation of a fresh SQLite database each time, upon which the growing number of schema migration calls must be performed. Could we not cache an empty but up to date copy of this database?
i.e. Something like this:
1. run_functional_tests.sh starts 2. If it exists, copy empty_test_sqlite_database.db, otherwise create new test SQLite database as now. 3. Check the schema version. 4. If the temporary SQlite database is out of date, run the migration, and then save a copy of this now up to date database as empty_test_sqlite_database.db, 5. run the tests
And that would be very easy to automate, with the environment variable GALAXY_TEST_DBURI set to the path to the empty, migrated database, and a few aliases or scripts to replace the test database file with the "template" file on each run.
I see that environment variable already exists, perhaps I can hack something together... at least enough to show a worthwhile time saving.
Perhaps the empty SQLite database could even be cached in BitBucket too, for a faster first test run with a clean checkout?
I'm not sure of that idea, it would feel a bit like adding a non-essential file to the codebase which is generated programmatically by existing code.
External caching would be enough (and cleaner).
Separately, running the tests themselves seems overly slow - can anything be tweaked here? For example, is there any point executing the external set_meta script in the test environment?
The switch to the external set_meta script was done because setting metadata internally was becoming computationally expensive. As I understand it, switching to internal metadata would actually lead to poorer performance in the functional tests.
Sorry, I was unclear - although that is interesting. What I meant was, do we even need the metadata for validating the test results? If not, skip generating it. Regards, Peter
--Dave B. On 09/16/2013 09:38 AM, Peter Cock wrote:
Separately, running the tests themselves seems overly slow - can anything be tweaked here? For example, is there any point executing the external set_meta script in the test environment?
The switch to the external set_meta script was done because setting metadata internally was becoming computationally expensive. As I understand it, switching to internal metadata would actually lead to poorer performance in the functional tests.
Sorry, I was unclear - although that is interesting.
What I meant was, do we even need the metadata for validating the test results? If not, skip generating it.
Unfortunately, the metadata needs to be set in order to populate and validate input fields, and I don't believe it's currently possible to limit metadata setting to uploaded files, and not output datasets.
Regards,
Peter
On Mon, Sep 16, 2013 at 2:56 PM, Dave Bouvier <dave@bx.psu.edu> wrote:
On 09/16/2013 09:38 AM, Peter Cock wrote:
What I meant was, do we even need the metadata for validating the test results? If not, skip generating it.
Unfortunately, the metadata needs to be set in order to populate and validate input fields, ...
That makes sense, although I personally would think the current approach via the upload tool is an unnecessary overhead in itself - why not work with the input files in the test-data folder directly without copying them? This could also solve the problem of only being able to write tests with input files which support uploading - right now for example I can't write tests using BLAST databases (composite datatypes) as inputs.
and I don't believe it's currently possible to limit metadata setting to uploaded files, and not output datasets.
Do you think it would save much time if that was disabled? Or is the metadata not that slow and just produces lots of logging output drawing attention to itself? Thanks, Peter
On Mon, Sep 16, 2013 at 2:38 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Mon, Sep 16, 2013 at 2:30 PM, Dave Bouvier <dave@bx.psu.edu> wrote:
Peter,
Yes, the functional test suite is a bit on the slow side, and one of my long-term goals has been to improve the performance as best I can.
...
Perhaps the empty SQLite database could even be cached in BitBucket too, for a faster first test run with a clean checkout?
I'm not sure of that idea, it would feel a bit like adding a non-essential file to the codebase which is generated programmatically by existing code.
External caching would be enough (and cleaner).
Hi Dave, I've tried using GALAXY_TEST_DBURI this way with a ready made SQLite database using the current schema (version 117), and it seems to work fine within my TravisCI tool testing [*], and does seem to made things a little faster. https://travis-ci.org/peterjc/galaxy_blast/builds/13309196 https://github.com/peterjc/galaxy_blast/commit/6095b8bcabbefa412259de3cc6bd6...
From that TravisCI build, the pre-test parts of the functional_tests.py now take about 16s, which is OK (but still frustratingly slow if testing a single tool).
In fact, out of the entire TravisCI run time of 14m35s, the setup overhead is now only about 1m30s, it is running the tests which is slow (about 13 minutes). So that seems the best place to focus, e.g. John's idea to switch from faking the HTML controls with twill to using the Galaxy API: https://trello.com/c/MsJctFem/1135-refactor-tool-testing-framework-to-use-ap... Peter [*] For background, see my blog post: http://blastedbio.blogspot.co.uk/2013/09/using-travis-ci-for-testing-galaxy-...
participants (2)
-
Dave Bouvier -
Peter Cock