errors and questions with run_tests.sh and install_and_test_tool_shed_repositories.sh
Hi, I have investigated the testing possibilities within a Galaxy local instance and within a toolshed. I met some difficulties and I have 4 questions listed below. 1- Tests on tools installed in a local Galaxy instance (with run_tests.sh) : It seems that it is only possible to test the tools that have been manually installed (run_test.sh -id ...), not the tools that have been installed through a toolshed. Is that true ? When I try to launch the tests of a toolshed installed tool, I have the following error message : ######### Failure: ValueError (No such test TestForTool_sartools_edger) ... ERROR ====================================================================== ERROR: Failure: ValueError (No such test TestForTool_sartools_edger) ---------------------------------------------------------------------- Traceback (most recent call last): File "/projet/htdocs/galaxy-dev/galaxy-dist/eggs/nose-0.11.1-py2.7.egg/nose/failure.py", line 39, in runTest raise self.exc_class(self.exc_val) ValueError: No such test TestForTool_sartools_edger ---------------------------------------------------------------------- Ran 1 test in 0.002s FAILED (errors=1) ######### 2- Tests on tools installed in a local Galaxy instance (with run_tests.sh) : Is it possible to test tools which have repository or tool dependencies ? I have a tool (manually installed in galaxy-dist/tools/) with a dependence to a datatype installed from a toolshed. The datatype (no_unzip.zip) is defined in the <param> input tag (in <test>) with attribute "ftype". When running run_tests.sh, it seems that the dataype is not known : ######### ====================================================================== ERROR: test_tool_000001 (functional.test_toolbox.TestForTool_sartools_edger) SARTools edgeR ( sartools_edger ) > Test-2 ---------------------------------------------------------------------- Traceback (most recent call last): File "/w/galaxy/galaxy4loraine/galaxy-dist/test/functional/test_toolbox.py", line 268, in test_tool self.do_it( td ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/functional/test_toolbox.py", line 37, in do_it stage_data_in_history( galaxy_interactor, testdef.test_data(), test_history, shed_tool_id ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/base/interactor.py", line 36, in stage_data_in_history upload_waits.append( galaxy_interactor.stage_data_async( test_data, history, shed_tool_id ) ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/base/interactor.py", line 182, in stage_data_async raise Exception(submit_response) Exception: {u'message': {u'type': u'error', u'data': {u'file_type': u"An invalid option was selected for file_type, u'no_unzip.zip', please verify.", u'files_metadata': [u"An invalid option was selected for file_type, u'no_unzip.zip', please verify."]}}} ######### 3- Toolshed Install and Test framework ( install_and_test_tool_shed_repositories.sh) : For my tool, the test fails if I don't explicitely defined the dbkey attribute to "unknown" (dbkey="?") in the <param> input tag (in <test>). The default dbkey value seems to be "hg17" (whereas hg17 is not defined on the server running my toolshed). So I have the following error message if I don't add dbkey="?" in <test> : ######### ====================================================================== ERROR: test_tool_000001 (functional.test_toolbox.TestForTool_galaxy4loraine:9009/repos/lgueguen/sartools_1_0_2/sartools_edger/20150209) SARTools edgeR ( galaxy4loraine:9009/repos/lgueguen/sartools_1_0_2/sartools_edger/20150209 )
Test-2
Traceback (most recent call last): File "/w/galaxy/galaxy4loraine/galaxy-dist/test/functional/test_toolbox.py", line 268, in test_tool self.do_it( td ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/functional/test_toolbox.py", line 37, in do_it stage_data_in_history( galaxy_interactor, testdef.test_data(), test_history, shed_tool_id ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/base/interactor.py", line 36, in stage_data_in_history upload_waits.append( galaxy_interactor.stage_data_async( test_data, history, shed_tool_id ) ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/base/interactor.py", line 182, in stage_data_async raise Exception(submit_response) Exception: {u'message': {u'type': u'error', u'data': {u'dbkey': u"An invalid option was selected for dbkey, u'hg17', please verify."}}} ######### 4- Is it possible to launch the tests against an external server ? That would be very useful to be able run the tests on a production Galaxy instance in order to test the tools in real condition, within the whole environment. Thanks in advance for your answers. Loraine -- Loraine Guéguen Plateforme ABiMS (Analyses and Bioinformatics for Marine Sciences) Service Informatique et Bioinformatique Station Biologique de Roscoff FR 2424 CNRS UPMC Place Georges Teissier CS 90074 29688 Roscoff cedex tel: ++33 (0)2 98 29 56 46 (interne : 415) http://abims.sb-roscoff.fr/ http://www.sb-roscoff.fr/
Hi, I am relatively new to Galaxy so if you get a difference answer from a core developer consider that one. Personally I never got the run_test.sh -id ...) to run but I did use planemo http://planemo.readthedocs.org/en/latest/readme.html With success. It is much easier, does not need the tool to be installed in a local galaxy to test. It does require the tool xml config file See: https://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax And depends on their being a <tests> section. See: https://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax Christian University of Manchester PS. I am not a developer of Planemo just a user so please keep all question on this mailing list ________________________________ From: galaxy-dev [galaxy-dev-bounces@lists.galaxyproject.org] on behalf of Loraine Guéguen [loraine.gueguen@sb-roscoff.fr] Sent: Tuesday, June 02, 2015 9:14 AM To: Galaxy Dev List Subject: [galaxy-dev] errors and questions with run_tests.sh and install_and_test_tool_shed_repositories.sh Hi, I have investigated the testing possibilities within a Galaxy local instance and within a toolshed. I met some difficulties and I have 4 questions listed below. 1- Tests on tools installed in a local Galaxy instance (with run_tests.sh) : It seems that it is only possible to test the tools that have been manually installed (run_test.sh -id ...), not the tools that have been installed through a toolshed. Is that true ? When I try to launch the tests of a toolshed installed tool, I have the following error message : ######### Failure: ValueError (No such test TestForTool_sartools_edger) ... ERROR ====================================================================== ERROR: Failure: ValueError (No such test TestForTool_sartools_edger) ---------------------------------------------------------------------- Traceback (most recent call last): File "/projet/htdocs/galaxy-dev/galaxy-dist/eggs/nose-0.11.1-py2.7.egg/nose/failure.py", line 39, in runTest raise self.exc_class(self.exc_val) ValueError: No such test TestForTool_sartools_edger ---------------------------------------------------------------------- Ran 1 test in 0.002s FAILED (errors=1) ######### 2- Tests on tools installed in a local Galaxy instance (with run_tests.sh) : Is it possible to test tools which have repository or tool dependencies ? I have a tool (manually installed in galaxy-dist/tools/) with a dependence to a datatype installed from a toolshed. The datatype (no_unzip.zip) is defined in the <param> input tag (in <test>) with attribute "ftype". When running run_tests.sh, it seems that the dataype is not known : ######### ====================================================================== ERROR: test_tool_000001 (functional.test_toolbox.TestForTool_sartools_edger) SARTools edgeR ( sartools_edger ) > Test-2 ---------------------------------------------------------------------- Traceback (most recent call last): File "/w/galaxy/galaxy4loraine/galaxy-dist/test/functional/test_toolbox.py", line 268, in test_tool self.do_it( td ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/functional/test_toolbox.py", line 37, in do_it stage_data_in_history( galaxy_interactor, testdef.test_data(), test_history, shed_tool_id ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/base/interactor.py", line 36, in stage_data_in_history upload_waits.append( galaxy_interactor.stage_data_async( test_data, history, shed_tool_id ) ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/base/interactor.py", line 182, in stage_data_async raise Exception(submit_response) Exception: {u'message': {u'type': u'error', u'data': {u'file_type': u"An invalid option was selected for file_type, u'no_unzip.zip', please verify.", u'files_metadata': [u"An invalid option was selected for file_type, u'no_unzip.zip', please verify."]}}} ######### 3- Toolshed Install and Test framework ( install_and_test_tool_shed_repositories.sh) : For my tool, the test fails if I don't explicitely defined the dbkey attribute to "unknown" (dbkey="?") in the <param> input tag (in <test>). The default dbkey value seems to be "hg17" (whereas hg17 is not defined on the server running my toolshed). So I have the following error message if I don't add dbkey="?" in <test> : ######### ====================================================================== ERROR: test_tool_000001 (functional.test_toolbox.TestForTool_galaxy4loraine:9009/repos/lgueguen/sartools_1_0_2/sartools_edger/20150209) SARTools edgeR ( galaxy4loraine:9009/repos/lgueguen/sartools_1_0_2/sartools_edger/20150209 ) > Test-2 ---------------------------------------------------------------------- Traceback (most recent call last): File "/w/galaxy/galaxy4loraine/galaxy-dist/test/functional/test_toolbox.py", line 268, in test_tool self.do_it( td ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/functional/test_toolbox.py", line 37, in do_it stage_data_in_history( galaxy_interactor, testdef.test_data(), test_history, shed_tool_id ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/base/interactor.py", line 36, in stage_data_in_history upload_waits.append( galaxy_interactor.stage_data_async( test_data, history, shed_tool_id ) ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/base/interactor.py", line 182, in stage_data_async raise Exception(submit_response) Exception: {u'message': {u'type': u'error', u'data': {u'dbkey': u"An invalid option was selected for dbkey, u'hg17', please verify."}}} ######### 4- Is it possible to launch the tests against an external server ? That would be very useful to be able run the tests on a production Galaxy instance in order to test the tools in real condition, within the whole environment. Thanks in advance for your answers. Loraine -- Loraine Guéguen Plateforme ABiMS (Analyses and Bioinformatics for Marine Sciences) Service Informatique et Bioinformatique Station Biologique de Roscoff FR 2424 CNRS UPMC Place Georges Teissier CS 90074 29688 Roscoff cedex tel: ++33 (0)2 98 29 56 46 (interne : 415) http://abims.sb-roscoff.fr/ http://www.sb-roscoff.fr/
On Tue, Jun 2, 2015 at 4:14 AM, Loraine Guéguen <loraine.gueguen@sb-roscoff.fr> wrote:
Hi,
I have investigated the testing possibilities within a Galaxy local instance and within a toolshed. I met some difficulties and I have 4 questions listed below.
1- Tests on tools installed in a local Galaxy instance (with run_tests.sh) : It seems that it is only possible to test the tools that have been manually installed (run_test.sh -id ...), not the tools that have been installed through a toolshed. Is that true ? When I try to launch the tests of a toolshed installed tool, I have the following error message :
#########
Failure: ValueError (No such test TestForTool_sartools_edger) ... ERROR ====================================================================== ERROR: Failure: ValueError (No such test TestForTool_sartools_edger) ---------------------------------------------------------------------- Traceback (most recent call last): File "/projet/htdocs/galaxy-dev/galaxy-dist/eggs/nose-0.11.1-py2.7.egg/nose/failure.py", line 39, in runTest raise self.exc_class(self.exc_val) ValueError: No such test TestForTool_sartools_edger
---------------------------------------------------------------------- Ran 1 test in 0.002s FAILED (errors=1)
./run_tests.sh -installed runs the tests of all installed tool shed tools. It might be possible to combine that with -id - for a specific tool but I am not sure. It may be possible to construct that test even if -id doesn't work - but it is probably pretty hacky. Let me know if -installed works for you but not when combined with -id and we can work out the details.
#########
2- Tests on tools installed in a local Galaxy instance (with run_tests.sh) : Is it possible to test tools which have repository or tool dependencies ? I have a tool (manually installed in galaxy-dist/tools/) with a dependence to a datatype installed from a toolshed. The datatype (no_unzip.zip) is defined in the <param> input tag (in <test>) with attribute "ftype". When running run_tests.sh, it seems that the dataype is not known :
#########
====================================================================== ERROR: test_tool_000001 (functional.test_toolbox.TestForTool_sartools_edger) SARTools edgeR ( sartools_edger ) > Test-2 ---------------------------------------------------------------------- Traceback (most recent call last): File "/w/galaxy/galaxy4loraine/galaxy-dist/test/functional/test_toolbox.py", line 268, in test_tool self.do_it( td ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/functional/test_toolbox.py", line 37, in do_it stage_data_in_history( galaxy_interactor, testdef.test_data(), test_history, shed_tool_id ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/base/interactor.py", line 36, in stage_data_in_history upload_waits.append( galaxy_interactor.stage_data_async( test_data, history, shed_tool_id ) ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/base/interactor.py", line 182, in stage_data_async raise Exception(submit_response) Exception: {u'message': {u'type': u'error', u'data': {u'file_type': u"An invalid option was selected for file_type, u'no_unzip.zip', please verify.", u'files_metadata': [u"An invalid option was selected for file_type, u'no_unzip.zip', please verify."]}}}
#########
It is possible it is the case that tool shed datatypes are not loaded unless running with the -installed flag. I would either setup the tests to run locally (manually add datatypes and tools) or run installed tools with -installed.
3- Toolshed Install and Test framework ( install_and_test_tool_shed_repositories.sh) : For my tool, the test fails if I don't explicitely defined the dbkey attribute to "unknown" (dbkey="?") in the <param> input tag (in <test>). The default dbkey value seems to be "hg17" (whereas hg17 is not defined on the server running my toolshed). So I have the following error message if I don't add dbkey="?" in <test> :
#########
====================================================================== ERROR: test_tool_000001 (functional.test_toolbox.TestForTool_galaxy4loraine:9009/repos/lgueguen/sartools_1_0_2/sartools_edger/20150209) SARTools edgeR ( galaxy4loraine:9009/repos/lgueguen/sartools_1_0_2/sartools_edger/20150209 )
Test-2
Traceback (most recent call last): File "/w/galaxy/galaxy4loraine/galaxy-dist/test/functional/test_toolbox.py", line 268, in test_tool self.do_it( td ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/functional/test_toolbox.py", line 37, in do_it stage_data_in_history( galaxy_interactor, testdef.test_data(), test_history, shed_tool_id ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/base/interactor.py", line 36, in stage_data_in_history upload_waits.append( galaxy_interactor.stage_data_async( test_data, history, shed_tool_id ) ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/base/interactor.py", line 182, in stage_data_async raise Exception(submit_response) Exception: {u'message': {u'type': u'error', u'data': {u'dbkey': u"An invalid option was selected for dbkey, u'hg17', please verify."}}}
#########
Well that is frustrating. I am not aware of the internals of that script or why that is happening - but my personal recommendation would be not to use the install and test framework at this point - it seems rather un-maintained. I recently added a subcommnd to planemo called shed_test (http://planemo.readthedocs.org/en/master/commands.html#shed-test-command) - that will install and test a tool shed repository. It probably requires modifying your tool shed development process to use planemo (http://planemo.readthedocs.org/en/latest/publishing.html#shed).
4- Is it possible to launch the tests against an external server ? That would be very useful to be able run the tests on a production Galaxy instance in order to test the tools in real condition, within the whole environment.
This would be great - but as I understand it - it is not possible at this time. The test framework needs local access to some aspects of the Galaxy app. Last year I outlined a Trello card to add this option back to Galaxy (https://trello.com/c/8K86l4Qk) - it was available years ago.
Thanks in advance for your answers.
Loraine
-- Loraine Guéguen
Plateforme ABiMS (Analyses and Bioinformatics for Marine Sciences) Service Informatique et Bioinformatique Station Biologique de Roscoff FR 2424 CNRS UPMC Place Georges Teissier CS 90074 29688 Roscoff cedex tel: ++33 (0)2 98 29 56 46 (interne : 415) http://abims.sb-roscoff.fr/ http://www.sb-roscoff.fr/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Thanks for your answers ! I will try "./run_tests.sh -installed" and see if it can solve the 2 first points. By the way, I didn't know this option (not displayed with run_tests.sh --help) or documented on the wiki if I'm not mistaken. For point 3, I'll try the new "planemo shed_test" subcommand. The subcommand "planemo test" uses the run_tests.sh script, isn't it ? Does the "planemo shed_test" subcommand use all or any part of the Tool Shed Install and Test Framework (https://wiki.galaxyproject.org/AutomatedToolTests) or is it completely different ? Is there any way to launch the tool tests against an external server (point 4) with Planemo ? Loraine Le 02/06/2015 15:16, John Chilton a écrit :
On Tue, Jun 2, 2015 at 4:14 AM, Loraine Guéguen <loraine.gueguen@sb-roscoff.fr> wrote:
Hi,
I have investigated the testing possibilities within a Galaxy local instance and within a toolshed. I met some difficulties and I have 4 questions listed below.
1- Tests on tools installed in a local Galaxy instance (with run_tests.sh) : It seems that it is only possible to test the tools that have been manually installed (run_test.sh -id ...), not the tools that have been installed through a toolshed. Is that true ? When I try to launch the tests of a toolshed installed tool, I have the following error message :
#########
Failure: ValueError (No such test TestForTool_sartools_edger) ... ERROR ====================================================================== ERROR: Failure: ValueError (No such test TestForTool_sartools_edger) ---------------------------------------------------------------------- Traceback (most recent call last): File "/projet/htdocs/galaxy-dev/galaxy-dist/eggs/nose-0.11.1-py2.7.egg/nose/failure.py", line 39, in runTest raise self.exc_class(self.exc_val) ValueError: No such test TestForTool_sartools_edger
---------------------------------------------------------------------- Ran 1 test in 0.002s FAILED (errors=1) ./run_tests.sh -installed runs the tests of all installed tool shed tools. It might be possible to combine that with -id - for a specific tool but I am not sure. It may be possible to construct that test even if -id doesn't work - but it is probably pretty hacky. Let me know if -installed works for you but not when combined with -id and we can work out the details. #########
2- Tests on tools installed in a local Galaxy instance (with run_tests.sh) : Is it possible to test tools which have repository or tool dependencies ? I have a tool (manually installed in galaxy-dist/tools/) with a dependence to a datatype installed from a toolshed. The datatype (no_unzip.zip) is defined in the <param> input tag (in <test>) with attribute "ftype". When running run_tests.sh, it seems that the dataype is not known :
#########
====================================================================== ERROR: test_tool_000001 (functional.test_toolbox.TestForTool_sartools_edger) SARTools edgeR ( sartools_edger ) > Test-2 ---------------------------------------------------------------------- Traceback (most recent call last): File "/w/galaxy/galaxy4loraine/galaxy-dist/test/functional/test_toolbox.py", line 268, in test_tool self.do_it( td ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/functional/test_toolbox.py", line 37, in do_it stage_data_in_history( galaxy_interactor, testdef.test_data(), test_history, shed_tool_id ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/base/interactor.py", line 36, in stage_data_in_history upload_waits.append( galaxy_interactor.stage_data_async( test_data, history, shed_tool_id ) ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/base/interactor.py", line 182, in stage_data_async raise Exception(submit_response) Exception: {u'message': {u'type': u'error', u'data': {u'file_type': u"An invalid option was selected for file_type, u'no_unzip.zip', please verify.", u'files_metadata': [u"An invalid option was selected for file_type, u'no_unzip.zip', please verify."]}}}
######### It is possible it is the case that tool shed datatypes are not loaded unless running with the -installed flag. I would either setup the tests to run locally (manually add datatypes and tools) or run installed tools with -installed.
3- Toolshed Install and Test framework ( install_and_test_tool_shed_repositories.sh) : For my tool, the test fails if I don't explicitely defined the dbkey attribute to "unknown" (dbkey="?") in the <param> input tag (in <test>). The default dbkey value seems to be "hg17" (whereas hg17 is not defined on the server running my toolshed). So I have the following error message if I don't add dbkey="?" in <test> :
#########
====================================================================== ERROR: test_tool_000001 (functional.test_toolbox.TestForTool_galaxy4loraine:9009/repos/lgueguen/sartools_1_0_2/sartools_edger/20150209) SARTools edgeR ( galaxy4loraine:9009/repos/lgueguen/sartools_1_0_2/sartools_edger/20150209 )
Test-2
Traceback (most recent call last): File "/w/galaxy/galaxy4loraine/galaxy-dist/test/functional/test_toolbox.py", line 268, in test_tool self.do_it( td ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/functional/test_toolbox.py", line 37, in do_it stage_data_in_history( galaxy_interactor, testdef.test_data(), test_history, shed_tool_id ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/base/interactor.py", line 36, in stage_data_in_history upload_waits.append( galaxy_interactor.stage_data_async( test_data, history, shed_tool_id ) ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/base/interactor.py", line 182, in stage_data_async raise Exception(submit_response) Exception: {u'message': {u'type': u'error', u'data': {u'dbkey': u"An invalid option was selected for dbkey, u'hg17', please verify."}}}
#########
Well that is frustrating. I am not aware of the internals of that script or why that is happening - but my personal recommendation would be not to use the install and test framework at this point - it seems rather un-maintained. I recently added a subcommnd to planemo called shed_test (http://planemo.readthedocs.org/en/master/commands.html#shed-test-command) - that will install and test a tool shed repository. It probably requires modifying your tool shed development process to use planemo (http://planemo.readthedocs.org/en/latest/publishing.html#shed).
4- Is it possible to launch the tests against an external server ? That would be very useful to be able run the tests on a production Galaxy instance in order to test the tools in real condition, within the whole environment.
This would be great - but as I understand it - it is not possible at this time. The test framework needs local access to some aspects of the Galaxy app. Last year I outlined a Trello card to add this option back to Galaxy (https://trello.com/c/8K86l4Qk) - it was available years ago.
Thanks in advance for your answers.
Loraine
-- Loraine Guéguen
Plateforme ABiMS (Analyses and Bioinformatics for Marine Sciences) Service Informatique et Bioinformatique Station Biologique de Roscoff FR 2424 CNRS UPMC Place Georges Teissier CS 90074 29688 Roscoff cedex tel: ++33 (0)2 98 29 56 46 (interne : 415) http://abims.sb-roscoff.fr/ http://www.sb-roscoff.fr/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
-- Loraine Guéguen Plateforme ABiMS (Analyses and Bioinformatics for Marine Sciences) Service Informatique et Bioinformatique Station Biologique de Roscoff FR 2424 CNRS UPMC Place Georges Teissier 29680 Roscoff tel: ++33 (0)2 98 29 25 50 http://abims.sb-roscoff.fr/ http://www.sb-roscoff.fr/
On Mon, Jun 8, 2015 at 6:30 AM, Loraine Guéguen <loraine.gueguen@sb-roscoff.fr> wrote:
Thanks for your answers !
I will try "./run_tests.sh -installed" and see if it can solve the 2 first points. By the way, I didn't know this option (not displayed with run_tests.sh --help) or documented on the wiki if I'm not mistaken.
I have opened a PR correcting this oversight - https://github.com/galaxyproject/galaxy/pull/323 - thanks for the catch.
For point 3, I'll try the new "planemo shed_test" subcommand. The subcommand "planemo test" uses the run_tests.sh script, isn't it ? Does the "planemo shed_test" subcommand use all or any part of the Tool Shed Install and Test Framework (https://wiki.galaxyproject.org/AutomatedToolTests) or is it completely different ?
It doesn't leverage the install and test framework - it uses run_tests.sh (with the -installed option) but it sets many extra things up. It may well not work with installed datatypes for the same reason - I am not sure. I've created an issue and will try to look into this (https://github.com/galaxyproject/planemo/issues/226).
Is there any way to launch the tool tests against an external server (point 4) with Planemo ?
Nope - not at this time. Thanks for the detailed e-mail and follow up. I hope this helps. There are a lot of discussions ongoing about datatypes and whether they belong in the tool shed. I generally discourage putting them there - I think the framework is sort of broken by design for datatypes. Certain team and community members disagree - hopefully they will help and put some effort into making datatypes in the Tool Shed more usable :). -John
Loraine
Le 02/06/2015 15:16, John Chilton a écrit :
On Tue, Jun 2, 2015 at 4:14 AM, Loraine Guéguen <loraine.gueguen@sb-roscoff.fr> wrote:
Hi,
I have investigated the testing possibilities within a Galaxy local instance and within a toolshed. I met some difficulties and I have 4 questions listed below.
1- Tests on tools installed in a local Galaxy instance (with run_tests.sh) : It seems that it is only possible to test the tools that have been manually installed (run_test.sh -id ...), not the tools that have been installed through a toolshed. Is that true ? When I try to launch the tests of a toolshed installed tool, I have the following error message :
#########
Failure: ValueError (No such test TestForTool_sartools_edger) ... ERROR ====================================================================== ERROR: Failure: ValueError (No such test TestForTool_sartools_edger) ---------------------------------------------------------------------- Traceback (most recent call last): File
"/projet/htdocs/galaxy-dev/galaxy-dist/eggs/nose-0.11.1-py2.7.egg/nose/failure.py", line 39, in runTest raise self.exc_class(self.exc_val) ValueError: No such test TestForTool_sartools_edger
---------------------------------------------------------------------- Ran 1 test in 0.002s FAILED (errors=1)
./run_tests.sh -installed runs the tests of all installed tool shed tools. It might be possible to combine that with -id - for a specific tool but I am not sure. It may be possible to construct that test even if -id doesn't work - but it is probably pretty hacky. Let me know if -installed works for you but not when combined with -id and we can work out the details.
#########
2- Tests on tools installed in a local Galaxy instance (with run_tests.sh) : Is it possible to test tools which have repository or tool dependencies ? I have a tool (manually installed in galaxy-dist/tools/) with a dependence to a datatype installed from a toolshed. The datatype (no_unzip.zip) is defined in the <param> input tag (in <test>) with attribute "ftype". When running run_tests.sh, it seems that the dataype is not known :
#########
====================================================================== ERROR: test_tool_000001 (functional.test_toolbox.TestForTool_sartools_edger) SARTools edgeR ( sartools_edger ) > Test-2 ---------------------------------------------------------------------- Traceback (most recent call last): File "/w/galaxy/galaxy4loraine/galaxy-dist/test/functional/test_toolbox.py", line 268, in test_tool self.do_it( td ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/functional/test_toolbox.py", line 37, in do_it stage_data_in_history( galaxy_interactor, testdef.test_data(), test_history, shed_tool_id ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/base/interactor.py", line 36, in stage_data_in_history upload_waits.append( galaxy_interactor.stage_data_async( test_data, history, shed_tool_id ) ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/base/interactor.py", line 182, in stage_data_async raise Exception(submit_response) Exception: {u'message': {u'type': u'error', u'data': {u'file_type': u"An invalid option was selected for file_type, u'no_unzip.zip', please verify.", u'files_metadata': [u"An invalid option was selected for file_type, u'no_unzip.zip', please verify."]}}}
#########
It is possible it is the case that tool shed datatypes are not loaded unless running with the -installed flag. I would either setup the tests to run locally (manually add datatypes and tools) or run installed tools with -installed.
3- Toolshed Install and Test framework ( install_and_test_tool_shed_repositories.sh) : For my tool, the test fails if I don't explicitely defined the dbkey attribute to "unknown" (dbkey="?") in the <param> input tag (in <test>). The default dbkey value seems to be "hg17" (whereas hg17 is not defined on the server running my toolshed). So I have the following error message if I don't add dbkey="?" in <test> :
#########
====================================================================== ERROR: test_tool_000001
(functional.test_toolbox.TestForTool_galaxy4loraine:9009/repos/lgueguen/sartools_1_0_2/sartools_edger/20150209) SARTools edgeR ( galaxy4loraine:9009/repos/lgueguen/sartools_1_0_2/sartools_edger/20150209 )
Test-2
---------------------------------------------------------------------- Traceback (most recent call last): File "/w/galaxy/galaxy4loraine/galaxy-dist/test/functional/test_toolbox.py", line 268, in test_tool self.do_it( td ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/functional/test_toolbox.py", line 37, in do_it stage_data_in_history( galaxy_interactor, testdef.test_data(), test_history, shed_tool_id ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/base/interactor.py", line 36, in stage_data_in_history upload_waits.append( galaxy_interactor.stage_data_async( test_data, history, shed_tool_id ) ) File "/w/galaxy/galaxy4loraine/galaxy-dist/test/base/interactor.py", line 182, in stage_data_async raise Exception(submit_response) Exception: {u'message': {u'type': u'error', u'data': {u'dbkey': u"An invalid option was selected for dbkey, u'hg17', please verify."}}}
#########
Well that is frustrating. I am not aware of the internals of that script or why that is happening - but my personal recommendation would be not to use the install and test framework at this point - it seems rather un-maintained. I recently added a subcommnd to planemo called shed_test (http://planemo.readthedocs.org/en/master/commands.html#shed-test-command) - that will install and test a tool shed repository. It probably requires modifying your tool shed development process to use planemo (http://planemo.readthedocs.org/en/latest/publishing.html#shed).
4- Is it possible to launch the tests against an external server ? That would be very useful to be able run the tests on a production Galaxy instance in order to test the tools in real condition, within the whole environment.
This would be great - but as I understand it - it is not possible at this time. The test framework needs local access to some aspects of the Galaxy app. Last year I outlined a Trello card to add this option back to Galaxy (https://trello.com/c/8K86l4Qk) - it was available years ago.
Thanks in advance for your answers.
Loraine
-- Loraine Guéguen
Plateforme ABiMS (Analyses and Bioinformatics for Marine Sciences) Service Informatique et Bioinformatique Station Biologique de Roscoff FR 2424 CNRS UPMC Place Georges Teissier CS 90074 29688 Roscoff cedex tel: ++33 (0)2 98 29 56 46 (interne : 415) http://abims.sb-roscoff.fr/ http://www.sb-roscoff.fr/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
-- Loraine Guéguen
Plateforme ABiMS (Analyses and Bioinformatics for Marine Sciences) Service Informatique et Bioinformatique Station Biologique de Roscoff FR 2424 CNRS UPMC Place Georges Teissier 29680 Roscoff tel: ++33 (0)2 98 29 25 50 http://abims.sb-roscoff.fr/ http://www.sb-roscoff.fr/
participants (3)
-
Christian Brenninkmeijer -
John Chilton -
Loraine Guéguen