How to test my functional tests?
Hi, I'm trying to find out what the best way is to see if my functional tests work as expected, and I'm also wondering when these tests are (supposed to be) run. Here are the details: Following instructions at https://wiki.galaxyproject.org/Admin/Tools/AddToolTutorial, I created a little toolExample tool under galaxy-dist/tools/myTools. I also created test input and output in a test-data directory and added a test to the toolExample.xml file: <tests> <test> <param name="input" value="testinput.fa"/> <output name="out_file1" file="testoutput.txt"/> </test> </tests> And yes, the files are named correctly: tools/myTools/test-data/testinput.fa tools/myTools/test-data/testoutput.txt I added the new tool to config/tools_conf.xml AND to config/tools_conf.xml.sample. Question #1: Why is a .sample file used by run_functional_tests.sh? I thought with .sample extensions were meant as examples for creating your own versions, I'm really surprised that any script would want to access these. I restarted galaxy to verify that the new tool showed up, then checked that I could test it: ./run_functional_tests.sh -list | grep fa_gc_content_1 It shows up in the list of tools. So then I tested it: ./run_functional_tests.sh -id fa_gc_content_1 This gives IOError: [Errno 2] No such file or directory: '/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/test-data/testinput.fa' So it looks like at some point the testinput should have been copied to galaxy-dist/test-data but wasn't. Question # 2: Is there some setting that makes this test input (and output) get copied to the correct location, or do I have to do this manually? I also tried testing by uploading the tool to the testtoolshed (as jtest1). I deliberatly put the wrong value in the testoutput.txt file because I'd like to see a test fail before I'll accept that it succeeds. It appears the testinput does not get tested on upload, because I got no automated test results, and no error messages anywhere. Is that expected behavior? I know the tools get tested every so often, I'm just surprised it doesn't happen on upload. The tool could then also be easily installed to my galaxy distribution, without errors in my log or on the browser pages. Question #3: At what point in creating or downloading a repository are these tests run? And lastly, #4: is either of these the best way to test my tests, or did I overlook something? Thank you! -Jeltje
On Thu, Jan 8, 2015 at 3:27 AM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote:
Hi,
I'm trying to find out what the best way is to see if my functional tests work as expected, and I'm also wondering when these tests are (supposed to be) run. Here are the details:
Following instructions at https://wiki.galaxyproject.org/Admin/Tools/AddToolTutorial, I created a little toolExample tool under galaxy-dist/tools/myTools. I also created test input and output in a test-data directory and added a test to the toolExample.xml file: <tests> <test> <param name="input" value="testinput.fa"/> <output name="out_file1" file="testoutput.txt"/> </test> </tests>
And yes, the files are named correctly: tools/myTools/test-data/testinput.fa tools/myTools/test-data/testoutput.txt
If you are manually installing the tool directly under the folder galaxy-dist/tools/myTools then you should put your test files under galaxy-dist/test-data (this is how all the tools used to work back before the ToolShed). If you want to do the tool installation via a ToolShed, things are different (more complicated for the tool development side, but it becomes easier for the end user to install things).
I added the new tool to config/tools_conf.xml AND to config/tools_conf.xml.sample. Question #1: Why is a .sample file used by run_functional_tests.sh? I thought with .sample extensions were meant as examples for creating your own versions, I'm really surprised that any script would want to access these.
Historically (for reasons unclear to me), the tools for testing were listed in galaxy-dist/tools_conf.xml.sample however that was changed (about a year ago? I forget now) to simply use galaxy-dist/tools_conf.xml (the master tool listing as used before the ToolShed existed). You might be using different settings in your main ini file, but on our setup it is galaxy-dist/tools_conf.xml and not galaxy-dist/config/tools_conf.xml (which seems to be in use on your system given the new tool showed up on restarting Galaxy):
I restarted galaxy to verify that the new tool showed up, then checked that I could test it:
./run_functional_tests.sh -list | grep fa_gc_content_1 It shows up in the list of tools. So then I tested it:
./run_functional_tests.sh -id fa_gc_content_1
Good - although run_functional_tests.sh is now obsolete and just a backward compatible alias for run_tests.sh. If you read this in some (out of date) documentation, please say where so we can update it.
This gives IOError: [Errno 2] No such file or directory: '/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/test-data/testinput.fa'
So it looks like at some point the testinput should have been copied to galaxy-dist/test-data but wasn't.
No, as noted above, you have to put the test files there by hand (as long as you are deliberately doing all your tool installation/development avoiding the ToolShed, which is quite a common approach if you have a separate Galaxy instance for development).
Question # 2: Is there some setting that makes this test input (and output) get copied to the correct location, or do I have to do this manually?
No, unless you go down the ToolShed route, do this manually.
I also tried testing by uploading the tool to the testtoolshed (as jtest1). I deliberatly put the wrong value in the testoutput.txt file because I'd like to see a test fail before I'll accept that it succeeds. It appears the testinput does not get tested on upload, because I got no automated test results, and no error messages anywhere. Is that expected behavior? I know the tools get tested every so often, I'm just surprised it doesn't happen on upload. The tool could then also be easily installed to my galaxy distribution, without errors in my log or on the browser pages.
The main and Test Tool Shed should now be running the tests every 48 hours or so - this service had been unavailable over the Christmas/New Year period but should be OK now.
Question #3: At what point in creating or downloading a repository are these tests run?
Never, assuming I understand your question. If you mean installing a tool from the ToolShed into a Galaxy instance via the Galaxy administrator options, then I think after installation any tests should be run automatically.
And lastly, #4: is either of these the best way to test my tests, or did I overlook something?
Thank you!
-Jeltje
While developing a new tool (or improving one), I personally use a development Galaxy server (not the one our users run all their real jobs on), and install the tools manually (as above), and then test via run_tests.sh -id my_tool_id. This is complemented by a TravisCI setup, see: http://blastedbio.blogspot.jp/2013/09/using-travis-ci-for-testing-galaxy-too... If that works, I upload the tool to the TestToolShed, and wait for those test results. If that all works and I am ready to make a public release, then I upload to the main ToolShed. I talked about this (in the context of the NCBI BLAST+ wrappers) at the GCC2014 conference, slides here: http://www.slideshare.net/pjacock/galaxy-blast-gcc2014 There is even a video of my talk (and most of the others) on the Galaxy website here: https://wiki.galaxyproject.org/Events/GCC2014/Abstracts I hope that helped. There is a work in progress "Best Practices" being worked on here: https://wiki.galaxyproject.org/Tools/BestPractices Regards, Peter
In addition to the two paths Peter laid out for testing tools: 1 - Embedding them into a Galaxy instance, registering the tools in $GALAXY_ROOT/config/tool_conf.xml, and populating $GALAXY_ROOT/test-data and Galaxy's index data as needed, and then running ./run_tests.sh -id <tool_id> 2 - Publishing them to the test toolshed along with dependencies and waiting for the tool shed nightly framework to register test results. There is a newer, light-weight alternative where you can use your local system dependencies but keep the tools separated out into their own repository and not worry about how Galaxy is configured during testing. This is using the new tool Planemo - there is a quick start here (http://planemo.readthedocs.org/en/latest/readme.html#quick-start). Installing Planemo (assuming you have Linuxbrew or Homebrew) is as easy % brew tap galaxyproject/tap % brew install planemo Then one can run tool tests using % planemo t --galaxy_root=/path/to/galaxy example_tool.xml Or if you don't have a Galaxy on the system % planemo t --install_galaxy example_tool.xml Planemo runs Galaxy with dynamic settings so there is no need to modify Galaxy's tool_conf.xml or need to to know the id of the tool. Planemo also produces prettier, more informative reports of tool test results (e.g. http://bit.ly/planemotestreportexample) and also offers linting functionality so you can catch more bugs quickly before you even need to run tests. More information at: http://planemo.readthedocs.org/ https://github.com/galaxyproject/planemo Hope this helps, -John On Thu, Jan 8, 2015 at 12:03 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Thu, Jan 8, 2015 at 3:27 AM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote:
Hi,
I'm trying to find out what the best way is to see if my functional tests work as expected, and I'm also wondering when these tests are (supposed to be) run. Here are the details:
Following instructions at https://wiki.galaxyproject.org/Admin/Tools/AddToolTutorial, I created a little toolExample tool under galaxy-dist/tools/myTools. I also created test input and output in a test-data directory and added a test to the toolExample.xml file: <tests> <test> <param name="input" value="testinput.fa"/> <output name="out_file1" file="testoutput.txt"/> </test> </tests>
And yes, the files are named correctly: tools/myTools/test-data/testinput.fa tools/myTools/test-data/testoutput.txt
If you are manually installing the tool directly under the folder galaxy-dist/tools/myTools then you should put your test files under galaxy-dist/test-data (this is how all the tools used to work back before the ToolShed).
If you want to do the tool installation via a ToolShed, things are different (more complicated for the tool development side, but it becomes easier for the end user to install things).
I added the new tool to config/tools_conf.xml AND to config/tools_conf.xml.sample. Question #1: Why is a .sample file used by run_functional_tests.sh? I thought with .sample extensions were meant as examples for creating your own versions, I'm really surprised that any script would want to access these.
Historically (for reasons unclear to me), the tools for testing were listed in galaxy-dist/tools_conf.xml.sample however that was changed (about a year ago? I forget now) to simply use galaxy-dist/tools_conf.xml (the master tool listing as used before the ToolShed existed).
You might be using different settings in your main ini file, but on our setup it is galaxy-dist/tools_conf.xml and not galaxy-dist/config/tools_conf.xml (which seems to be in use on your system given the new tool showed up on restarting Galaxy):
I restarted galaxy to verify that the new tool showed up, then checked that I could test it:
./run_functional_tests.sh -list | grep fa_gc_content_1 It shows up in the list of tools. So then I tested it:
./run_functional_tests.sh -id fa_gc_content_1
Good - although run_functional_tests.sh is now obsolete and just a backward compatible alias for run_tests.sh.
If you read this in some (out of date) documentation, please say where so we can update it.
This gives IOError: [Errno 2] No such file or directory: '/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/test-data/testinput.fa'
So it looks like at some point the testinput should have been copied to galaxy-dist/test-data but wasn't.
No, as noted above, you have to put the test files there by hand (as long as you are deliberately doing all your tool installation/development avoiding the ToolShed, which is quite a common approach if you have a separate Galaxy instance for development).
Question # 2: Is there some setting that makes this test input (and output) get copied to the correct location, or do I have to do this manually?
No, unless you go down the ToolShed route, do this manually.
I also tried testing by uploading the tool to the testtoolshed (as jtest1). I deliberatly put the wrong value in the testoutput.txt file because I'd like to see a test fail before I'll accept that it succeeds. It appears the testinput does not get tested on upload, because I got no automated test results, and no error messages anywhere. Is that expected behavior? I know the tools get tested every so often, I'm just surprised it doesn't happen on upload. The tool could then also be easily installed to my galaxy distribution, without errors in my log or on the browser pages.
The main and Test Tool Shed should now be running the tests every 48 hours or so - this service had been unavailable over the Christmas/New Year period but should be OK now.
Question #3: At what point in creating or downloading a repository are these tests run?
Never, assuming I understand your question.
If you mean installing a tool from the ToolShed into a Galaxy instance via the Galaxy administrator options, then I think after installation any tests should be run automatically.
And lastly, #4: is either of these the best way to test my tests, or did I overlook something?
Thank you!
-Jeltje
While developing a new tool (or improving one), I personally use a development Galaxy server (not the one our users run all their real jobs on), and install the tools manually (as above), and then test via run_tests.sh -id my_tool_id.
This is complemented by a TravisCI setup, see: http://blastedbio.blogspot.jp/2013/09/using-travis-ci-for-testing-galaxy-too...
If that works, I upload the tool to the TestToolShed, and wait for those test results. If that all works and I am ready to make a public release, then I upload to the main ToolShed.
I talked about this (in the context of the NCBI BLAST+ wrappers) at the GCC2014 conference, slides here: http://www.slideshare.net/pjacock/galaxy-blast-gcc2014
There is even a video of my talk (and most of the others) on the Galaxy website here: https://wiki.galaxyproject.org/Events/GCC2014/Abstracts
I hope that helped. There is a work in progress "Best Practices" being worked on here: https://wiki.galaxyproject.org/Tools/BestPractices
Regards,
Peter ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Thanks for the quick responses John and Peter! I feel fortified to carry on. -Jeltje On Wed, Jan 7, 2015 at 9:03 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Thu, Jan 8, 2015 at 3:27 AM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote:
Hi,
I'm trying to find out what the best way is to see if my functional tests work as expected, and I'm also wondering when these tests are (supposed to be) run. Here are the details:
Following instructions at https://wiki.galaxyproject.org/Admin/Tools/AddToolTutorial, I created a little toolExample tool under galaxy-dist/tools/myTools. I also created test input and output in a test-data directory and added a test to the toolExample.xml file: <tests> <test> <param name="input" value="testinput.fa"/> <output name="out_file1" file="testoutput.txt"/> </test> </tests>
And yes, the files are named correctly: tools/myTools/test-data/testinput.fa tools/myTools/test-data/testoutput.txt
If you are manually installing the tool directly under the folder galaxy-dist/tools/myTools then you should put your test files under galaxy-dist/test-data (this is how all the tools used to work back before the ToolShed).
If you want to do the tool installation via a ToolShed, things are different (more complicated for the tool development side, but it becomes easier for the end user to install things).
I added the new tool to config/tools_conf.xml AND to config/tools_conf.xml.sample. Question #1: Why is a .sample file used by run_functional_tests.sh? I thought with .sample extensions were meant as examples for creating your own versions, I'm really surprised that any script would want to access these.
Historically (for reasons unclear to me), the tools for testing were listed in galaxy-dist/tools_conf.xml.sample however that was changed (about a year ago? I forget now) to simply use galaxy-dist/tools_conf.xml (the master tool listing as used before the ToolShed existed).
You might be using different settings in your main ini file, but on our setup it is galaxy-dist/tools_conf.xml and not galaxy-dist/config/tools_conf.xml (which seems to be in use on your system given the new tool showed up on restarting Galaxy):
I restarted galaxy to verify that the new tool showed up, then checked that I could test it:
./run_functional_tests.sh -list | grep fa_gc_content_1 It shows up in the list of tools. So then I tested it:
./run_functional_tests.sh -id fa_gc_content_1
Good - although run_functional_tests.sh is now obsolete and just a backward compatible alias for run_tests.sh.
If you read this in some (out of date) documentation, please say where so we can update it.
This gives IOError: [Errno 2] No such file or directory:
'/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/test-data/testinput.fa'
So it looks like at some point the testinput should have been copied to galaxy-dist/test-data but wasn't.
No, as noted above, you have to put the test files there by hand (as long as you are deliberately doing all your tool installation/development avoiding the ToolShed, which is quite a common approach if you have a separate Galaxy instance for development).
Question # 2: Is there some setting that makes this test input (and output) get copied to the correct location, or do I have to do this manually?
No, unless you go down the ToolShed route, do this manually.
I also tried testing by uploading the tool to the testtoolshed (as jtest1). I deliberatly put the wrong value in the testoutput.txt file because I'd like to see a test fail before I'll accept that it succeeds. It appears the testinput does not get tested on upload, because I got no automated test results, and no error messages anywhere. Is that expected behavior? I know the tools get tested every so often, I'm just surprised it doesn't happen on upload. The tool could then also be easily installed to my galaxy distribution, without errors in my log or on the browser pages.
The main and Test Tool Shed should now be running the tests every 48 hours or so - this service had been unavailable over the Christmas/New Year period but should be OK now.
Question #3: At what point in creating or downloading a repository are these tests run?
Never, assuming I understand your question.
If you mean installing a tool from the ToolShed into a Galaxy instance via the Galaxy administrator options, then I think after installation any tests should be run automatically.
And lastly, #4: is either of these the best way to test my tests, or did I overlook something?
Thank you!
-Jeltje
While developing a new tool (or improving one), I personally use a development Galaxy server (not the one our users run all their real jobs on), and install the tools manually (as above), and then test via run_tests.sh -id my_tool_id.
This is complemented by a TravisCI setup, see:
http://blastedbio.blogspot.jp/2013/09/using-travis-ci-for-testing-galaxy-too...
If that works, I upload the tool to the TestToolShed, and wait for those test results. If that all works and I am ready to make a public release, then I upload to the main ToolShed.
I talked about this (in the context of the NCBI BLAST+ wrappers) at the GCC2014 conference, slides here: http://www.slideshare.net/pjacock/galaxy-blast-gcc2014
There is even a video of my talk (and most of the others) on the Galaxy website here: https://wiki.galaxyproject.org/Events/GCC2014/Abstracts
I hope that helped. There is a work in progress "Best Practices" being worked on here: https://wiki.galaxyproject.org/Tools/BestPractices
Regards,
Peter
participants (3)
-
Jeltje van Baren -
John Chilton -
Peter Cock