Thanks for the quick responses John and Peter! I feel fortified to carry on. -Jeltje On Wed, Jan 7, 2015 at 9:03 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Thu, Jan 8, 2015 at 3:27 AM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote:
Hi,
I'm trying to find out what the best way is to see if my functional tests work as expected, and I'm also wondering when these tests are (supposed to be) run. Here are the details:
Following instructions at https://wiki.galaxyproject.org/Admin/Tools/AddToolTutorial, I created a little toolExample tool under galaxy-dist/tools/myTools. I also created test input and output in a test-data directory and added a test to the toolExample.xml file: <tests> <test> <param name="input" value="testinput.fa"/> <output name="out_file1" file="testoutput.txt"/> </test> </tests>
And yes, the files are named correctly: tools/myTools/test-data/testinput.fa tools/myTools/test-data/testoutput.txt
If you are manually installing the tool directly under the folder galaxy-dist/tools/myTools then you should put your test files under galaxy-dist/test-data (this is how all the tools used to work back before the ToolShed).
If you want to do the tool installation via a ToolShed, things are different (more complicated for the tool development side, but it becomes easier for the end user to install things).
I added the new tool to config/tools_conf.xml AND to config/tools_conf.xml.sample. Question #1: Why is a .sample file used by run_functional_tests.sh? I thought with .sample extensions were meant as examples for creating your own versions, I'm really surprised that any script would want to access these.
Historically (for reasons unclear to me), the tools for testing were listed in galaxy-dist/tools_conf.xml.sample however that was changed (about a year ago? I forget now) to simply use galaxy-dist/tools_conf.xml (the master tool listing as used before the ToolShed existed).
You might be using different settings in your main ini file, but on our setup it is galaxy-dist/tools_conf.xml and not galaxy-dist/config/tools_conf.xml (which seems to be in use on your system given the new tool showed up on restarting Galaxy):
I restarted galaxy to verify that the new tool showed up, then checked that I could test it:
./run_functional_tests.sh -list | grep fa_gc_content_1 It shows up in the list of tools. So then I tested it:
./run_functional_tests.sh -id fa_gc_content_1
Good - although run_functional_tests.sh is now obsolete and just a backward compatible alias for run_tests.sh.
If you read this in some (out of date) documentation, please say where so we can update it.
This gives IOError: [Errno 2] No such file or directory:
'/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/test-data/testinput.fa'
So it looks like at some point the testinput should have been copied to galaxy-dist/test-data but wasn't.
No, as noted above, you have to put the test files there by hand (as long as you are deliberately doing all your tool installation/development avoiding the ToolShed, which is quite a common approach if you have a separate Galaxy instance for development).
Question # 2: Is there some setting that makes this test input (and output) get copied to the correct location, or do I have to do this manually?
No, unless you go down the ToolShed route, do this manually.
I also tried testing by uploading the tool to the testtoolshed (as jtest1). I deliberatly put the wrong value in the testoutput.txt file because I'd like to see a test fail before I'll accept that it succeeds. It appears the testinput does not get tested on upload, because I got no automated test results, and no error messages anywhere. Is that expected behavior? I know the tools get tested every so often, I'm just surprised it doesn't happen on upload. The tool could then also be easily installed to my galaxy distribution, without errors in my log or on the browser pages.
The main and Test Tool Shed should now be running the tests every 48 hours or so - this service had been unavailable over the Christmas/New Year period but should be OK now.
Question #3: At what point in creating or downloading a repository are these tests run?
Never, assuming I understand your question.
If you mean installing a tool from the ToolShed into a Galaxy instance via the Galaxy administrator options, then I think after installation any tests should be run automatically.
And lastly, #4: is either of these the best way to test my tests, or did I overlook something?
Thank you!
-Jeltje
While developing a new tool (or improving one), I personally use a development Galaxy server (not the one our users run all their real jobs on), and install the tools manually (as above), and then test via run_tests.sh -id my_tool_id.
This is complemented by a TravisCI setup, see:
http://blastedbio.blogspot.jp/2013/09/using-travis-ci-for-testing-galaxy-too...
If that works, I upload the tool to the TestToolShed, and wait for those test results. If that all works and I am ready to make a public release, then I upload to the main ToolShed.
I talked about this (in the context of the NCBI BLAST+ wrappers) at the GCC2014 conference, slides here: http://www.slideshare.net/pjacock/galaxy-blast-gcc2014
There is even a video of my talk (and most of the others) on the Galaxy website here: https://wiki.galaxyproject.org/Events/GCC2014/Abstracts
I hope that helped. There is a work in progress "Best Practices" being worked on here: https://wiki.galaxyproject.org/Tools/BestPractices
Regards,
Peter