On Thu, Jan 8, 2015 at 3:27 AM, Jeltje van Baren
<jeltje.van.baren@gmail.com> wrote:
> Hi,
>
> I'm trying to find out what the best way is to see if my functional tests
> work as expected, and I'm also wondering when these tests are (supposed to
> be) run. Here are the details:
>
> Following instructions at
> https://wiki.galaxyproject.org/Admin/Tools/AddToolTutorial, I created a
> little toolExample tool under galaxy-dist/tools/myTools. I also created test
> input and output in a test-data directory and added a test to the
> toolExample.xml file:
>   <tests>
>     <test>
>       <param name="input" value="testinput.fa"/>
>       <output name="out_file1" file="testoutput.txt"/>
>     </test>
>   </tests>
>
> And yes, the files are named correctly:
> tools/myTools/test-data/testinput.fa
> tools/myTools/test-data/testoutput.txt

If you are manually installing the tool directly under the folder
galaxy-dist/tools/myTools then you should put your test
files under galaxy-dist/test-data (this is how all the tools
used to work back before the ToolShed).

If you want to do the tool installation via a ToolShed,
things are different (more complicated for the tool
development side, but it becomes easier for the end
user to install things).

> I added the new tool to config/tools_conf.xml AND to
> config/tools_conf.xml.sample.
> Question #1: Why is a .sample file used by run_functional_tests.sh? I
> thought with .sample extensions were meant as examples for creating your own
> versions, I'm really surprised that any script would want to access these.

Historically (for reasons unclear to me), the tools for testing
were listed in galaxy-dist/tools_conf.xml.sample however
that was changed (about a year ago? I forget now) to simply
use galaxy-dist/tools_conf.xml (the master tool listing as used
before the ToolShed existed).

You might be using different settings in your main ini file,
but on our setup it is galaxy-dist/tools_conf.xml and not
galaxy-dist/config/tools_conf.xml (which seems to be in
use on your system given the new tool showed up on
restarting Galaxy):

> I restarted galaxy to verify that the new tool showed up,
> then checked that I could test it:
>
> ./run_functional_tests.sh -list | grep fa_gc_content_1
> It shows up in the list of tools. So then I tested it:
>
> ./run_functional_tests.sh -id fa_gc_content_1

Good - although run_functional_tests.sh is now obsolete
and just a backward compatible alias for run_tests.sh.

If you read this in some (out of date) documentation,
please say where so we can update it.

> This gives
> IOError: [Errno 2] No such file or directory:
> '/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/test-data/testinput.fa'
>
> So it looks like at some point the testinput should have been copied to
> galaxy-dist/test-data but wasn't.

No, as noted above, you have to put the test files there
by hand (as long as you are deliberately doing all your
tool installation/development avoiding the ToolShed,
which is quite a common approach if you have a
separate Galaxy instance for development).

> Question # 2: Is there some setting that makes this test input (and output)
> get copied to the correct location, or do I have to do this manually?

No, unless you go down the ToolShed route, do this manually.

> I also tried testing by uploading the tool to the testtoolshed (as jtest1).
> I deliberatly put the wrong value in the testoutput.txt file because I'd
> like to see a test fail before I'll accept that it succeeds. It appears the
> testinput does not get tested on upload, because I got no automated test
> results, and no error messages anywhere. Is that expected behavior? I know
> the tools get tested every so often, I'm just surprised it doesn't happen on
> upload.
> The tool could then also be easily installed to my galaxy distribution,
> without errors in my log or on the browser pages.

The main and Test Tool Shed should now be running the tests
every 48 hours or so - this service had been unavailable over
the Christmas/New Year period but should be OK now.

> Question #3: At what point in creating or downloading a repository
> are these tests run?

Never, assuming I understand your question.

If you mean installing a tool from the ToolShed into a Galaxy
instance via the Galaxy administrator options, then I think
after installation any tests should be run automatically.

> And lastly, #4: is either of these the best way to test my tests, or did I
> overlook something?
>
> Thank you!
>
> -Jeltje

While developing a new tool (or improving one), I personally
use a development Galaxy server (not the one our users run
all their real jobs on), and install the tools manually (as above),
and then test via run_tests.sh -id my_tool_id.

This is complemented by a TravisCI setup, see:
http://blastedbio.blogspot.jp/2013/09/using-travis-ci-for-testing-galaxy-tools.html

If that works, I upload the tool to the TestToolShed, and wait
for those test results. If that all works and I am ready to make
a public release, then I upload to the main ToolShed.

I talked about this (in the context of the NCBI BLAST+
wrappers) at the GCC2014 conference, slides here:
http://www.slideshare.net/pjacock/galaxy-blast-gcc2014

There is even a video of my talk (and most of the others)
on the Galaxy website here:
https://wiki.galaxyproject.org/Events/GCC2014/Abstracts

I hope that helped. There is a work in progress
"Best Practices" being worked on here:
https://wiki.galaxyproject.org/Tools/BestPractices

Regards,

Peter