New subject: Twill compares wrong file; was: Twill comparisons ignore GALAXY_TEST_NO_CLEANUP

11 Feb 2014

On Tue, Feb 11, 2014 at 3:41 PM, Dave Bouvier dave@bx.psu.edu wrote:
...
Peter,
Thanks for pointing this out, I've applied your changes in
12478:15fc8675064e on the default branch.
--Dave B.
Thanks Dave,
RE: http://lists.bx.psu.edu/pipermail/galaxy-dev/2014-February/018154.html
https://bitbucket.org/galaxy/galaxy-central/commits/15fc8675064ea46b7e081d96...
I'm still digging into the underlying problem, by focussing on just
one unit test from
https://github.com/peterjc/pico_galaxy/blob/master/tools/mira4/mira4_de_novo...
Namely:
<test>
            <param name="job_type" value="genome" />
            <param name="job_quality" value="accurate" />
            <param name="type" value="none" />
            <param name="filenames" value="ecoli.fastq" ftype="fastqsanger" />
            <output name="out_fasta" file="ecoli.mira4_de_novo.fasta"
ftype="fasta" />
        </test>
After the test runs, we have five files, which is expected as I have
one input and four outputs:
<outputs>
        <data name="out_fasta" format="fasta" label="MIRA de novo
contigs (FASTA)" />
        <data name="out_bam" format="bam" label="MIRA de novo assembly (BAM)" />
        <data name="out_maf" format="mira" label="MIRA de novo assembly" />
        <data name="out_log" format="txt" label="MIRA de novo log" />
    </outputs>
$ ls /tmp/tmp09TawY/tmpFOX9hS/database/files/000/
dataset_1.dat  dataset_2.dat  dataset_3.dat  dataset_4.dat  dataset_5.dat
dataset_1.dat --> test-data/ecoli.fastq
dataset_2.dat --> out_fasta
dataset_3.dat --> out_bam
dataset_4.dat --> out_maf
dataset_5.dat --> out_log
Of these, the test should only attempt to compare dataset_2.dat (out_fasta)
to my sample output file test-data/ecoli.mira4_de_novo.fasta which it matches:
$ diff test-data/ecoli.mira4_de_novo.fasta
/tmp/tmp09TawY/tmpFOX9hS/database/files/000/dataset_2.dat
However, for reasons unknown, it is comparing dataset_5.dat (out_log)
to the sample output file test-data/ecoli.mira4_de_novo.fasta instead.
A little more logging (e.g. [1]) shows the problem is when calling into the
verify_dataset_correctness method (should use hid=2):
base.interactor: DEBUG: About to call
verify_dataset_correctness('ecoli.mira4_de_novo.fasta', '5', ...)
base.interactor: DEBUG: About to call verify_dataset_correctness(...)
for output_data 'state': 'queued'
base.interactor: DEBUG: About to call verify_dataset_correctness(...)
for output_data 'hid': '5'
base.interactor: DEBUG: About to call verify_dataset_correctness(...)
for output_data 'id': '5'
base.interactor: DEBUG: About to call verify_dataset_correctness(...)
for output_data 'dbkey': 'hg17'
base.interactor: DEBUG: About to call verify_dataset_correctness(...)
for output_data 'name': 'MIRA de novo log'
base.twilltestcase: DEBUG: Verifying dataset correctness for
'ecoli.mira4_de_novo.fasta' (hid='5')
Now what is strange is how can the call to get the hid for the current
outfile can know if it should be 2, 3, 4, or 5? The bug appears to be
in method _verify_outputs in test/functional/test_toolbox.py - which
was last touched by John...
output_data = data_list[ len(data_list) - len(testdef.outputs) + output_index ]
In this example, we have data_list[5 - 1 + 0] = 4, meaning since
data_list is essentially [dataset_1, ..., dataset_5] we get dataset_5
I believe the error is that rather than the number of defined test outputs
(here 1), you should be subtracting the number of tool outputs (here 4).
i.e. data_list[5 - 4 + 0] = data_list[1] = dataset_2
As a workaround, I could probably provide expected data for all
four output files...
Regards,
Peter
[1] Patch which adds more logging to test/base/interactor.py
(similar debugging added to other files to trace this.)
$ hg diff test/base/interactor.py
diff -r f3dc213a5773 test/base/interactor.py

--- a/test/base/interactor.py    Mon Feb 10 22:13:35 2014 -0600
+++ b/test/base/interactor.py    Tue Feb 11 16:19:51 2014 +0000
@@ -309,6 +309,9 @@
def verify_output( self, history, output_data, outfile,
attributes, shed_tool_id, maxseconds ):
         hid = output_data.get( 'hid' )
+        log.debug("About to call verify_dataset_correctness(%r, %r,
...)" % (outfile, hid))
+        for key in output_data.keys():
+           log.debug("About to call verify_dataset_correctness(...)
for output_data %r: %r" % (key, output_data.get(key)))
         self.twill_test_case.verify_dataset_correctness( outfile,
hid=hid, attributes=attributes, shed_tool_id=shed_tool_id,
maxseconds=maxseconds )
def get_job_stream( self, history_id, output_data, stream ):

    