Twill comparisons ignore GALAXY_TEST_NO_CLEANUP

11 Feb 2014

Hi all,

Something very handy for tool developers running functional tests
is if the environment variable GALAXY_TEST_NO_CLEANUP
is set, run_functional_tests.sh doesn't remove the test output.
This is great when debugging why a test is failing.

However, I was having trouble with failures like this:

base.twilltestcase: INFO: ## files diff on
/mnt/galaxy/galaxy-central/test-data/ecoli.mira4_de_novo.fasta and
/tmp/tmpTvyobN/tmpYsEezt/new_files_path_pk7YKe/tmpXr9NYeecoli.mira4_de_novo.fasta
lines_diff=0, found diff = 3010

Sadly the temp file here was being deleted - the patch at the
end of the email fixes the diff code to respect the configuration
setting.

Having done that and looked at the preserved file, somehow
the wrong output file was being compared to the FASTA
output file (it was looking at my log file instead).

Because the diff was really between two different files,
showing the first forty lines wasn't informative. Instead
I've made it show the first 25 and last 25 lines of the diff.
I find this a great deal more informative :)

Meanwhile this points at a possible bug in the test
framework, if anyone wants to attempt to reproduce
it I used this version of my MIRA4 wrapper:
https://github.com/peterjc/pico_galaxy/commit/3f095eac3f5609b2890440b851029a...

$ export GALAXY_TEST_NO_CLEANUP=on
$ ./run_functional_tests.sh -id mira_4_0_de_novo
...

$ hg summary
parent: 13513:f3dc213a5773 tip
 Backout of 28d43f4.
branch: default
commit: 7 modified, 229 unknown
update: (current)


Regards,

Peter

--

$ hg diff test/base/twilltestcase.py
diff -r f3dc213a5773 test/base/twilltestcase.py

--- a/test/base/twilltestcase.py    Mon Feb 10 22:13:35 2014 -0600
+++ b/test/base/twilltestcase.py    Tue Feb 11 14:09:30 2014 +0000
@@ -95,7 +95,10 @@
                 diff = list( difflib.unified_diff( local_file,
history_data, "local_file", "history_data" ) )
                 diff_lines = get_lines_diff( diff )
                 if diff_lines > allowed_diff_count:
-                    diff_slice = diff[0:40]
+                    if len(diff) < 60:
+                        diff_slice = diff[0:40]
+                    else:
+                        diff_slice = diff[:25] + ["********\n", "*
SNIP *\n", "********\n"] + diff[-25:]
                     #FIXME: This pdf stuff is rather special cased
and has not been updated to consider lines_diff
                     #due to unknown desired behavior when used in
conjunction with a non-zero lines_diff
                     #PDF forgiveness can probably be handled better
by not special casing by __extension__ here
@@ -897,7 +900,8 @@
                 errmsg += str( err )
                 raise AssertionError( errmsg )
             finally:
-                os.remove( temp_name )
+                if 'GALAXY_TEST_NO_CLEANUP' not in os.environ:
+                    os.remove( temp_name )

     def __default_dataset_fetcher( self ):
         def fetcher( hda_id, filename=None ):
@@ -971,7 +975,8 @@
             errmsg += str( err )
             raise AssertionError( errmsg )
         finally:
-            os.remove( temp_name )
+            if 'GALAXY_TEST_NO_CLEANUP' not in os.environ:
+                os.remove( temp_name )

     def is_zipped( self, filename ):
         if not zipfile.is_zipfile( filename ):

    

Peter Cock

Dave Bouvier

tags

participants (2)