Functional tests using gzipped inputs have broken
Hi all, Using gzipped input files in tool functional tests (which would be decompressed by the upload tool) used to work, e.g. https://travis-ci.org/peterjc/pico_galaxy/builds/42395911 https://github.com/peterjc/pico_galaxy/blob/539260fa4348ed454ce464e48c9dbbcc... But with more recent versions of Galaxy this has broken, e.g. https://travis-ci.org/peterjc/pico_galaxy/builds/44038046 https://github.com/peterjc/pico_galaxy/blob/e3064cc8f42253efd1f8eed56844d4c5... According to the TravisCI failure, the command run was: python /home/travis/build/peterjc/pico_galaxy/galaxy-central-master/tools/seq_filter_by_mapping/seq_filter_by_mapping.py -i "None" -f "data" -m lax -p /tmp/tmpaCsMm5/tmpEpBIX1/database/files/000/dataset_50.dat /tmp/tmpaCsMm5/tmpEpBIX1/database/files/000/dataset_49.dat In this case the -i argument was $input_file, and -f was its extension/datatype. Somehow the gzipped FASTQ was not found, which is why the filename is just "None" and its datatype the default "data". My tool is sensibly returning error code 1, and: Missing input file: 'None' I have made some minor changes to this tool between those TravisCI tests, but the test is the same: <tests> <test> <param name="input_file" value="SRR639755_mito_pairs.fastq.gz" ftype="fastqsanger" /> <param name="mapping_file" value="SRR639755_sample_by_coord.sam" ftype="sam" /> <param name="pair_mode" value="lax" /> <param name="output_choice" value="pos" /> <output name="output_pos" file="SRR639755_sample_lax.fastq" ftype="fastqsanger" /> </test> <test> <param name="input_file" value="SRR639755_mito_pairs.fastq.gz" ftype="fastqsanger" /> <param name="mapping_file" value="SRR639755_sample_by_coord.sam" ftype="sam" /> <param name="pair_mode" value="strict" /> <param name="output_choice" value="pos" /> <output name="output_pos" file="SRR639755_sample_strict.fastq" ftype="fastqsanger" /> </test> </tests> -- I could reproduce this on my development machine running an older Galaxy: $ hg log | head changeset: 16732:212e1d5e9be5 branch: stable tag: tip parent: 16716:2db0fb9594d6 parent: 16731:7adac1842adf user: John Chilton <jmchilton@gmail.com> date: Wed Dec 10 12:20:55 2014 -0500 summary: Merged in dannon/galaxy-central/stable (pull request #602) And again updating to the current tip, $ hg log | head changeset: 16850:e614c7c1e0e9 tag: tip parent: 16848:1cf0af0a6324 parent: 16849:d13f2b265dbf user: Dannon Baker <dannon.baker@gmail.com> date: Tue Dec 16 15:48:44 2014 -0500 summary: Merged in nitesh1989/galaxy-central3 (pull request #615) I can "fix" it by swapping the gzipped test input_file SRR639755_mito_pairs.fastq.gz to the uncompressed SRR639755_mito_pairs.fastq instead, as in this commit: https://github.com/peterjc/pico_galaxy/commit/126e3e716965016808ec2ba3a4be3b... https://travis-ci.org/peterjc/pico_galaxy/builds/44317785 I would like to revert this workaround since the uncompressed FASTQ file is a bit large to bundle as a test case - so I hope this is simply due to an accidental regression in Galaxy itself. Thanks, Peter
Hey Peter, Thanks for the very detailed bug report - it really helped track things down. I have pushed a fix to next-stable and default (the bug never reached stable). https://bitbucket.org/galaxy/galaxy-central/commits/a74d32014b0512613042c2a4... The commit also includes a test case so hopefully this won't break again. -John On Wed, Dec 17, 2014 at 6:56 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
Hi all,
Using gzipped input files in tool functional tests (which would be decompressed by the upload tool) used to work, e.g.
https://travis-ci.org/peterjc/pico_galaxy/builds/42395911 https://github.com/peterjc/pico_galaxy/blob/539260fa4348ed454ce464e48c9dbbcc...
But with more recent versions of Galaxy this has broken, e.g.
https://travis-ci.org/peterjc/pico_galaxy/builds/44038046 https://github.com/peterjc/pico_galaxy/blob/e3064cc8f42253efd1f8eed56844d4c5...
According to the TravisCI failure, the command run was:
python /home/travis/build/peterjc/pico_galaxy/galaxy-central-master/tools/seq_filter_by_mapping/seq_filter_by_mapping.py -i "None" -f "data" -m lax -p /tmp/tmpaCsMm5/tmpEpBIX1/database/files/000/dataset_50.dat /tmp/tmpaCsMm5/tmpEpBIX1/database/files/000/dataset_49.dat
In this case the -i argument was $input_file, and -f was its extension/datatype. Somehow the gzipped FASTQ was not found, which is why the filename is just "None" and its datatype the default "data".
My tool is sensibly returning error code 1, and:
Missing input file: 'None'
I have made some minor changes to this tool between those TravisCI tests, but the test is the same:
<tests> <test> <param name="input_file" value="SRR639755_mito_pairs.fastq.gz" ftype="fastqsanger" /> <param name="mapping_file" value="SRR639755_sample_by_coord.sam" ftype="sam" /> <param name="pair_mode" value="lax" /> <param name="output_choice" value="pos" /> <output name="output_pos" file="SRR639755_sample_lax.fastq" ftype="fastqsanger" /> </test> <test> <param name="input_file" value="SRR639755_mito_pairs.fastq.gz" ftype="fastqsanger" /> <param name="mapping_file" value="SRR639755_sample_by_coord.sam" ftype="sam" /> <param name="pair_mode" value="strict" /> <param name="output_choice" value="pos" /> <output name="output_pos" file="SRR639755_sample_strict.fastq" ftype="fastqsanger" /> </test> </tests>
--
I could reproduce this on my development machine running an older Galaxy:
$ hg log | head changeset: 16732:212e1d5e9be5 branch: stable tag: tip parent: 16716:2db0fb9594d6 parent: 16731:7adac1842adf user: John Chilton <jmchilton@gmail.com> date: Wed Dec 10 12:20:55 2014 -0500 summary: Merged in dannon/galaxy-central/stable (pull request #602)
And again updating to the current tip,
$ hg log | head changeset: 16850:e614c7c1e0e9 tag: tip parent: 16848:1cf0af0a6324 parent: 16849:d13f2b265dbf user: Dannon Baker <dannon.baker@gmail.com> date: Tue Dec 16 15:48:44 2014 -0500 summary: Merged in nitesh1989/galaxy-central3 (pull request #615)
I can "fix" it by swapping the gzipped test input_file SRR639755_mito_pairs.fastq.gz to the uncompressed SRR639755_mito_pairs.fastq instead, as in this commit:
https://github.com/peterjc/pico_galaxy/commit/126e3e716965016808ec2ba3a4be3b... https://travis-ci.org/peterjc/pico_galaxy/builds/44317785
I would like to revert this workaround since the uncompressed FASTQ file is a bit large to bundle as a test case - so I hope this is simply due to an accidental regression in Galaxy itself.
Thanks,
Peter ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Thanks John, That sounds like quite a subtle root cause - well done. I've reverted my workaround and the TravisCI tests pass again: https://github.com/peterjc/pico_galaxy/commit/265bf7f62ad4fa7c8557a6bfa5af65... https://travis-ci.org/peterjc/pico_galaxy/builds/44334522 Regards, Peter On Wed, Dec 17, 2014 at 2:45 PM, John Chilton <jmchilton@gmail.com> wrote:
Hey Peter,
Thanks for the very detailed bug report - it really helped track things down.
I have pushed a fix to next-stable and default (the bug never reached stable).
https://bitbucket.org/galaxy/galaxy-central/commits/a74d32014b0512613042c2a4...
The commit also includes a test case so hopefully this won't break again.
-John
On Wed, Dec 17, 2014 at 6:56 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
Hi all,
Using gzipped input files in tool functional tests (which would be decompressed by the upload tool) used to work, e.g.
https://travis-ci.org/peterjc/pico_galaxy/builds/42395911 https://github.com/peterjc/pico_galaxy/blob/539260fa4348ed454ce464e48c9dbbcc...
But with more recent versions of Galaxy this has broken, e.g.
https://travis-ci.org/peterjc/pico_galaxy/builds/44038046 https://github.com/peterjc/pico_galaxy/blob/e3064cc8f42253efd1f8eed56844d4c5...
According to the TravisCI failure, the command run was:
python /home/travis/build/peterjc/pico_galaxy/galaxy-central-master/tools/seq_filter_by_mapping/seq_filter_by_mapping.py -i "None" -f "data" -m lax -p /tmp/tmpaCsMm5/tmpEpBIX1/database/files/000/dataset_50.dat /tmp/tmpaCsMm5/tmpEpBIX1/database/files/000/dataset_49.dat
In this case the -i argument was $input_file, and -f was its extension/datatype. Somehow the gzipped FASTQ was not found, which is why the filename is just "None" and its datatype the default "data".
My tool is sensibly returning error code 1, and:
Missing input file: 'None'
I have made some minor changes to this tool between those TravisCI tests, but the test is the same:
<tests> <test> <param name="input_file" value="SRR639755_mito_pairs.fastq.gz" ftype="fastqsanger" /> <param name="mapping_file" value="SRR639755_sample_by_coord.sam" ftype="sam" /> <param name="pair_mode" value="lax" /> <param name="output_choice" value="pos" /> <output name="output_pos" file="SRR639755_sample_lax.fastq" ftype="fastqsanger" /> </test> <test> <param name="input_file" value="SRR639755_mito_pairs.fastq.gz" ftype="fastqsanger" /> <param name="mapping_file" value="SRR639755_sample_by_coord.sam" ftype="sam" /> <param name="pair_mode" value="strict" /> <param name="output_choice" value="pos" /> <output name="output_pos" file="SRR639755_sample_strict.fastq" ftype="fastqsanger" /> </test> </tests>
--
I could reproduce this on my development machine running an older Galaxy:
$ hg log | head changeset: 16732:212e1d5e9be5 branch: stable tag: tip parent: 16716:2db0fb9594d6 parent: 16731:7adac1842adf user: John Chilton <jmchilton@gmail.com> date: Wed Dec 10 12:20:55 2014 -0500 summary: Merged in dannon/galaxy-central/stable (pull request #602)
And again updating to the current tip,
$ hg log | head changeset: 16850:e614c7c1e0e9 tag: tip parent: 16848:1cf0af0a6324 parent: 16849:d13f2b265dbf user: Dannon Baker <dannon.baker@gmail.com> date: Tue Dec 16 15:48:44 2014 -0500 summary: Merged in nitesh1989/galaxy-central3 (pull request #615)
I can "fix" it by swapping the gzipped test input_file SRR639755_mito_pairs.fastq.gz to the uncompressed SRR639755_mito_pairs.fastq instead, as in this commit:
https://github.com/peterjc/pico_galaxy/commit/126e3e716965016808ec2ba3a4be3b... https://travis-ci.org/peterjc/pico_galaxy/builds/44317785
I would like to revert this workaround since the uncompressed FASTQ file is a bit large to bundle as a test case - so I hope this is simply due to an accidental regression in Galaxy itself.
Thanks,
Peter ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
participants (2)
-
John Chilton -
Peter Cock