Branch: refs/heads/dev Home: https://github.com/galaxyproject/galaxy Commit: 9a24338d12001cdcea336703ce41ac4032c275b8 https://github.com/galaxyproject/galaxy/commit/9a24338d12001cdcea336703ce41a... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M lib/galaxy/datatypes/binary.py M lib/galaxy/datatypes/sniff.py R lib/galaxy/datatypes/test/1.vcf.gz A lib/galaxy/datatypes/test/1.vcf_bgzip R lib/galaxy/datatypes/test/Acanium.hmm A lib/galaxy/datatypes/test/Acanium.snaphmm R lib/galaxy/datatypes/test/biom2_sparse_otu_table_hdf5.biom A lib/galaxy/datatypes/test/biom2_sparse_otu_table_hdf5.biom2 R lib/galaxy/datatypes/test/test.CEL A lib/galaxy/datatypes/test/test.cel R lib/galaxy/datatypes/test/test.idpDB A lib/galaxy/datatypes/test/test.idpdb R lib/galaxy/datatypes/test/test_ensembl.tab A lib/galaxy/datatypes/test/test_ensembl.tabular A test/integration/test_datatype_upload.py M test/unit/datatypes/test_vcf.py Log Message: ----------- Name datatype test files according to datatype extension Commit: b2276319d409c7d88bf3af7ff0cfa8cd4df2ab98 https://github.com/galaxyproject/galaxy/commit/b2276319d409c7d88bf3af7ff0cfa... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M lib/galaxy/datatypes/binary.py Log Message: ----------- Change more biom2 instances Commit: 7ce69df57c7a2bb6e7c34108a8e2e0b67487cd5d https://github.com/galaxyproject/galaxy/commit/7ce69df57c7a2bb6e7c34108a8e2e... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M test/integration/test_datatype_upload.py Log Message: ----------- Add integration test for uploading datasets of known datatype We upload everything in lib/galaxy/datatypes/test and make sure that we get the correct datatype. For datatypes that have a sniffer we use `auto`, otherwise we set the correct datatype. This is fairly basic for now, but it would be trivial to extend this to also verify file integrity and metadata elements as well as testing options that affect uploads, like `auto_decompress` settings. Commit: f2d8218a5366f25601dac8342601c02a74ca6f22 https://github.com/galaxyproject/galaxy/commit/f2d8218a5366f25601dac8342601c... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M lib/galaxy/datatypes/binary.py Log Message: ----------- Fix uncompressed bcf This actually also fails, but at least doesn't crash the interpreter. Commit: 63eb6a145c9683651532fc7afaded7a5d930743a https://github.com/galaxyproject/galaxy/commit/63eb6a145c9683651532fc7afaded... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M lib/galaxy/datatypes/sequence.py Log Message: ----------- Tighten up memepsp sniffer Values need to be floats, but float("1") also doesn't raise an exception. This prevents 454qual files to be sniffed as memepsp. Commit: f7773c39b1fb7d9ac31115b69c60f5f487e40a13 https://github.com/galaxyproject/galaxy/commit/f7773c39b1fb7d9ac31115b69c60f... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M lib/galaxy/datatypes/blast.py A lib/galaxy/datatypes/test/tblastn_four_human_vs_rhodopsin.blastxml R lib/galaxy/datatypes/test/tblastn_four_human_vs_rhodopsin.xml Log Message: ----------- Move tblastn_four_human_vs_rhodopsin.xml to tblastn_four_human_vs_rhodopsin.blastxml Commit: 38e18c9daff58a02f1da0be23329ec9ec5d4cbe7 https://github.com/galaxyproject/galaxy/commit/38e18c9daff58a02f1da0be23329e... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M lib/galaxy/datatypes/interval.py M lib/galaxy/datatypes/sniff.py A lib/galaxy/datatypes/test/gff.gff3 R lib/galaxy/datatypes/test/gff_version_3.gff Log Message: ----------- Rename gff3 test file Commit: 3529295d5512bee68e7981d036147b18a518fc6c https://github.com/galaxyproject/galaxy/commit/3529295d5512bee68e7981d036147... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: R lib/galaxy/datatypes/test/sequence.qual Log Message: ----------- Delete unused test file Commit: bc0a3c5244181a5b6af2a4d394a0d4375ceff453 https://github.com/galaxyproject/galaxy/commit/bc0a3c5244181a5b6af2a4d394a0d... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M lib/galaxy/datatypes/interval.py M lib/galaxy/datatypes/sniff.py A lib/galaxy/datatypes/test/interv1.bed R lib/galaxy/datatypes/test/interval1.bed Log Message: ----------- Rename interval1.bed to interv1.bed Otherwise guess_datatype would expect interval as the datatype. Commit: 81155c635a38eb0ad1201f92bb70fbdf9faf170b https://github.com/galaxyproject/galaxy/commit/81155c635a38eb0ad1201f92bb70f... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: R lib/galaxy/datatypes/test/2.shuffled.bam A lib/galaxy/datatypes/test/2.shuffled.unsorted.bam M test/unit/datatypes/test_bam.py Log Message: ----------- Rename 2.shuffled.bam to 2.shuffled.unsorted.bam Commit: a151b63a622424c1f5f21f97169121b4c7bcc5d5 https://github.com/galaxyproject/galaxy/commit/a151b63a622424c1f5f21f9716912... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M config/datatypes_conf.xml.sample M lib/galaxy/datatypes/registry.py M lib/galaxy/datatypes/sequence.py R lib/galaxy/datatypes/test/2.fastqsanger A lib/galaxy/datatypes/test/3.fastq Log Message: ----------- Tighten up fastq sniffers significantly This requires the sequence regex to match over the entire sequence, which precludes fastqcssanger files to be sniffed as fastqsanger (fixes https://github.com/galaxyproject/galaxy/issues/6734). Further adds a sniffer for FastqCSSanger, which contains numbers in the sequence and so shouldn't interfere with the other fastq datatypes. Commit: b74f225f189688ea885ea6b52bf0f36367cd8d03 https://github.com/galaxyproject/galaxy/commit/b74f225f189688ea885ea6b52bf0f... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M test/integration/test_datatype_upload.py Log Message: ----------- Map fastqsolex to fastq datatype in integration test Commit: 720386999828150ed3b639c3600f491e75c7217b https://github.com/galaxyproject/galaxy/commit/720386999828150ed3b639c3600f4... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M lib/galaxy/datatypes/binary.py M lib/galaxy/datatypes/test/1.bcf_uncompressed Log Message: ----------- Drop index file for bcf_uncompressed Even `bcftools index` complains that the file is not compressed. I think this is just not supported. Commit: 85a47ccc47b6f3504360e9f4a0486cefe3f6faa8 https://github.com/galaxyproject/galaxy/commit/85a47ccc47b6f3504360e9f4a0486... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M lib/galaxy/datatypes/sequence.py Log Message: ----------- Fix totally broken memepsp sniffer The sniffer could never reach 100 sniffed lines, and the test file contains an empty line, so the comment and code that says that empty lines are not valid is wrong, but was never called. Commit: 48bdc97fce177c5f867cd31e3600ad5670a6c958 https://github.com/galaxyproject/galaxy/commit/48bdc97fce177c5f867cd31e3600a... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M test/base/populators.py M test/integration/test_datatype_upload.py Log Message: ----------- Verify state of uploaded datasets and download datasets to verify that the file is intact after upload Commit: be939310fbd4736eaaf8e3ecb00081e7fc080463 https://github.com/galaxyproject/galaxy/commit/be939310fbd4736eaaf8e3ecb0008... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: A lib/galaxy/datatypes/test/1.fastqcssanger Log Message: ----------- Add a fastqcssanger test file Commit: 7b955277566a8cb647b1a689cf1c0d18fba274f2 https://github.com/galaxyproject/galaxy/commit/7b955277566a8cb647b1a689cf1c0... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M lib/galaxy/datatypes/sniff.py Log Message: ----------- Missed one idpDB instance Commit: dd6a4bb2f33e39889577e9a0b60a6314355eab6b https://github.com/galaxyproject/galaxy/commit/dd6a4bb2f33e39889577e9a0b60a6... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M config/datatypes_conf.xml.sample M lib/galaxy/datatypes/sniff.py M lib/galaxy/datatypes/tabular.py A lib/galaxy/datatypes/test/1.excel.xls R lib/galaxy/datatypes/test/1.xls Log Message: ----------- Move xls file to excel.xls and change xls to be excel.xls Commit: d80e4bd0a13dc5996c35cd5ecbca9c4439e5e2a1 https://github.com/galaxyproject/galaxy/commit/d80e4bd0a13dc5996c35cd5ecbca9... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M test/base/populators.py M test/integration/test_datatype_upload.py Log Message: ----------- Fix test logic Commit: e01e6ba2bdcf557668995f402812ff8b93474134 https://github.com/galaxyproject/galaxy/commit/e01e6ba2bdcf557668995f402812f... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M lib/galaxy/datatypes/test/1.afg M lib/galaxy/datatypes/test/1.fastq M lib/galaxy/datatypes/test/1.fastqsanger M lib/galaxy/datatypes/test/1.fastqsolexa M lib/galaxy/datatypes/test/1.owl M lib/galaxy/datatypes/test/1.stockholm M lib/galaxy/datatypes/test/2.fastq M lib/galaxy/datatypes/test/2.fastqsolexa M lib/galaxy/datatypes/test/2.txt M lib/galaxy/datatypes/test/3.fastq M lib/galaxy/datatypes/test/3.txt M lib/galaxy/datatypes/test/454Score.pdf M lib/galaxy/datatypes/test/alignment.lav M lib/galaxy/datatypes/test/interv1.bed M lib/galaxy/datatypes/test/mothur_datatypetest_true.mothur.filter M lib/galaxy/datatypes/test/sequence.qual454 M lib/galaxy/datatypes/test/sequence.qualsolid M lib/galaxy/datatypes/test/test_tab.bed M lib/galaxy/datatypes/test/ucsc.customtrack M lib/galaxy/datatypes/test/wiggle.wig Log Message: ----------- Fix up various test datasets Commit: ec193d929fd81b8f0438c097ebbb3aa3fb55e224 https://github.com/galaxyproject/galaxy/commit/ec193d929fd81b8f0438c097ebbb3... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M lib/galaxy/datatypes/sequence.py Log Message: ----------- Fix minor typo Commit: d3cd57e29975b66ca3883b6e8dfb63839a49aaee https://github.com/galaxyproject/galaxy/commit/d3cd57e29975b66ca3883b6e8dfb6... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M lib/galaxy/model/custom_types.py Log Message: ----------- Safe JSON encoding of MetadataParameters, JSONType This fixes a couple of upload tests on python 3 (loom, Biom2). Commit: 09601ad4efdd877375dfda3e62e2657899101ae2 https://github.com/galaxyproject/galaxy/commit/09601ad4efdd877375dfda3e62e26... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M lib/galaxy/datatypes/sniff.py Log Message: ----------- Skip newlines/space2tab conversion for binary datatypes that fail is_binary check Commit: d168042b366dcdbacabfabe692f120e9118a2de9 https://github.com/galaxyproject/galaxy/commit/d168042b366dcdbacabfabe692f12... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M lib/galaxy/datatypes/sniff.py Log Message: ----------- Fix newline conversion on python 3 if no encoding is set Commit: 738bde01278c50a62f4d97c7afc6e4f8b919254f https://github.com/galaxyproject/galaxy/commit/738bde01278c50a62f4d97c7afc6e... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M lib/galaxy/util/checkers.py Log Message: ----------- Fix check_html on python 3 We can open the file in binary mode and then use unicodify on the lines, as we have to do that anyway if `chunks` is provided. Also adds the unicode flag to re.compile. Commit: 1a407429ba7ffe1476317c5ac0f5b1ad19f1e0d7 https://github.com/galaxyproject/galaxy/commit/1a407429ba7ffe1476317c5ac0f5b... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M lib/galaxy/datatypes/mothur.py M lib/galaxy/datatypes/phylip.py Log Message: ----------- Tighten up mothur and phylip sniffers Commit: c50a0fccace2d94565ba85f0703e516139132b4b https://github.com/galaxyproject/galaxy/commit/c50a0fccace2d94565ba85f0703e5... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M lib/galaxy/datatypes/text.py Log Message: ----------- Don't sniff simple json types These should instead be text. Commit: 0fae117e672e626cc7093d0169dae241c4f54128 https://github.com/galaxyproject/galaxy/commit/0fae117e672e626cc7093d0169dae... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M lib/galaxy/datatypes/sequence.py M lib/galaxy/datatypes/sniff.py Log Message: ----------- Use raw strings in regex and allow lowercase fastq sequences Commit: ecc02bb659ce89e78b1a5aeb49dd020280864617 https://github.com/galaxyproject/galaxy/commit/ecc02bb659ce89e78b1a5aeb49dd0... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M lib/galaxy/datatypes/upload_util.py Log Message: ----------- Fix UploadProblemException Commit: ba5d76eb934598227102ae1cb0ff4e89e0ab0b9c https://github.com/galaxyproject/galaxy/commit/ba5d76eb934598227102ae1cb0ff4... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M lib/galaxy/datatypes/sequence.py Log Message: ----------- Avoid exception if no match This fixes the sniffing unit test, the datatype framework would just continue anyway. Commit: 16962ad6a16c5b5dcc5b3d06921d2c4c89c6d9d0 https://github.com/galaxyproject/galaxy/commit/16962ad6a16c5b5dcc5b3d06921d2... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M lib/galaxy/datatypes/test/5e5z.pqr Log Message: ----------- Add newline to lib/galaxy/datatypes/test/5e5z.pqr Commit: 9713556b16acb3e4218fc9e28463cdaba654a0c5 https://github.com/galaxyproject/galaxy/commit/9713556b16acb3e4218fc9e28463c... Author: John Chilton <jmchilton@gmail.com> Date: 2018-10-30 (Tue, 30 Oct 2018) Changed paths: M config/datatypes_conf.xml.sample M lib/galaxy/datatypes/binary.py M lib/galaxy/datatypes/blast.py M lib/galaxy/datatypes/interval.py M lib/galaxy/datatypes/mothur.py M lib/galaxy/datatypes/phylip.py M lib/galaxy/datatypes/registry.py M lib/galaxy/datatypes/sequence.py M lib/galaxy/datatypes/sniff.py M lib/galaxy/datatypes/tabular.py M lib/galaxy/datatypes/test/1.afg M lib/galaxy/datatypes/test/1.bcf_uncompressed A lib/galaxy/datatypes/test/1.excel.xls M lib/galaxy/datatypes/test/1.fastq A lib/galaxy/datatypes/test/1.fastqcssanger M lib/galaxy/datatypes/test/1.fastqsanger M lib/galaxy/datatypes/test/1.fastqsolexa M lib/galaxy/datatypes/test/1.owl M lib/galaxy/datatypes/test/1.stockholm R lib/galaxy/datatypes/test/1.vcf.gz A lib/galaxy/datatypes/test/1.vcf_bgzip R lib/galaxy/datatypes/test/1.xls M lib/galaxy/datatypes/test/2.fastq R lib/galaxy/datatypes/test/2.fastqsanger M lib/galaxy/datatypes/test/2.fastqsolexa R lib/galaxy/datatypes/test/2.shuffled.bam A lib/galaxy/datatypes/test/2.shuffled.unsorted.bam M lib/galaxy/datatypes/test/2.txt A lib/galaxy/datatypes/test/3.fastq M lib/galaxy/datatypes/test/3.txt M lib/galaxy/datatypes/test/454Score.pdf M lib/galaxy/datatypes/test/5e5z.pqr R lib/galaxy/datatypes/test/Acanium.hmm A lib/galaxy/datatypes/test/Acanium.snaphmm M lib/galaxy/datatypes/test/alignment.lav R lib/galaxy/datatypes/test/biom2_sparse_otu_table_hdf5.biom A lib/galaxy/datatypes/test/biom2_sparse_otu_table_hdf5.biom2 A lib/galaxy/datatypes/test/gff.gff3 R lib/galaxy/datatypes/test/gff_version_3.gff A lib/galaxy/datatypes/test/interv1.bed R lib/galaxy/datatypes/test/interval1.bed M lib/galaxy/datatypes/test/mothur_datatypetest_true.mothur.filter R lib/galaxy/datatypes/test/sequence.qual M lib/galaxy/datatypes/test/sequence.qual454 M lib/galaxy/datatypes/test/sequence.qualsolid A lib/galaxy/datatypes/test/tblastn_four_human_vs_rhodopsin.blastxml R lib/galaxy/datatypes/test/tblastn_four_human_vs_rhodopsin.xml R lib/galaxy/datatypes/test/test.CEL A lib/galaxy/datatypes/test/test.cel R lib/galaxy/datatypes/test/test.idpDB A lib/galaxy/datatypes/test/test.idpdb R lib/galaxy/datatypes/test/test_ensembl.tab A lib/galaxy/datatypes/test/test_ensembl.tabular M lib/galaxy/datatypes/test/test_tab.bed M lib/galaxy/datatypes/test/ucsc.customtrack M lib/galaxy/datatypes/test/wiggle.wig M lib/galaxy/datatypes/text.py M lib/galaxy/datatypes/upload_util.py M lib/galaxy/model/custom_types.py M lib/galaxy/util/checkers.py M test/base/populators.py A test/integration/test_datatype_upload.py M test/unit/datatypes/test_bam.py M test/unit/datatypes/test_vcf.py Log Message: ----------- Merge pull request #6869 from mvdbeek/datatype_upload_integration_test Datatype upload integration test and fixes Compare: https://github.com/galaxyproject/galaxy/compare/0de9c738cf8c...9713556b16ac **NOTE:** This service has been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/ Functionality will be removed from GitHub.com on January 31st, 2019.