Branch: refs/heads/dev Home: https://github.com/galaxyproject/galaxy Commit: ea1bfb0056edbb69ab149d8664098127494e7b60 https://github.com/galaxyproject/galaxy/commit/ea1bfb0056edbb69ab149d8664098... Author: John Chilton <jmchilton@gmail.com> Date: 2018-03-28 (Wed, 28 Mar 2018) Changed paths: M lib/galaxy/datatypes/annotation.py M lib/galaxy/datatypes/assembly.py M lib/galaxy/datatypes/blast.py M lib/galaxy/datatypes/constructive_solid_geometry.py M lib/galaxy/datatypes/data.py M lib/galaxy/datatypes/genetics.py M lib/galaxy/datatypes/interval.py M lib/galaxy/datatypes/molecules.py M lib/galaxy/datatypes/mothur.py M lib/galaxy/datatypes/msa.py M lib/galaxy/datatypes/phylip.py M lib/galaxy/datatypes/plant_tribes.py M lib/galaxy/datatypes/proteomics.py M lib/galaxy/datatypes/qualityscore.py M lib/galaxy/datatypes/sequence.py M lib/galaxy/datatypes/sniff.py M lib/galaxy/datatypes/tabular.py A lib/galaxy/datatypes/test/1.afg A lib/galaxy/datatypes/test/1.arff A lib/galaxy/datatypes/test/1.hdt A lib/galaxy/datatypes/test/1.obo A lib/galaxy/datatypes/test/1.owl A lib/galaxy/datatypes/test/1.phyloxml A lib/galaxy/datatypes/test/1.stockholm A lib/galaxy/datatypes/test/1.ttl A lib/galaxy/datatypes/test/1.xmfa A lib/galaxy/datatypes/test/2.tabular A lib/galaxy/datatypes/test/2.txt A lib/galaxy/datatypes/test/3.txt A lib/galaxy/datatypes/test/Acanium.hmm M lib/galaxy/datatypes/text.py M lib/galaxy/datatypes/triples.py M lib/galaxy/datatypes/xml.py M lib/galaxy/util/compression_utils.py Log Message: ----------- Framework for constrained memory, I/O sniffing. - Every unbound readline is a huge potential memory problem, this shrinks the number of such readlines down quite a bit. - There should be no reason to re-read the same file as much as we do, this keeps a fixed amount of the beginning of the file in memory and tries to use it as much as possible. - Lazy loads the size of the file and reuses that to reduce disc contention in that way also. With new sniffing tests for various types. Many datatypes will not just inspect the first GALAXY_SNIFF_PREFIX_BYTES of the file. This value defaults to 1 Mb but can be increased to increase the amount of memory and time used by the sniffers (getting more correctness also). Commit: 30e3658b8b0e2f6b975dc6ccccb0cc8cc040247c https://github.com/galaxyproject/galaxy/commit/30e3658b8b0e2f6b975dc6ccccb0c... Author: Martin Cech <marten@bx.psu.edu> Date: 2018-04-13 (Fri, 13 Apr 2018) Changed paths: M lib/galaxy/datatypes/annotation.py M lib/galaxy/datatypes/assembly.py M lib/galaxy/datatypes/blast.py M lib/galaxy/datatypes/constructive_solid_geometry.py M lib/galaxy/datatypes/data.py M lib/galaxy/datatypes/genetics.py M lib/galaxy/datatypes/interval.py M lib/galaxy/datatypes/molecules.py M lib/galaxy/datatypes/mothur.py M lib/galaxy/datatypes/msa.py M lib/galaxy/datatypes/phylip.py M lib/galaxy/datatypes/plant_tribes.py M lib/galaxy/datatypes/proteomics.py M lib/galaxy/datatypes/qualityscore.py M lib/galaxy/datatypes/sequence.py M lib/galaxy/datatypes/sniff.py M lib/galaxy/datatypes/tabular.py A lib/galaxy/datatypes/test/1.afg A lib/galaxy/datatypes/test/1.arff A lib/galaxy/datatypes/test/1.hdt A lib/galaxy/datatypes/test/1.obo A lib/galaxy/datatypes/test/1.owl A lib/galaxy/datatypes/test/1.phyloxml A lib/galaxy/datatypes/test/1.stockholm A lib/galaxy/datatypes/test/1.ttl A lib/galaxy/datatypes/test/1.xmfa A lib/galaxy/datatypes/test/2.tabular A lib/galaxy/datatypes/test/2.txt A lib/galaxy/datatypes/test/3.txt A lib/galaxy/datatypes/test/Acanium.hmm M lib/galaxy/datatypes/text.py M lib/galaxy/datatypes/triples.py M lib/galaxy/datatypes/xml.py M lib/galaxy/util/compression_utils.py Log Message: ----------- Merge pull request #5793 from jmchilton/bounded_memory_datatypes Sniffing framework with constrained memory and I/O. Compare: https://github.com/galaxyproject/galaxy/compare/225f6af3c07b...30e3658b8b0e