On Wed, May 16, 2012 at 9:39 PM, Nate Coraor <nate(a)bx.psu.edu&gt; wrote: Hi Nate, Where does this sit on the Galaxy team's priorities now, 18 months on? I think I asked about this at the GCC2013, any it was seen as important but not yet at the top of the priorities list. I know that the main Galaxy instance has moved from Penn State to the iPlant Collaborative at the Texas Advanced Computing Center, and that probably gave you some breathing room for storage - but surely using compressed files would help enormously? I would still love to see built in support for a range of compressions, starting with none and gzipped as the priorities, but also ideally the option of BGZF (the random access gzip subtype), BZ2 (better compression), XY, blocked XY, etc. My vision is that in the tool XML we'd have optional tags on input data parameters to define supported compression types as well as supported file types, likewise for the output tags. For example, consider a FASTQ to FASTQ tool like a read trimmer or filter, like my seq_filter_by_id.xml https://github.com/peterjc/pico_galaxy/blob/master/tools/seq_filter_by_id... ... <param name="input_file" type="data" format="fasta,fastq,sff" label="Sequence file to filter on the identifiers" help="FASTA, FASTQ, or SFF format." /> ... If I modified this tool to handle gzipped files, then I would tell Galaxy that in some way, e.g. ... <param name="input_file" type="data" format="fasta,fastq,sff" compression="none,gzip" label="Sequence file to filter on the identifiers" help="FASTA, FASTQ, or SFF format." /> ... (For any input file without a compression tag, assume it expects no compression) In the same way that the Cheetah syntax gives access to the input file's type via $input_file.ext we'd need to access the compression type via somthing like $input_file.compression (e.g. if I needed to add -gzip switch to the command line). Then, before running the tool, Galaxy would have to compare the current compression of $input_file as stored on disk to the range of compression types the tool can accept. If the tool is happy as is, great - just run it. If not, then Galaxy would have to add a decompression step (e.g. gunzip). For the output files, Galaxy could probably sniff the files to spot the compression automatically, or it could be specified by the tool wrapper's XML. If we want Galaxy to automatically store files gzipped/bz2/..., then after running the tool it would have to call gzip/bzip/etc - but the tool and tool wrapper don't need to care about this. -- These potential (de)compressions could initially be done via $TEMP (or some configurable scratch space local to each cluster node), but we could avoid creating temp files by using named pipes. This would only work where the tool streamed the input/output file without using random access (seek calls), thus we would need additional XML markup, e.g. ... <param name="input_file" type="data" format="fasta,fastq,sff" compression="none,gzip" stream_file="true" label="Sequence file to filter on the identifiers" help="FASTA, FASTQ, or SFF format." /> ... Having tools mark input/output files as streamable would open the door to some fancy optimisations for chaining tools together where the intermediate files are never stored directly - but piped from one tool to another. This would be very attractive for running some workflows, see e.g.: http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-November/017422.html However, a simple decompression/compression step with staging files in $TEMP alone would be a big step forward. Regards, Peter

Our approach has been to integrate compress / un-compress, with the job splitter / cluster launch layer that we've been working on (current incarnation is https://bitbucket.org/agr-bifo/tardis ). tardis approach is to sniff the input(s) and if necessary insert an uncompressor into the input stream - the original data is left compressed in-place, and only the splits of the data are uncompressed. Each uncompressed data chunk is launched for processing on the cluster as soon as it becomes available from the uncompressed input stream. This can be quite a big performance advantage, as compared with uncompressing the entire input file, before splitting it, which can be quite slow and has a bigger disk footprint. We find that job splitting, compression handling (and potentially other low level data file transforms - e.g. handling list files, random sampling of input) are interdependent. Potentially best encapsulated in a lower layer, which would avoid cluttering Galaxy's high level bio-oriented data type ontologies ? Currently (as per earlier post) we are integrating this approach into Galaxy by modifying selected tool config files, longer term we think it could be possible to slot this low level data transform and task splitting layer into the core galaxy stack. John Chilton kindly set up a trello card on the general topic of task splitting - this is at https://trello.com/c/H87LotF7 (Apologies if this repeats some of my earlier post - the point here is that e.g. compress/un-compress and task splitting are interdependent in practice, we find) (Having said the above - encapsulating data transforms such as compress/un-compress even lower down the stack, in the file-system itself , as per the public Galaxy instance (ZFS), could be pretty hard to beat !)