Hello Assaf,
Assaf Gordon wrote:
Hello,
As users add more and more data files to our galaxy server, disk space becomes a problem... What I'd ultimately like is to store the data files in some compressed manner (at least some of the textual files), how would you suggest to do that ?
A common scenario is:
- User uploads a big Fastq/solexa file (=> 1.2 GB)
- FASTQ file converted to FASTA file (=> 0.6 GB)
- FASTA file trimmed, clipped, stripped, etc. (=> 100 MB)
- BLAT, Histograms and other reports (=> ~50 MB)
The first three data sets take about 1.9 GB of disk space - and aren't really needed by the user (as he/she is mostly interested in the resulting report files). Since these are textual files, they compress really well.
See the http://g2.trac.bx.psu.edu/wiki/PurgeHistoriesAndDatasets wiki. You may find it useful to configure the cleanup_datasets scripts in cron to removed "deleted" datasets from disk after a configured number of days. Let me know if you have any questions about this process.
Currently, I store the FASTQ gzip'ed in galaxy, and my tools know how to read gzip'ed data.
There are two shortcomings with this method:
- datasets (green squares) of gzip'ed files don't display any data in
the peek window
This can be corrected in a fairly easy way. Just add "gzip" as a new data type ( see http://g2.trac.bx.psu.edu/wiki/AddingDatatypes ). In your "GZIP" class, include a "display_peek()" method or a "make_html_table()" method that will display what you want for the "gzip" data type. A close example of what you may need is available in the Binseq() class in ~/lib/galaxy/datatypes/images.py.
- Other galaxy tools which require FASTQ file as input can't read my file.
Perl has an I/O module (PerlIO::gzip) which makes reading gzipped files transparent to the rest of the program. I think python has something very similar (http://www.python.org/doc/lib/module-gzip.html).
If it's not too much to ask, would it be possible to add support for reading gzip'ed files ? At least in the peek/preview window ?
We'll certainly take this under consideration. Galaxy currently does support retrieving compressed files from external data sources ( UCSC ) as well as uploading them via the upload utility. However, they are currently decompressed on-the-fly. Allowing them to remain compressed would require tools to decompress them - we'll see if maybe this makes sense for some tools.
Comments are welcomed,
Thanks, Gordon. _______________________________________________ galaxy-user mailing list galaxy-user@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user