- galaxy-user - lists.galaxyproject.org

Storing compressed data files
by Assaf Gordon 10 Jun '08

10 Jun '08

Hello, As users add more and more data files to our galaxy server, disk space becomes a problem... What I'd ultimately like is to store the data files in some compressed manner (at least some of the textual files), how would you suggest to do that ? A common scenario is: 1. User uploads a big Fastq/solexa file (=> 1.2 GB) 2. FASTQ file converted to FASTA file (=> 0.6 GB) 3. FASTA file trimmed, clipped, stripped, etc. (=> 100 MB) 4. BLAT, Histograms and other reports (=> ~50 MB) The first three data sets take about 1.9 GB of disk space - and aren't really needed by the user (as he/she is mostly interested in the resulting report files). Since these are textual files, they compress really well. Currently, I store the FASTQ gzip'ed in galaxy, and my tools know how to read gzip'ed data. There are two shortcomings with this method: 1. datasets (green squares) of gzip'ed files don't display any data in the peek window 2. Other galaxy tools which require FASTQ file as input can't read my file. Perl has an I/O module (PerlIO::gzip) which makes reading gzipped files transparent to the rest of the program. I think python has something very similar (http://www.python.org/doc/lib/module-gzip.html) If it's not too much to ask, would it be possible to add support for reading gzip'ed files ? At least in the peek/preview window ? Comments are welcomed, Thanks, Gordon.

2 1

Two Administration questions
by Assaf Gordon 10 Jun '08

10 Jun '08

Hello, I have two administration-related questions: 1. I have a tool which uses dynamic options from an external file. A cron script updates this file, but the updated data doesn't appear in the tool's list-box until galaxy is restarted. Is there a way to re-load an option file without restarting galaxy? 2. In order to do some routine galaxy maintenance, I need to stop the daemon - Is there way to make sure no jobs are currently running before stopping the daemon? Thanks, Gordon.

2 1

Re: [galaxy-user] Storing compressed data files
by Asim Siddiqui 06 Jun '08

06 Jun '08

Hi Assaf, An alternative approach is to utilize a binary file format specifically designed for sequence data in a compact manner. An example of that is Sequence Read Format (SRF). SRF has been incorporated into the Illumina and Helicos pipelines and will be available for the AB platform shortly. SRF includes support for compression using several schemes including ZLIB. This thread has been captured on the genographia website and I've commented there. There is also a link to more information on SRF. Note: in terms of implementation, there is a C version (most complete), a C++ prototype (with a complete C++ implementation coming soon) and an early Java implementation. http://www.genographia.org/portal/topics/sequence-read-format-srf/galaxy-and -file-size-management http://www.genographia.org/portal/topics/sequence-read-format-srf/sequence-r ead-format-srf Asim ------ Forwarded Message From: Assaf Gordon <gordon(a)cshl.edu> Date: Wed, 4 Jun 2008 16:26:30 -0700 To: <galaxy-user(a)bx.psu.edu> Subject: [galaxy-user] Storing compressed data files Hello, As users add more and more data files to our galaxy server, disk space becomes a problem... What I'd ultimately like is to store the data files in some compressed manner (at least some of the textual files), how would you suggest to do that ? A common scenario is: 1. User uploads a big Fastq/solexa file (=> 1.2 GB) 2. FASTQ file converted to FASTA file (=> 0.6 GB) 3. FASTA file trimmed, clipped, stripped, etc. (=> 100 MB) 4. BLAT, Histograms and other reports (=> ~50 MB) The first three data sets take about 1.9 GB of disk space - and aren't really needed by the user (as he/she is mostly interested in the resulting report files). Since these are textual files, they compress really well. Currently, I store the FASTQ gzip'ed in galaxy, and my tools know how to read gzip'ed data. There are two shortcomings with this method: 1. datasets (green squares) of gzip'ed files don't display any data in the peek window 2. Other galaxy tools which require FASTQ file as input can't read my file. Perl has an I/O module (PerlIO::gzip) which makes reading gzipped files transparent to the rest of the program. I think python has something very similar (http://www.python.org/doc/lib/module-gzip.html) If it's not too much to ask, would it be possible to add support for reading gzip'ed files ? At least in the peek/preview window ? Comments are welcomed, Thanks, Gordon. _______________________________________________ galaxy-user mailing list galaxy-user(a)bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user ------ End of Forwarded Message

1 0

galaxy query
by Whiteley, Gareth 05 Jun '08

05 Jun '08

Hello, If I was to choose the following search criteria Group:all tracks, track:SNP, region: chr5:7922217-7954237, then click INTERSECT and choose Group:all tracks, track:TFBS conserved and select GTF output format. This example searches for any SNPs that are in the Transcription factor binding sites within the genomic region that codes for the MTRR gene, but the output does not tell me that the SNP rs6868871 is found in the TFBS V$58_01 914, I have to work that out for myself by then searching the whole table again but from a TFBS start point and intersecting with SNPs. I am trying to use galaxy to Join two Queries side by side on a specified field. I am trying to relate SNPs to the TFBS they are associated with. For example, SNP rs6868871 is found in TFBS V$58_01 914. However, i can not seem to get galaxy to work, i think this is because the SNP site is something like this '165878639 - 165878640' and a TFBS site is something like this '165878630 - 165878641' and although the positions overlap, galaxy can not tell that. Is this the case? Or do you know of how i can get around it? Regards, Gareth Whiteley Gareth Whiteley University of Liverpool Department of Pharmacology and Therapeutics The Sherrington Buildings Ashton Street Liverpool L69 7GE Tel: 0151 795 4224 E-mail: g.whiteley(a)liverpool.ac.uk

2 1

admin password?
by Davide Cittaro 04 Jun '08

04 Jun '08

Can anybody use the admin tool to reload modules? I've set the password in universe_wsgi.ini file but I always get an "Invalid password" error... thanks d /* Davide Cittaro Cogentech - Consortium for Genomic Technologies via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro(a)ifom-ieo-campus.it */

2 1

TFBS search
by Tony Oosterveen 27 May '08

27 May '08

Hello, I am just starting to use galaxy, and I was wondering if there is a simple way to search genome wide for clustering of custom transcription binding sites (preferentially with weight matrixes). All other programs (~10) I have used till so far have their limitations, either the length of the sequence they can handle is rather limited, they use only in house transcription factor binding sites or they are not able to check for evolutionary conservation. Thanks, Tony Tony Oosterveen Karolinska Institute Stockholm Sweden

1 0

Looking For Tool DTD
by Scott Kuehn 24 May '08

24 May '08

Hello, I'm looking for an appropriate DTD to use as a basis for designing new tools (section 3, here: http://g2.trac.bx.psu.edu/wiki/AddToolTutorial) Can someone point me toward this? I looked on the website and the latest source distribution, and could not find anything. Thanks, Scott Scott Kuehn University of Washington Medical Genetics 2211 Elliot: 206-267-1091 x215

2 1

Sharing HIstories?
by Jacob Biesinger 13 May '08

13 May '08

Is it possible to share histories with other collaborators? I'd like them to be able to see my data without having to step through the same steps as I did to generate them. Can this be done in galaxy?

2 1

filter of 3'UTR conserved sequences
by Naum Gershenzon 05 May '08

05 May '08

Dear Sir/Madam, I figured out how to retrieve all conserved human 3'UTR sequences from Table Browser, however they came with multiple repetition (up to 5 times). So my question is “How can I filter these data in order to have all conserved human 3'UTR sequences WITHOUT DUPLICATION THE SAME SEQUEMCES MULTIPLE TIMES?” Sincerely Naum Gershenzon

2 2

feature requests
by Belinda M. Giardine 02 May '08

02 May '08

UCSC Genome Browser output with frame on left with links. Able to upload as custom track at browser of choice eg genome-test or dev browser, watson browser?, ... implemented as url you can point your browser at? Featured datasets eg hard to load dbSNP, too large, keep local? easy access to Ross and other local datasets New Tools To add chr for chroms (Ensembl doesn't include chr) To get uniq rows from a dataset (cat sort uniq) New join choice in Operate on Genomic Intervals -treat interval fields separate so can keep interval file on full join (will work with exact matches to intervals) History - A way of organizing the history into sections/folders, can't just use new histories without reloading core queries, when history is large it is hard to find which query you want when many similar operations done, - A way to archive a history, keep loaded datasets and how to run operations (what was done) but not store all results indefinitely. - When you open a history item could the display not return to the top.

1 0