- galaxy-dev - lists.galaxyproject.org

disk space and file formats
by Patrick Page-McCaw 06 Oct '11

06 Oct '11

I'm not a bioinformaticist or programmer so apologies if this is a silly question. I've been occasionally running galaxy on my laptop and on the public server and I love it. The issue that I have is that my workflow requires many steps (what I do is probably very unusual). Each step creates a new large fastq file as the sequences are iteratively trimmed of junk. This fills my laptop and fills the public server with lots of unnecessary very large files. I've been thinking about the structure of the files and my workflow and it seems to me that a more space efficient system would be to have a single file (or a sql database) on which each tool can work. Most of what I do is remove adapter sequences, extract barcodes, trim by quality, map to the genome and then process my hits by type (exon, intron etc). Since the clean up tools in FASTX aren't written with my problem in mind, it takes several passes to get the sequences trimmed up before mapping. If I had a file that had a format something like (here as tab delimited): Header Seq Phred Start Len Barcode etc Each tool could read the Seq and Phred starting at Start and running Len nucleotides and work on that. The tool could then write a new Start and Len to reflect the trimming it has done[1]. For convenience let me call this an HSPh format. So it would be a real pain, no doubt, to rewrite all the tools. The little that I can read the tools it seems that the way the input is handled internally varies quite a bit. But it seems to me (naively?) that it would be relatively easy to write a conversion tool that would take the HSPh format and turn it into fastq or fast on the fly for the tools. Since most tools take fastq or fasta, it should be a write once, use many times, plugin. The harder (and slower) part would be mapping the fastq output back onto HSPh format. But again, this should be a write once, use for many tools plugin. Both of the intermediating files would be deleted when done. Just as a real quick test I thought I would see how long it takes to run sed on a fastq 1.35GB file and it was so fast on my laptop, < 2 minutes, that it was done before I noticed. Then as people are interested, the tools could be converted to take as input the new format. It may well be true in these days of $100 terabyte drives, this is not useful, that cycles are limiting, not drive space. But I think if the tools were rewritten to take and write to a HSPh format, processing would be faster too. It seems like some effort has been made to create the tab delimited format and maybe someone is already working on something like this (no doubt better designed). I may have a comp sci undergrad working in the lab this fall. With help we (well, he) might manage some parts of this. He is apparently quite a talented and hard working C++ programmer. Is it worth while? thanks [1] It could even do something like: Header Seq Phred Start Len Tool Parameter Start Len Tool Parameter Start Len etc Tool is the tool name, Parameter a list of parameters used, Start and Len would be the latest trim positions. And the last Start Len pair would be the one to use by default for the next tool, but this would keep an edit history without doubling the space needs with each processing cycle. I wouldn't need this but it might be more friendly for users, an "undo" means removing 4 columns. A format like this would probably be better as a sql database.

8 20

Bug importing from data library to history
by Lance Parsons 05 Oct '11

05 Oct '11

I'm running a local instance of Galaxy that has been working quite well. However, I've recently run into a problem when importing datasets from a data library into a history. The Size on Disk of the history grows by the size of the imported dataset, however, the new dataset is not visible (and the number of datasets does not change in the saved histories view). Any thoughts on what might be going on here? Thanks. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University

2 5

Bug in cufflinks_wrapper.xml?
by Curt Palm 05 Oct '11

05 Oct '11

I have been having trouble getting cufflinks to run with the "-g" or "-G" parameters via Galaxy. when I select either "use reference annotation" or "use reference annotation as guide" from the pulldown menu, cufflinks runs , but the "-g" or "-G" is not added to the cufflinks command. I think I have found the problem in the cufflinks_wrapper.xml file. line17 #if $reference_annotation.use_ref == "use reference annotation": line18 -G $reference_annotation.reference_annotation_file line19 #end if line20 #if $reference_annotation.use_ref == "use reference annotation guide": line21 -g $reference_annotation_guide.reference_annotation_guide_file ..... line56 <conditional name="reference_annotation"> line57 <param name="use_ref" type="select" label="Use Reference Annotation"> line58 <option value="No">No</option> line59 <option value="Use reference annotation">Use reference annotation</option> line60 <option value="Use reference annotation guide">Use reference annotation as guide</option> line61 </param> note that "Use" is capitalized in lines57 and 59 but not in lines 17 and 20. I made these changes in lines 17,20 and 21 and I now get the "-G" and "-g" are passed to the cuffflinks command "use" to "Use" in lines 17 and 20 reference_annotation_guide.reference_annotation_guide_file to "reference_annotation.reference_annotation_guide_file" in line 21 line17 #if $reference_annotation.use_ref == "Use reference annotation": line18 -G $reference_annotation.reference_annotation_file line19 #end if line20 #if $reference_annotation.use_ref == "Use reference annotation guide": line21 -g $reference_annotation.reference_annotation_guide_file -curt ******************************************************* Curtis J. Palm cpalm(a)stanford.edu Stanford Genome Technology Center MC: 8307 office: 650-812-1994 cell: 408 858-7849 *******************************************************

2 1

hg issue with resolve
by michael burrell (TOC) 05 Oct '11

05 Oct '11

Good afternoon all, I am having some issues with hg performing an update, it wants to use vimdiff and this is not the way I like to work. According to the mercurial simply placing the folloing in the ~/.hgrc should make mercurial change back to leaving the >>>> <<<<< marks when it cannot resolve something. Whatever I specify it still runs vimdiff. [ui] ui.merge = internal:merge any suggestions would be greatly received. Thanks Michael. PS, a little more info on the version of HG I am using. galaxy@jic55666:~/software/galaxy-central$ hg --version Mercurial Distributed SCM (version 1.6.4) ----------------------------------------------------------------------------------------------------- Please note, that my email address is no longer givenname.surname(a)bbsrc.ac.uk. Please update your records to reflect my new email address of givenname.surname(a)nbi.ac.uk as shown on this email.

2 2

Running the doctet unit tests in Galaxy
by Duddy, John 05 Oct '11

05 Oct '11

There are several files in Datatypes with doctest tests in them. Is there a convenient wrapper script to run them all? John Duddy Sr. Staff Software Engineer Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858-736-3584 E-mail: jduddy(a)illumina.com<mailto:jduddy@illumina.com>

2 1

Galaxy fronting multiple clusters
by Ann Black 05 Oct '11

05 Oct '11

Hello - I am working on standing up our own galaxy installation. We would like to have galaxy front multiple clusters, and I have some questions I was hoping someone could help with. 1) From reading other forum posts on this subject, it seems I need to minimally do the following ... is this correct?: A) have galaxy server w/ sge register as a job submitting host to the head node of each cluster B) Configure multiple tool runners for each tool per remote cluster? 2) When galaxy would submit a job, how would a backend remote cluster be selected? When running workflows, would the same cluster be used to run the entire workflow - or could the workflow then span remote clusters? 3) I am trying to understand some of the source code, where is the logic that would then dispatch the job and select a job runner to use? 4) Other advice or steps needed in order to get galaxy to front multiple remote clusters? Thanks so much, Ann

4 7

Reload a tool from the command line
by Jean-Baptiste Denis 05 Oct '11

05 Oct '11

Hello evyerbody, i guess this question has already been answered, but i didn't find any :/ Is there a way to reload a tool file from the command line ? I'd like to automate the process when a user upload a new version of an existing tool. Am I reinventing the wheel here, is there something already existing for this purpose ? Regards, Jean-Baptiste

2 1

odd behaviour of files from data libraries
by Bossers, Alex 05 Oct '11

05 Oct '11

Since some time we are observing strange things happening to using files from our local data libraries in our local galaxy-central instance: The files and hierarchy stay there but when copied to the history..... they turn up as being deleted and can only be used by selecting the view_deleted and undelete option. This didn't happen before. New libraries I make and adding files to it, do not show this behaviour. We already updated to the latest central version but it did not solve the issue. Could it have happened by an earlier update and got unnoticed? Has anyone else seen this behaviour on existing data libraries and if so, how did you fix it? Or some guides in how to debug it ourselves :-P Most of these files were linked to be used in galaxy instead of copied. The file locations and rights did not change (AFAIK). Thanks Alex

2 1

broken-pipe error on NGS Fastq tools "clip"
by Colleen Doherty 05 Oct '11

05 Oct '11

I am trying to set up a local instance. When running Clip adapter sequences I get the following error: fastx_clipper: Invalid quality score value (char 'J' ord 74 quality value 41) on line 36 gzip: stdout: Broken pipe I'm stuck on where to try to trouble-shoot this. This is on groomed fastq data in my history. I have the same error on other datasets. Other tools (Bowtie, FastQC) work on this dataset. I'm confused why it is gzip (but I clearly don't understand how Galaxy stores the data). Thanks for any suggestions!

2 2

prevent downloaded files with .txt added
by David Hoover 05 Oct '11

05 Oct '11

Does anyone know how to prevent the .txt suffix from being added to files downloaded from Galaxy? For example, a fasta file is downloaded as xxx.fasta.txt, instead of xxx.fasta. Wondering, David Hoover Helix Systems Staff http://helix.nih.gov

2 2