October 2011 - galaxy-dev - lists.galaxyproject.org

help with complicated conditional output format
by Sarah Diehl 06 Oct '11

06 Oct '11

Hello, I'm trying to write a wrapper for the intersectBed tool from the BEDTools suite. The difficulty lies in the possible combinations of input and output data formats. Input can be either bed or bam. When the input is bed, the output is also bed. However, when the input is bam the output can be either bam (default) or bed (by an additional command-line switch). I cannot figure out how to dynamically set the output format correctly. The wrapper always fails in at least one of the three cases. I collect the input as follows: <inputs> <conditional name="intype"> <param name="inselect" type="select" label="Select input file type"> <option value="bed" selected="true">BED, GFF</option> <option value="bam">BAM</option> </param> <when value="bed"> <param name="inputBed" type="data" format="bed,gff" label="Input A" help="Each feature in A is compared to B in search of overlaps."/> </when> <when value="bam"> <param name="inputBam" type="data" format="bam" label="Input A" help="Each BAM alignment in A is compared to B in search of overlaps."/> <param name="bed" type="boolean" truevalue="-bed" falsevalue="" checked="False" label="Write output as BED" help="The default is to write output in BAM when using BAM as input."/> </when> </conditional> <param name="input" type="data" format="bed" label="Input B"/> </inputs> I really want to keep the conditional and the select, because the program behaves slightly different depending on the input format. I tried to define the output like that: <outputs> <data name="output" format="input"/> <change_format> <when input="intype.bed" value="" format="bam"/> </change_format> </outputs> However, this fails when the input is bam and I want a bed output. I tried many different things I found in the wiki and on the mailing list, but nothing worked properly. I'm grateful for any ideas how to solve this problem. Thanks for your help in advance! Sarah

1 1

Galaxy tools with redirected python logging output get stuck past a certain amount of logging output
by Tim te Beek 06 Oct '11

06 Oct '11

Hi all, I've got some Python applications that I've hooked into Galaxy, which use the Python logging module for output. To capture this output in my Galaxy history I've created a wrapper that adds a logging.FileHandler to the root logger, which uses an Galaxy tool output argument as file to write to. This all works fine, up to a certain amount of logging output... I've created a proof of concept Galaxy tool that uses my method of capturing logging output (30 lines in total for tool & config) and added that to this email. When run with the default settings on my Galaxy instance this tool only writes 6625 lines of logging output out of 10,000, after which I have to manually kill the process because it gets stuck. Could you please look into why producing more than a certain amount of logging output can lead to a tool getting stuck, and tell me what I can do to resolve this? (Preferably without changing the way I capture output: I don't think my use case is that much of a stretch) Greets, Tim

1 0

disk space and file formats
by Patrick Page-McCaw 06 Oct '11

06 Oct '11

I'm not a bioinformaticist or programmer so apologies if this is a silly question. I've been occasionally running galaxy on my laptop and on the public server and I love it. The issue that I have is that my workflow requires many steps (what I do is probably very unusual). Each step creates a new large fastq file as the sequences are iteratively trimmed of junk. This fills my laptop and fills the public server with lots of unnecessary very large files. I've been thinking about the structure of the files and my workflow and it seems to me that a more space efficient system would be to have a single file (or a sql database) on which each tool can work. Most of what I do is remove adapter sequences, extract barcodes, trim by quality, map to the genome and then process my hits by type (exon, intron etc). Since the clean up tools in FASTX aren't written with my problem in mind, it takes several passes to get the sequences trimmed up before mapping. If I had a file that had a format something like (here as tab delimited): Header Seq Phred Start Len Barcode etc Each tool could read the Seq and Phred starting at Start and running Len nucleotides and work on that. The tool could then write a new Start and Len to reflect the trimming it has done[1]. For convenience let me call this an HSPh format. So it would be a real pain, no doubt, to rewrite all the tools. The little that I can read the tools it seems that the way the input is handled internally varies quite a bit. But it seems to me (naively?) that it would be relatively easy to write a conversion tool that would take the HSPh format and turn it into fastq or fast on the fly for the tools. Since most tools take fastq or fasta, it should be a write once, use many times, plugin. The harder (and slower) part would be mapping the fastq output back onto HSPh format. But again, this should be a write once, use for many tools plugin. Both of the intermediating files would be deleted when done. Just as a real quick test I thought I would see how long it takes to run sed on a fastq 1.35GB file and it was so fast on my laptop, < 2 minutes, that it was done before I noticed. Then as people are interested, the tools could be converted to take as input the new format. It may well be true in these days of $100 terabyte drives, this is not useful, that cycles are limiting, not drive space. But I think if the tools were rewritten to take and write to a HSPh format, processing would be faster too. It seems like some effort has been made to create the tab delimited format and maybe someone is already working on something like this (no doubt better designed). I may have a comp sci undergrad working in the lab this fall. With help we (well, he) might manage some parts of this. He is apparently quite a talented and hard working C++ programmer. Is it worth while? thanks [1] It could even do something like: Header Seq Phred Start Len Tool Parameter Start Len Tool Parameter Start Len etc Tool is the tool name, Parameter a list of parameters used, Start and Len would be the latest trim positions. And the last Start Len pair would be the one to use by default for the next tool, but this would keep an edit history without doubling the space needs with each processing cycle. I wouldn't need this but it might be more friendly for users, an "undo" means removing 4 columns. A format like this would probably be better as a sql database.

8 20

Bug importing from data library to history
by Lance Parsons 05 Oct '11

05 Oct '11

I'm running a local instance of Galaxy that has been working quite well. However, I've recently run into a problem when importing datasets from a data library into a history. The Size on Disk of the history grows by the size of the imported dataset, however, the new dataset is not visible (and the number of datasets does not change in the saved histories view). Any thoughts on what might be going on here? Thanks. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University

2 5

Bug in cufflinks_wrapper.xml?
by Curt Palm 05 Oct '11

05 Oct '11

I have been having trouble getting cufflinks to run with the "-g" or "-G" parameters via Galaxy. when I select either "use reference annotation" or "use reference annotation as guide" from the pulldown menu, cufflinks runs , but the "-g" or "-G" is not added to the cufflinks command. I think I have found the problem in the cufflinks_wrapper.xml file. line17 #if $reference_annotation.use_ref == "use reference annotation": line18 -G $reference_annotation.reference_annotation_file line19 #end if line20 #if $reference_annotation.use_ref == "use reference annotation guide": line21 -g $reference_annotation_guide.reference_annotation_guide_file ..... line56 <conditional name="reference_annotation"> line57 <param name="use_ref" type="select" label="Use Reference Annotation"> line58 <option value="No">No</option> line59 <option value="Use reference annotation">Use reference annotation</option> line60 <option value="Use reference annotation guide">Use reference annotation as guide</option> line61 </param> note that "Use" is capitalized in lines57 and 59 but not in lines 17 and 20. I made these changes in lines 17,20 and 21 and I now get the "-G" and "-g" are passed to the cuffflinks command "use" to "Use" in lines 17 and 20 reference_annotation_guide.reference_annotation_guide_file to "reference_annotation.reference_annotation_guide_file" in line 21 line17 #if $reference_annotation.use_ref == "Use reference annotation": line18 -G $reference_annotation.reference_annotation_file line19 #end if line20 #if $reference_annotation.use_ref == "Use reference annotation guide": line21 -g $reference_annotation.reference_annotation_guide_file -curt ******************************************************* Curtis J. Palm cpalm(a)stanford.edu Stanford Genome Technology Center MC: 8307 office: 650-812-1994 cell: 408 858-7849 *******************************************************

2 1

hg issue with resolve
by michael burrell (TOC) 05 Oct '11

05 Oct '11

Good afternoon all, I am having some issues with hg performing an update, it wants to use vimdiff and this is not the way I like to work. According to the mercurial simply placing the folloing in the ~/.hgrc should make mercurial change back to leaving the >>>> <<<<< marks when it cannot resolve something. Whatever I specify it still runs vimdiff. [ui] ui.merge = internal:merge any suggestions would be greatly received. Thanks Michael. PS, a little more info on the version of HG I am using. galaxy@jic55666:~/software/galaxy-central$ hg --version Mercurial Distributed SCM (version 1.6.4) ----------------------------------------------------------------------------------------------------- Please note, that my email address is no longer givenname.surname(a)bbsrc.ac.uk. Please update your records to reflect my new email address of givenname.surname(a)nbi.ac.uk as shown on this email.

2 2

Running the doctet unit tests in Galaxy
by Duddy, John 05 Oct '11

05 Oct '11

There are several files in Datatypes with doctest tests in them. Is there a convenient wrapper script to run them all? John Duddy Sr. Staff Software Engineer Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858-736-3584 E-mail: jduddy(a)illumina.com<mailto:jduddy@illumina.com>

2 1

Galaxy fronting multiple clusters
by Ann Black 05 Oct '11

05 Oct '11

Hello - I am working on standing up our own galaxy installation. We would like to have galaxy front multiple clusters, and I have some questions I was hoping someone could help with. 1) From reading other forum posts on this subject, it seems I need to minimally do the following ... is this correct?: A) have galaxy server w/ sge register as a job submitting host to the head node of each cluster B) Configure multiple tool runners for each tool per remote cluster? 2) When galaxy would submit a job, how would a backend remote cluster be selected? When running workflows, would the same cluster be used to run the entire workflow - or could the workflow then span remote clusters? 3) I am trying to understand some of the source code, where is the logic that would then dispatch the job and select a job runner to use? 4) Other advice or steps needed in order to get galaxy to front multiple remote clusters? Thanks so much, Ann

4 7

Reload a tool from the command line
by Jean-Baptiste Denis 05 Oct '11

05 Oct '11

Hello evyerbody, i guess this question has already been answered, but i didn't find any :/ Is there a way to reload a tool file from the command line ? I'd like to automate the process when a user upload a new version of an existing tool. Am I reinventing the wheel here, is there something already existing for this purpose ? Regards, Jean-Baptiste

2 1

odd behaviour of files from data libraries
by Bossers, Alex 05 Oct '11

05 Oct '11

Since some time we are observing strange things happening to using files from our local data libraries in our local galaxy-central instance: The files and hierarchy stay there but when copied to the history..... they turn up as being deleted and can only be used by selecting the view_deleted and undelete option. This didn't happen before. New libraries I make and adding files to it, do not show this behaviour. We already updated to the latest central version but it did not solve the issue. Could it have happened by an earlier update and got unnoticed? Has anyone else seen this behaviour on existing data libraries and if so, how did you fix it? Or some guides in how to debug it ourselves :-P Most of these files were linked to be used in galaxy instead of copied. The file locations and rights did not change (AFAIK). Thanks Alex

2 1