September 2011 - galaxy-dev - lists.galaxyproject.org

samtools mpileup update
by David Hoover 12 Oct '11

12 Oct '11

Is there any good reason why Galaxy can't use a more recent version of samtools? The only difference I can see is the mpileup command. The algorithm by which mpileup finds SNPs is a little different. Does anyone know if the file format has changed? The only files that need to change are sam_pileup.py and sam_pileup.xml. David Hoover Helix Systems Staff http://helix.nih.gov

4 3

disk space and file formats
by Patrick Page-McCaw 06 Oct '11

06 Oct '11

I'm not a bioinformaticist or programmer so apologies if this is a silly question. I've been occasionally running galaxy on my laptop and on the public server and I love it. The issue that I have is that my workflow requires many steps (what I do is probably very unusual). Each step creates a new large fastq file as the sequences are iteratively trimmed of junk. This fills my laptop and fills the public server with lots of unnecessary very large files. I've been thinking about the structure of the files and my workflow and it seems to me that a more space efficient system would be to have a single file (or a sql database) on which each tool can work. Most of what I do is remove adapter sequences, extract barcodes, trim by quality, map to the genome and then process my hits by type (exon, intron etc). Since the clean up tools in FASTX aren't written with my problem in mind, it takes several passes to get the sequences trimmed up before mapping. If I had a file that had a format something like (here as tab delimited): Header Seq Phred Start Len Barcode etc Each tool could read the Seq and Phred starting at Start and running Len nucleotides and work on that. The tool could then write a new Start and Len to reflect the trimming it has done[1]. For convenience let me call this an HSPh format. So it would be a real pain, no doubt, to rewrite all the tools. The little that I can read the tools it seems that the way the input is handled internally varies quite a bit. But it seems to me (naively?) that it would be relatively easy to write a conversion tool that would take the HSPh format and turn it into fastq or fast on the fly for the tools. Since most tools take fastq or fasta, it should be a write once, use many times, plugin. The harder (and slower) part would be mapping the fastq output back onto HSPh format. But again, this should be a write once, use for many tools plugin. Both of the intermediating files would be deleted when done. Just as a real quick test I thought I would see how long it takes to run sed on a fastq 1.35GB file and it was so fast on my laptop, < 2 minutes, that it was done before I noticed. Then as people are interested, the tools could be converted to take as input the new format. It may well be true in these days of $100 terabyte drives, this is not useful, that cycles are limiting, not drive space. But I think if the tools were rewritten to take and write to a HSPh format, processing would be faster too. It seems like some effort has been made to create the tab delimited format and maybe someone is already working on something like this (no doubt better designed). I may have a comp sci undergrad working in the lab this fall. With help we (well, he) might manage some parts of this. He is apparently quite a talented and hard working C++ programmer. Is it worth while? thanks [1] It could even do something like: Header Seq Phred Start Len Tool Parameter Start Len Tool Parameter Start Len etc Tool is the tool name, Parameter a list of parameters used, Start and Len would be the latest trim positions. And the last Start Len pair would be the one to use by default for the next tool, but this would keep an edit history without doubling the space needs with each processing cycle. I wouldn't need this but it might be more friendly for users, an "undo" means removing 4 columns. A format like this would probably be better as a sql database.

8 20

Bug importing from data library to history
by Lance Parsons 05 Oct '11

05 Oct '11

I'm running a local instance of Galaxy that has been working quite well. However, I've recently run into a problem when importing datasets from a data library into a history. The Size on Disk of the history grows by the size of the imported dataset, however, the new dataset is not visible (and the number of datasets does not change in the saved histories view). Any thoughts on what might be going on here? Thanks. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University

2 5

hg issue with resolve
by michael burrell (TOC) 05 Oct '11

05 Oct '11

Good afternoon all, I am having some issues with hg performing an update, it wants to use vimdiff and this is not the way I like to work. According to the mercurial simply placing the folloing in the ~/.hgrc should make mercurial change back to leaving the >>>> <<<<< marks when it cannot resolve something. Whatever I specify it still runs vimdiff. [ui] ui.merge = internal:merge any suggestions would be greatly received. Thanks Michael. PS, a little more info on the version of HG I am using. galaxy@jic55666:~/software/galaxy-central$ hg --version Mercurial Distributed SCM (version 1.6.4) ----------------------------------------------------------------------------------------------------- Please note, that my email address is no longer givenname.surname(a)bbsrc.ac.uk. Please update your records to reflect my new email address of givenname.surname(a)nbi.ac.uk as shown on this email.

2 2

Galaxy fronting multiple clusters
by Ann Black 05 Oct '11

05 Oct '11

Hello - I am working on standing up our own galaxy installation. We would like to have galaxy front multiple clusters, and I have some questions I was hoping someone could help with. 1) From reading other forum posts on this subject, it seems I need to minimally do the following ... is this correct?: A) have galaxy server w/ sge register as a job submitting host to the head node of each cluster B) Configure multiple tool runners for each tool per remote cluster? 2) When galaxy would submit a job, how would a backend remote cluster be selected? When running workflows, would the same cluster be used to run the entire workflow - or could the workflow then span remote clusters? 3) I am trying to understand some of the source code, where is the logic that would then dispatch the job and select a job runner to use? 4) Other advice or steps needed in order to get galaxy to front multiple remote clusters? Thanks so much, Ann

4 7

Reload a tool from the command line
by Jean-Baptiste Denis 05 Oct '11

05 Oct '11

Hello evyerbody, i guess this question has already been answered, but i didn't find any :/ Is there a way to reload a tool file from the command line ? I'd like to automate the process when a user upload a new version of an existing tool. Am I reinventing the wheel here, is there something already existing for this purpose ? Regards, Jean-Baptiste

2 1

prevent downloaded files with .txt added
by David Hoover 04 Oct '11

04 Oct '11

Does anyone know how to prevent the .txt suffix from being added to files downloaded from Galaxy? For example, a fasta file is downloaded as xxx.fasta.txt, instead of xxx.fasta. Wondering, David Hoover Helix Systems Staff http://helix.nih.gov

2 2

iRODS integration
by James Vincent 03 Oct '11

03 Oct '11

Hello, I've read about a number of side projects that integrate iRODS or other file handling things into Galaxy. What are the chances Galaxy could be made to use iRODS as it's primary file store? Instead of storing all files in some local space on a machine, Galaxy would call and act upon iRODS objects. I can think of many, many benefits of such a system, but is there any chance? Jim

2 1

(Composite) Dataset Upload not Setting Metadata
by Paniagua, Eric 03 Oct '11

03 Oct '11

Hi everyone, I've been getting my feet wet with Galaxy development working to get some of the rexpression tools online, and I've run into a snag that I've traced back to a set_meta datatype method not being able to find a file from which it wants to extract metadata. After reading the code, I believe this would also be a problem for non-composite datatypes. The specific test case I've been looking at is uploading an affybatch file (and associated pheno file) using Galaxy's built-in upload tool and selecting the File Format manually (ie choosing "affybatch" in the dropdown). I am using unmodified datatype definitions provided in lib/galaxy/datatypes/genetics.py and unmodified core Galaxy upload code as of 5955:949e4f5fa03a. (I am also testing with modified versions, but I am able to reproduce and track this bug in the specified clean version). The crux of the cause of error is that in JobWrapper.finish(), dataset.set_meta() is called (lib/galaxy/jobs/__init__.py:607) before the composite dataset uploaded files are moved (in a call to a Tool method "self.tool.collect_associated_files(out_data, self.working_directory)" on line 670) from the job working directory to the final destination under config.file_path (which defaults to "database/files"). In my test case, "database.set_meta( overwrite = False )" eventually calls lib/galaxy/datatypes/genetics.py:Rexp.set_meta(dataset, **kwd). As far as I can tell, the only ways to construct a path to a file (or the file) in a dataset without using hard-coded paths from external knowledge is to use the Dataset.get_file_name or Dataset.extra_files_path properties. Unless explicitly told otherwise, both of these methods construct a path based on the Dataset.file_path class data member, whose value is set during Galaxy startup to config.file_path (default "database/files"). However, at the time set_meta is called in this case, the files are not under config.file_path, but rather under the job working directory. Attempting to open files from the dataset therefore fails when using these paths. However, unless the job working directory is passed to set_meta or during construction of the underlying Dataset object, there doesn't appear to be a way for a Dataset method to access the currently running job (for instance to get its job ID or working directory). (The second suggestion is actually not possible; since the standard upload is asynchronous, the Dataset object is created (and persisted) before the Job that will process it is created.) Thoughts? This issue affects Rexp.set_peek also, as well as any other functions that may want to read data from the uploaded files before they are moved to permanent location. This is why if you have an affybatch file and its associated pheno file and you test this on, say, the public Galaxy server at http://main.g2.bx.psu.edu/ you'll see that the peek info says (for example): "##failed to find /galaxy/main_database/files/002/948/dataset_2948818_files/affybatch_test.pheno" It seems that if the current way that Dataset.file_path, Dataset.file_name, and Dataset.extra_files_path is part of the desired design of Galaxy, that methods like set_meta should be run after the files have been moved to config.file_path so they can set metadata based on file contents. It looks like this is intended to happen at least in some cases, from looking at lib/galaxy/jobs/__init__.py:568-586. However, in my tests this code is not kicking in because hda_tool_output is None. Any clarification on what's happening here, what's supposed to be happening for setting metadata on (potentially composite) uploads, why dataset.set_meta() isn't already being called after the files are moved to config.file_path, or any insights on related Galaxy design decisions I may not know about or design constraints I may have missed would be very greatly appreciated. I'd also be glad to provide further detail or test files upon request. Thank you, Eric Paniagua PS: Further notes on passing the job working directory to set_meta or set_peek - I have been successful modifying the code to do this for set_meta since the call chain starting from dataset.set_meta() in JobWrapper.finish() to Rexp.set_meta() accepts and forwards keyword argument dictionaries along the way. However, set_peek does not accept arbitrary keyword arguments, making it harder to pass along the job working directory when needed without stepping on the toes of any other code.

3 12

BWA built-in indexes in Galaxy
by Bruno Zeitouni 03 Oct '11

03 Oct '11

Dear all, I would like to use our built-in color-spaces indexes for BWA with Galaxy but I failed to do it. I added the following line in the "bwa_index_color.loc" file of the "tool-data" directory : hg19 hg19 hg19 my_path_to/hg19.fa but I don't see the reference hg19 appeared onto the browser... ("Map with BWA for SOLiD") Maybe I missed something here because there is a note about the original path for hg19 for workflow backwards compatibility (?), but I don't understand what it means.. For the BFAST indexes, It is OK. Thank you for your help Bruno -- Bruno Zeitouni, PhD Institut Curie - U900 INSERM - Mines ParisTech Equipe Bioinformatique Bâtiment de Biologie du Développement 26 rue d'ULM 75248 Paris Cedex 05 - FRANCE Tél : +33 1 56 24 69 31 Mail : bruno.zeitouni(a)curie.fr

2 1