- galaxy-dev - lists.galaxyproject.org

Storing a dict as metadata
by Duddy, John 27 Aug '11

27 Aug '11

I'd like to have a datatype with a dict as metadata. This dict() would store file offsets to enable seeking around to process different sections of the file. How do I add a dictionary data metadata element? John Duddy Sr. Staff Software Engineer Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858-736-3584 E-mail: jduddy(a)illumina.com<mailto:jduddy@illumina.com>

2 5

using Galaxy for map/reduce
by Andrew Straw 27 Aug '11

27 Aug '11

Hi all, I've been investigating use of Galaxy for our lab and it has many attractive aspects -- a big thank you to all involved. We still have a couple of related sticking points, however, that I would like to get the Galaxy developers' feedback on. Basically, I want to use Galaxy to run Map/Reduce type analysis on many initial data files. What I mean is that I want to take many initial datasets (e.g. 250 or more), perhaps already stored in a library, and then apply a workflow to each and every one of them (the Map step). Then, on the many result datasets (one from each of the initial datasets), I want to run a Reduce step which creates a single dataset. I have achieved this in an imperfect and not-quite-working way with a few tricks, but I hope that with a little work, Galaxy could be much better for this type of use case. I have a couple of specific problems and a proposal for a general solution: 1) My first specific problem is that loading many datasets (e.g. 250) into history causes the javascript running locally withing a browser to be extremely slow. 2) My second specific problem is that applying a workflow with N steps to many datasets creates even more datasets (Nx250 additional datasets). In addition to the slow Javascript problem, there seems to be other issues I haven't diagnosed further, but the console in which I'm running run.sh indicates many errors of the type "Exception AssertionError: AssertionError('State <sqlalchemy.orm.state.MutableAttrInstanceState object at 0x7f5c18c47990> is not present in this identity map',) in <bound method MutableAttrInstanceState._cleanup of <sqlalchemy.orm.state.MutableAttrInstanceState object at 0x7f5c18c47990>> ignored". Furthermore the webserver gets slow and my nginx frontend proxy gives 504 gateway time-outs. 3) There's no good way to do reduce within Galaxy. Currently I work around this by having a tool type which takes as an input a dataset and then uploads this to a self-written webserver, which then collects such uploads, performs the reduce, and offers a download link for the user to collect the reduced dataset. The user must manually then upload this dataset back into Galaxy for further processing. My proposal for a general solution, and what I'd be interested in feedback on, is an idea of a "dataset container" (this is just a working name). It would look and act much like a dataset in the history, but would in fact be a logical construct that merely bundles together a homogeneous bunch of datasets. When a tool (or a workflow) is applied to a dataset container, Galaxy would automatically create a new container in which each dataset in this new container is the result of running the tool. (Workflows with N steps would thus generate N new containers.) The thing I like about this idea is that it preserves the ability to use tools and workflows on both individual datasets and, with some additional logic, on these new containers. In particular, I don't think the tools and workflows themselves would have to be modified. This would seemingly mitigate the slow Javascript issue by only showing a few items in the history window (even though Galaxy may have launched many jobs in the background). Furthermore, a new Reduce tool type could then act to take a dataset container as input and output a single dataset. A library doesn't seem a good candidate for the dataset container idea I have above. I realize that a library also bundles together datasets, but it has other attributes that don't play well with the above idea (the idea of hierarchically arranged folders and heterogeneous datasets) nor can it be represented in the history. I'm interested in thoughts on this proposal, as I think it would really help us, and I think our use case may be representative of what others might also like to do. I realize that in my text above I write "with some additional logic" to describe the work required to implement this idea, but the fact is that I have very little idea about how much work this would be. So, practically speaking, my question boils down to how hard would implementing this be, given the existing code base and goals? And, would such an implementation - if done to the taste of the Galaxy devs, of course - have a chance of making into the Galaxy distribution? Thanks, Andrew -- Andrew D. Straw, Ph.D. Research Institute of Molecular Pathology (IMP) Vienna, Austria http://strawlab.org/

6 8

Problem to link bam files when uploading through system paths
by Zhibin Lu 27 Aug '11

27 Aug '11

Hello, When i try to upload bam files into galaxy data library though system paths with "link to files without copying into galaxy" option, galaxy always complains that "The uploaded files need grooming, so change your*Copy data into Galaxy?*selection to be*Copy files into Galaxy*instead of*Link to files without copying into Galaxy*so grooming can be performed." no matter if the bam files are sorted or not. in galaxy-dist/tools/data_source/upload.py: 285: if dataset.type in ( 'server_dir', 'path_paste' ) and link_data_only == 'link_to_files': 286: # Never alter a file that will not be copied to Galaxy's local file store. 287: if datatype.dataset_content_needs_grooming( output_path ): 288: err_msg = 'The uploaded files need grooming, so change your <b>Copy data into Galaxy?</b> selection to be ' + \ 289: '<b>Copy files into Galaxy</b> instead of <b>Link to files without copying into Galaxy</b> so grooming can be performed.' The 'output_path' is alwasy *None* when using *'link_to_files'*, so the grooming check always returns False. I think we need to check the original bam file instead at line 287 like: *287: if datatype.dataset_content_needs_grooming( databset.path ):* And the following code also needs to change: 314: if datatype.dataset_content_needs_grooming( output_path ): 315: # Groom the dataset content if necessary 316: datatype.groom_dataset_content( output_path ) A quick dirty fix would be: *314: if not (dataset.type in ( 'server_dir', 'path_paste' ) and link_data_only == 'link_to_files') and datatype.dataset_content_needs_grooming( output_path ): * Please advise, -- *Zhibin Lu* Bioinformatics Support *Ontario Institute for Cancer Research* MaRS Centre, South Tower 101 College Street, Suite 800 Toronto, Ontario, Canada M5G 0A3 Tel: 647-260-7944 Toll-free: 1-866-678-6427 www.oicr.on.ca <http://www.oicr.on.ca/>

2 1

downloading large files
by Edward Kirton 27 Aug '11

27 Aug '11

i thought i recalled reading about downloading files from a history via ftp, but i could been mistaken -- couldn't find anything on the wiki or mailing list archives. does this feature exist? what's the best way for users to download many or large files other than via the browser?

2 2

upload multiple files
by Chorny, Ilya 27 Aug '11

27 Aug '11

I noticed that when uploading multiple files only one job gets submitted to the queue. Is there a way for galaxy to split that up on multiple nodes? Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: ichorny(a)illumina.com<mailto:ichorny@illumina.com> Website: www.illumina.com<http://www.illumina.com>

2 1

Selective storage of galaxy files
by Dave Walton 27 Aug '11

27 Aug '11

Dear Galaxy developers, Our institution is trying solve our storage problem (we need lots, especially for NGS data, and someone needs to fund it). What we would like to be able to do, is based on some criteria control in what location a file gets written to disk. This criteria could be an individual user, a role or group they belong to, or a project the file is associated with. What we'd like to know are the following 3 things: 1) Is anyone already working on something like this? 2) Are there other institutions that would be interested in this type of functionality? 3) If we were to attempt to implement this ourselves, would anyone be interested in giving us some input with respect to how to implement and how to make it generic enough to meet the needs of most institutions? If we're going to do it, we'll need to be able to produce an estimate of what the effort would be like so that we could get institutional funding to develop the functionality. Thanks for any input you can provide. Dave -- Dave Walton Computational Sciences The Jackson Laboratory Bar Harbor, Maine

5 6

LDAP & NGINX
by Holger Klein 27 Aug '11

27 Aug '11

Dear all, I'd like to integrate external user authentication with our local galaxy installation. So far we are running nginx as proxy, but all examples I found for LDAP and galaxy are for apache. Is anybody out there who has a working installation with LDAP and nginx? Cheers, Holger -- Dr. Holger Klein Core Facility Bioinformatics Institute of Molecular Biology gGmbH (IMB) http://www.imb-mainz.de/ Tel: +49(6131) 39 21511

2 1

Customizing/reusing the workflows/run.mako template
by Duddy, John 27 Aug '11

27 Aug '11

I'd like to integrate with Galaxy and launch workflows, and I am hoping to reuse Galaxy's support for prompting the user for items that are tagged as being set at runtime. It looks to me that I could load the workflows/run page in a frame on my app (on the same server) and reuse it. The problem I'm having is that I don't understand how the parameters for the input files are encoded. For example, here is one for adding a column to a dataset: <select name="22|input" last_selected_value="53"> <option value="53" selected>1: Add column on data 6</option> </select> <input type="hidden" name="22|__force_update__input" value="true" /> </div> <div style="clear: both"></div> </div> What are 22 and 53? They don't seem to correspond to dataset IDs. Is there a way that I can encode data library IDs such that I can submit this form and have Galaxy run the workflow with the right files in a new history? Thanks! John Duddy Sr. Staff Software Engineer Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858-736-3584 E-mail: jduddy(a)illumina.com<mailto:jduddy@illumina.com>

3 3

Re: [galaxy-dev] external display app
by Bram Slabbinck 27 Aug '11

27 Aug '11

Hi A small question regarding the setup of an external display application. At this moment, I have a display application that is linked to a particular datatype and everything shows up fine. In a second step, I want to make the visualisation more dynamic and therefore I need to have access to the input parameters of the tool that is used to create the particular data set. Is there a way to access these parameters from within my display application xml form? Maybe similar to the functionality of the info button of the dataset history item, which shows a list of all tool input parameters? I have looked at the according mako file and associated scripts, and the external display application wiki page but did not find a practical solution. As an alternative, I could put my tool parameters on top of my tool form and parse the value of the $data.info parameter (<param type="data">; column 'info' in the history_dataset_association table) within a <param type="template"> tag. This value shows however only the first x variables and, therefore, this approach is not a very decent way of parameter handling. thx for any suggestions Bram

1 0

Metadata error
by Chorny, Ilya 27 Aug '11

27 Aug '11

I get these two error in the log file. The job finishes correctly and a metadata file is created but I still get the error. Can anyone provide me with more detail about what this error means? The second error is incorrect because the files were deleted. setting metadata externally failed for HistoryDatasetAssociation 344: [Errno 1] Operation not permitted: '/mnt/data/galaxy/database/files/000' and Failed to cleanup MetadataTempFile temp files from ../../mnt/data/galaxy/database/tmp/metadata_out_HistoryDatasetAssociation_344_NUDu3A: No JSON object could be decoded: line 1 column 0 (char 0) Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: ichorny(a)illumina.com<mailto:ichorny@illumina.com> Website: www.illumina.com<http://www.illumina.com>

2 1