August 2011 - galaxy-dev - lists.galaxyproject.org

[API] Error message forwarding
by Louise-Amélie Schmitt 29 Aug '11

29 Aug '11

Hi, I ran into a little something that is a bit annoying for debug when trying to upload files through the API with library_upload_from_import_dir.py. When the specified folder is wrong, python tries to process the error tuple like a dict, so the original error is hard to find. I modified a little the code to avoid that. It might not work in all cases but at least it solved my problem. Here is the original code: lib/galaxy/web/api/contents.py l.145->end else: rval = [] for k, v in output.items(): if type( v ) == trans.app.model.LibraryDatasetDatasetAssociation: v = v.library_dataset encoded_id = trans.security.encode_id( create_type + '.' + str( v.id ) ) rval.append( dict( id = encoded_id, name = v.name, url = url_for( 'content', library_id=library_id, id=encoded_id ) ) ) return rval Here is how I modified it: else: rval = [] try: for k, v in output.items(): if type( v ) == trans.app.model.LibraryDatasetDatasetAssociation: v = v.library_dataset encoded_id = trans.security.encode_id( create_type + '.' + str( v.id ) ) rval.append( dict( id = encoded_id, name = v.name, url = url_for( 'content', library_id=library_id, id=encoded_id ) ) ) return rval except: log.debug( 'HTTP Error %s: %s' % ( output[0], output[1] ) ) return None Since it saves me some time & energy I just wanted to share it, so I hope it can help. Best L-A

2 1

next galaxy-dist update
by Glen Beane 29 Aug '11

29 Aug '11

We are in need of a galaxy upgrade (we missed the last galaxy-dist update at the end of June). However, we don't want to update to a 2 month old release only to have a new release come out shortly after. Could someone on the development team give me an estimate of when the next galaxy-dist update will happen? -- Glen L. Beane Senior Software Engineer The Jackson Laboratory (207) 288-6153

2 2

contribute tools to galaxy?
by Dongjun Chung 28 Aug '11

28 Aug '11

Hi All, I'm a newbie to galaxy and enjoying it a lot these days. Thanks for the great work. I have a question regarding contribution of software to galaxy. We developed a ChIP-seq peak calling algorithm and software (R package) and hope to contribute it to galaxy. I have read the wiki and prior mailing list about the contribution system but it is still somewhat confusing to me. 1. It seems that I can contribute our software to "tool shed" if I prepare appropriate code & definition files. Then, users can download and use it with their locally installed galaxy. However, these files committed to tool shed will not appear in galaxy main or test servers. Am I correct? 2. What is clear relationship between main/test servers & tool shed? Can we contribute our software to main or test servers as well? Or only galaxy core developers can add new tools to galaxy main or test servers? If so, which software is considered to be added to servers? Are they chosen from tools contributed to tool shed? 3. If our software is a R package, then users need to download and install it first in their R system before they use our software within their galaxy system even in the case they have appropriate definition files. Am I right? Or is there any better solution for this? Thanks! Best, Dongjun

1 0

Database migration
by Jack Zhu 27 Aug '11

27 Aug '11

Hi all, I have problems with updating my local instance of Galaxy. Specifically I can not migrate the database schema from 80 to 81: -------------- $$ sh manage_db.sh upgrade 80 -> 81... Migration script to add a 'tool_version' column to the hda/ldda tables. Traceback (most recent call last): File "./scripts/manage_db.py", line 63, in <module> main( repository=repo, url=db_url ) File "/home/zhujack/bin/galaxy/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/shell.py", line 150, in main ret = command_func(**kwargs) File "/home/zhujack/bin/galaxy/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py", line 221, in upgrade return _migrate(url, repository, version, upgrade=True, err=err, **opts) File "/home/zhujack/bin/galaxy/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py", line 349, in _migrate schema.runchange(ver, change, changeset.step) File "/home/zhujack/bin/galaxy/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/schema.py", line 184, in runchange change.run(self.engine, step) File "/home/zhujack/bin/galaxy/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/py.py", line 101, in run func() File "lib/galaxy/model/migrate/versions/0081_add_tool_version_to_hda_ldda.py", line 17, in upgrade metadata.reflect() File "/home/zhujack/bin/galaxy/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/schema.py", line 1733, in reflect Table(name, self, **reflect_opts) File "/home/zhujack/bin/galaxy/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/schema.py", line 108, in __call__ return type.__call__(self, name, metadata, *args, **kwargs) File "/home/zhujack/bin/galaxy/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/schema.py", line 236, in __init__ _bind_or_error(metadata).reflecttable(self, include_columns=include_columns) File "/home/zhujack/bin/galaxy/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 1265, in reflecttable self.dialect.reflecttable(conn, table, include_columns) File "/home/zhujack/bin/galaxy/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/databases/mysql.py", line 1673, in reflecttable only=include_columns) File "/home/zhujack/bin/galaxy/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/databases/mysql.py", line 2139, in reflect self._add_column(table, line, charset, only) File "/home/zhujack/bin/galaxy/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/databases/mysql.py", line 2227, in _add_column type_instance = col_type(*type_args, **type_kw) File "/home/zhujack/bin/galaxy/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/databases/mysql.py", line 569, in __init__ super(MSBigInteger, self).__init__(display_width, **kw) TypeError: super(type, obj): obj must be an instance or subtype of type -------------- Your help will be greatly appreciated. Jack

1 0

Storing a dict as metadata
by Duddy, John 26 Aug '11

26 Aug '11

I'd like to have a datatype with a dict as metadata. This dict() would store file offsets to enable seeking around to process different sections of the file. How do I add a dictionary data metadata element? John Duddy Sr. Staff Software Engineer Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858-736-3584 E-mail: jduddy(a)illumina.com<mailto:jduddy@illumina.com>

2 5

using Galaxy for map/reduce
by Andrew Straw 26 Aug '11

26 Aug '11

Hi all, I've been investigating use of Galaxy for our lab and it has many attractive aspects -- a big thank you to all involved. We still have a couple of related sticking points, however, that I would like to get the Galaxy developers' feedback on. Basically, I want to use Galaxy to run Map/Reduce type analysis on many initial data files. What I mean is that I want to take many initial datasets (e.g. 250 or more), perhaps already stored in a library, and then apply a workflow to each and every one of them (the Map step). Then, on the many result datasets (one from each of the initial datasets), I want to run a Reduce step which creates a single dataset. I have achieved this in an imperfect and not-quite-working way with a few tricks, but I hope that with a little work, Galaxy could be much better for this type of use case. I have a couple of specific problems and a proposal for a general solution: 1) My first specific problem is that loading many datasets (e.g. 250) into history causes the javascript running locally withing a browser to be extremely slow. 2) My second specific problem is that applying a workflow with N steps to many datasets creates even more datasets (Nx250 additional datasets). In addition to the slow Javascript problem, there seems to be other issues I haven't diagnosed further, but the console in which I'm running run.sh indicates many errors of the type "Exception AssertionError: AssertionError('State <sqlalchemy.orm.state.MutableAttrInstanceState object at 0x7f5c18c47990> is not present in this identity map',) in <bound method MutableAttrInstanceState._cleanup of <sqlalchemy.orm.state.MutableAttrInstanceState object at 0x7f5c18c47990>> ignored". Furthermore the webserver gets slow and my nginx frontend proxy gives 504 gateway time-outs. 3) There's no good way to do reduce within Galaxy. Currently I work around this by having a tool type which takes as an input a dataset and then uploads this to a self-written webserver, which then collects such uploads, performs the reduce, and offers a download link for the user to collect the reduced dataset. The user must manually then upload this dataset back into Galaxy for further processing. My proposal for a general solution, and what I'd be interested in feedback on, is an idea of a "dataset container" (this is just a working name). It would look and act much like a dataset in the history, but would in fact be a logical construct that merely bundles together a homogeneous bunch of datasets. When a tool (or a workflow) is applied to a dataset container, Galaxy would automatically create a new container in which each dataset in this new container is the result of running the tool. (Workflows with N steps would thus generate N new containers.) The thing I like about this idea is that it preserves the ability to use tools and workflows on both individual datasets and, with some additional logic, on these new containers. In particular, I don't think the tools and workflows themselves would have to be modified. This would seemingly mitigate the slow Javascript issue by only showing a few items in the history window (even though Galaxy may have launched many jobs in the background). Furthermore, a new Reduce tool type could then act to take a dataset container as input and output a single dataset. A library doesn't seem a good candidate for the dataset container idea I have above. I realize that a library also bundles together datasets, but it has other attributes that don't play well with the above idea (the idea of hierarchically arranged folders and heterogeneous datasets) nor can it be represented in the history. I'm interested in thoughts on this proposal, as I think it would really help us, and I think our use case may be representative of what others might also like to do. I realize that in my text above I write "with some additional logic" to describe the work required to implement this idea, but the fact is that I have very little idea about how much work this would be. So, practically speaking, my question boils down to how hard would implementing this be, given the existing code base and goals? And, would such an implementation - if done to the taste of the Galaxy devs, of course - have a chance of making into the Galaxy distribution? Thanks, Andrew -- Andrew D. Straw, Ph.D. Research Institute of Molecular Pathology (IMP) Vienna, Austria http://strawlab.org/

6 8

Problem to link bam files when uploading through system paths
by Zhibin Lu 26 Aug '11

26 Aug '11

Hello, When i try to upload bam files into galaxy data library though system paths with "link to files without copying into galaxy" option, galaxy always complains that "The uploaded files need grooming, so change your*Copy data into Galaxy?*selection to be*Copy files into Galaxy*instead of*Link to files without copying into Galaxy*so grooming can be performed." no matter if the bam files are sorted or not. in galaxy-dist/tools/data_source/upload.py: 285: if dataset.type in ( 'server_dir', 'path_paste' ) and link_data_only == 'link_to_files': 286: # Never alter a file that will not be copied to Galaxy's local file store. 287: if datatype.dataset_content_needs_grooming( output_path ): 288: err_msg = 'The uploaded files need grooming, so change your Copy data into Galaxy? selection to be ' + \ 289: 'Copy files into Galaxy instead of Link to files without copying into Galaxy so grooming can be performed.' The 'output_path' is alwasy *None* when using *'link_to_files'*, so the grooming check always returns False. I think we need to check the original bam file instead at line 287 like: *287: if datatype.dataset_content_needs_grooming( databset.path ):* And the following code also needs to change: 314: if datatype.dataset_content_needs_grooming( output_path ): 315: # Groom the dataset content if necessary 316: datatype.groom_dataset_content( output_path ) A quick dirty fix would be: *314: if not (dataset.type in ( 'server_dir', 'path_paste' ) and link_data_only == 'link_to_files') and datatype.dataset_content_needs_grooming( output_path ): * Please advise, -- *Zhibin Lu* Bioinformatics Support *Ontario Institute for Cancer Research* MaRS Centre, South Tower 101 College Street, Suite 800 Toronto, Ontario, Canada M5G 0A3 Tel: 647-260-7944 Toll-free: 1-866-678-6427 www.oicr.on.ca <http://www.oicr.on.ca/>

2 1

downloading large files
by Edward Kirton 26 Aug '11

26 Aug '11

i thought i recalled reading about downloading files from a history via ftp, but i could been mistaken -- couldn't find anything on the wiki or mailing list archives. does this feature exist? what's the best way for users to download many or large files other than via the browser?

2 2

upload multiple files
by Chorny, Ilya 26 Aug '11

26 Aug '11

I noticed that when uploading multiple files only one job gets submitted to the queue. Is there a way for galaxy to split that up on multiple nodes? Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: ichorny(a)illumina.com<mailto:ichorny@illumina.com> Website: www.illumina.com<http://www.illumina.com>

2 1

Selective storage of galaxy files
by Dave Walton 26 Aug '11

26 Aug '11

Dear Galaxy developers, Our institution is trying solve our storage problem (we need lots, especially for NGS data, and someone needs to fund it). What we would like to be able to do, is based on some criteria control in what location a file gets written to disk. This criteria could be an individual user, a role or group they belong to, or a project the file is associated with. What we'd like to know are the following 3 things: 1) Is anyone already working on something like this? 2) Are there other institutions that would be interested in this type of functionality? 3) If we were to attempt to implement this ourselves, would anyone be interested in giving us some input with respect to how to implement and how to make it generic enough to meet the needs of most institutions? If we're going to do it, we'll need to be able to produce an estimate of what the effort would be like so that we could get institutional funding to develop the functionality. Thanks for any input you can provide. Dave -- Dave Walton Computational Sciences The Jackson Laboratory Bar Harbor, Maine

5 6