March 2012 - galaxy-dev - lists.galaxyproject.org

problem after last upgrade
by Michal Stuglik 16 Mar '12

16 Mar '12

Dear galaxy, I have upgraded my galaxy-dist with last official version ( upgrade: % hg pull -u -r 40f1816d6857 ). I also upgraded database (sh manage_db.sh upgrade). now I have problem with running galaxy instance and this is what I get: /home/galaxy/galaxy-dist/eggs/pysam-0.4.2_kanwei_b10f6e722e9a-py2.6-linux-x86_64-ucs4.egg/pysam/__init__.py:1: RuntimeWarning: __builtin__.file size changed, may indicate binary incompatibility from csamtools import * python path is: /home/galaxy/galaxy-dist/eggs/numpy-1.6.0-py2.6-linux-x86_64-ucs4.egg, /home/galaxy/galaxy-dist/eggs/pysam-0.4.2_kanwei_b10f6e722e9a-py2.6-linux-x86_64-ucs4.egg, /home/galaxy/galaxy-dist/eggs/boto-2.2.2-py2.6.egg, /home/galaxy/galaxy-dist/eggs/Whoosh-0.3.18-py2.6.egg, /home/galaxy/galaxy-dist/eggs/pycrypto-2.0.1-py2.6-linux-x86_64-ucs4.egg, /home/galaxy/galaxy-dist/eggs/python_lzo-1.08_2.03_static-py2.6-linux-x86_64-ucs4.egg, /home/galaxy/galaxy-dist/eggs/bx_python-0.7.1_7b95ff194725-py2.6-linux-x86_64-ucs4.egg, /home/galaxy/galaxy-dist/eggs/amqplib-0.6.1-py2.6.egg, /home/galaxy/galaxy-dist/eggs/pexpect-2.4-py2.6.egg, /home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg, /home/galaxy/galaxy-dist/eggs/Babel-0.9.4-py2.6.egg, /home/galaxy/galaxy-dist/eggs/Mako-0.4.1-py2.6.egg, /home/galaxy/galaxy-dist/eggs/WebHelpers-0.2-py2.6.egg, /home/galaxy/galaxy-dist/eggs/simplejson-2.1.1-py2.6-linux-x86_64-ucs4.egg, /home/galaxy/galaxy-dist/eggs/wchartype-0.1-py2.6.egg, /home/galaxy/galaxy-dist/eggs/elementtree-1.2.6_20050316-py2.6.egg, /home/galaxy/galaxy-dist/eggs/docutils-0.7-py2.6.egg, /home/galaxy/galaxy-dist/eggs/WebOb-0.8.5-py2.6.egg, /home/galaxy/galaxy-dist/eggs/Routes-1.12.3-py2.6.egg, /home/galaxy/galaxy-dist/eggs/Cheetah-2.2.2-py2.6-linux-x86_64-ucs4.egg, /home/galaxy/galaxy-dist/eggs/PasteDeploy-1.3.3-py2.6.egg, /home/galaxy/galaxy-dist/eggs/PasteScript-1.7.3-py2.6.egg, /home/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg, /home/galaxy/galaxy-dist/lib, /usr/lib/python2.6, /usr/lib/python2.6/plat-linux2, /usr/lib/python2.6/lib-tk, /usr/lib/python2.6/lib-old, /usr/lib/python2.6/lib-dynload, /usr/local/lib/python2.6/dist-packages, /usr/lib/python2.6/dist-packages/PIL, /usr/lib/pymodules/python2.6/gtk-2.0, /usr/lib/python2.6/dist-packages/gst-0.10, /usr/lib/python2.6/dist-packages/gtk-2.0, /usr/lib/pymodules/python2.6/ubuntuone-storage-protocol, /usr/lib/pymodules/python2.6/ubuntuone-client, /usr/lib/python2.6/dist-packages, /usr/lib/pymodules/python2.6 galaxy.model.migrate.check DEBUG 2012-03-14 16:29:23,928 psycopg2 egg successfully loaded for postgres dialect galaxy.model.migrate.check INFO 2012-03-14 16:29:24,000 At database version 93 galaxy.tool_shed.migrate.check DEBUG 2012-03-14 16:29:24,007 psycopg2 egg successfully loaded for postgres dialect Traceback (most recent call last): File "/home/galaxy/galaxy-dist/lib/galaxy/web/buildapp.py", line 82, in app_factory app = UniverseApplication( global_conf = global_conf, **kwargs ) File "/home/galaxy/galaxy-dist/lib/galaxy/app.py", line 37, in __init__ verify_tools( self, db_url, kwargs.get( 'global_conf', {} ).get( '__file__', None ), self.config.database_engine_options ) File "/home/galaxy/galaxy-dist/lib/galaxy/tool_shed/migrate/check.py", line 42, in verify_tools db_schema = schema.ControlledSchema( engine, migrate_repository ) File "/home/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/schema.py", line 24, in __init__ self._load() File "/home/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/schema.py", line 42, in _load data = list(result)[0] IndexError: list index out of range best, michal

2 4

Compare two datasets fails due to encoding error
by Mikel Egaña Aranguren 16 Mar '12

16 Mar '12

Hi; I have developed a Perl tool that generates a tabular file with 3 columns. I need to compare it with another tabular file (Gene Ontology Annotation file) and I use "Join, substract, group", "Compare two datasets". When I try to do so, I get the error pasted bellow. I assume my perl code is generating the file with the wrong encoding, but I have tried to generate the file in UTF8 to no avail. Any clues? Thanks Traceback (most recent call last): File "/home/megana/galaxy-dist/lib/galaxy/jobs/runners/local.py", line 125, in run_job job_wrapper.finish( stdout, stderr ) File "/home/megana/galaxy-dist/lib/galaxy/jobs/__init__.py", line 658, in finish self.sa_session.flush() File "/home/megana/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/scoping.py", line 127, in do return getattr(self.registry(), name)(*args, **kwargs) File "/home/megana/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/session.py", line 1356, in flush self._flush(objects) File "/home/megana/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/session.py", line 1434, in _flush flush_context.execute() File "/home/megana/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/unitofwork.py", line 261, in execute UOWExecutor().execute(self, tasks) File "/home/megana/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/unitofwork.py", line 753, in execute self.execute_save_steps(trans, task) File "/home/megana/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/unitofwork.py", line 773, in execute_save_steps self.execute_cyclical_dependencies(trans, task, False) File "/home/megana/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/unitofwork.py", line 789, in execute_cyclical_dependencies self.execute(trans, [t], isdelete) File "/home/megana/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/unitofwork.py", line 753, in execute self.execute_save_steps(trans, task) File "/home/megana/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/unitofwork.py", line 768, in execute_save_steps self.save_objects(trans, task) File "/home/megana/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/unitofwork.py", line 759, in save_objects task.mapper._save_obj(task.polymorphic_tosave_objects, trans) File "/home/megana/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/mapper.py", line 1413, in _save_obj c = connection.execute(statement.values(value_params), params) File "/home/megana/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 824, in execute return Connection.executors[c](self, object, multiparams, params) File "/home/megana/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 874, in _execute_clauseelement return self.__execute_context(context) File "/home/megana/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 896, in __execute_context self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context) File "/home/megana/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 950, in _cursor_execute self._handle_dbapi_exception(e, statement, parameters, cursor, context) File "/home/megana/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 931, in _handle_dbapi_exception raise exc.DBAPIError.instance(statement, parameters, e, connection_invalidated=is_disconnect) ProgrammingError: (ProgrammingError) You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings. u'UPDATE history_dataset_association SET update_time=?, info=?, blurb=?, peek=?, tool_version=?, metadata=? WHERE history_dataset_association.id = ?' ['2012-03-16 15:20:29.158287', 'join (GNU coreutils) 7.4\nCopyright \xc2\xa9 2009 Free Software Foundation, Inc.\nLicencia GPLv3+: GNU GPL versi\xc3\xb3n 3 o superior <http://gnu.org/licenses/gpl.html>.\nEsto es software libre: es libre de cambiarlo y redistribuirlo.\nNO HAY NINGUNA GARANT\xc3\x8dA, en la me', '5,547 lines', u'UniProtKB\tB5B2P3\tNFATC2\t\tGO:0001816\tGO_REF:0000019\tIEA\tEnsembl:ENSMUSP00000074198\tP\tNuclear factor of activated T-cells c2 isoform IB-IIL-deltaXa\tB5B2P3_HUMAN|NFATC2|hCG_2018585|IPI00297845\tprotein\ttaxon:9606\t20110709\tENSEMBL\n\nUniProtKB\tE7DVW4\tFABP4\t\tGO:0001816\tGO_REF:0000019\tIEA\tEnsembl:ENSMUSP00000029041\tP\tFatty acid binding protein 4, adipocyte\tE7DVW4_HUMAN|FABP4|IPI00215746\tprotein\ttaxon:9606\t20110709\tENSEMBL\n\nUniProtKB\tE9PHR7\tNFAT5\t\tGO:0001816\tGO_REF:0000019\tIEA\tEnsembl:ENSMUSP00000127784\tP\tUncharacterized protein\tE9PHR7_HUMAN|NFAT5|IPI00514115\tprotein\ttaxon:9606\t20110709\tENSEMBL\n\nUniProtKB\tE9PJN7\tZCCHC11\t\tGO:0001816\tGO_REF:0000019\tIEA\tEnsembl:ENSMUSP00000044836\tP\tUncharacterized protein\tE9PJN7_HUMAN|ZCCHC11|IPI00985026\tprotein\ttaxon:9606\t20110709\tENSEMBL\n\nUniProtKB\tE9PQS7\tZCCHC11\t\tGO:0001816\tGO_REF:0000019\tIEA\tEnsembl:ENSMUSP00000044836\tP\tUncharacterized protein\tE9PQS7_HUMAN|ZCCHC11|IPI00975697\tprotein\ttaxon:9606\t20110709\tENSEMBL\n\nUniProtKB\tP01730\tCD4\t\tGO:0001816\tGO_REF:0000019\tIEA\tEnsembl:ENSMUSP00000024044\tP\tT-cell surface glycoprotein CD4\tCD4_HUMAN|CD4|IPI00003983|B2R737|D3DUS5|Q4ZGK2|Q5U066|Q9UDE5\tprotein\ttaxon:9606\t20110709\tENSEMBL\n', '', <read-only buffer for 0x7ad9210, size -1, offset 0 at 0x7f7054536770>, 118] -- Mikel Egaña Aranguren, PhD http://mikeleganaaranguren.com Marie Curie post-doc at Ontology Engineering Group, UPM http://www.oeg-upm.net/

1 0

Error Setting BAM Metadata
by Liisa Koski 16 Mar '12

16 Mar '12

Hello, I am trying to upload BAM files (by pasting a URL) to my history(or DataLibrary) and get the following error. These are bam files which I had previously uploaded with no problems. Traceback (most recent call last): File "/doolittle/Galaxy/galaxy_dist/lib/galaxy/jobs/runners/local.py", line 126, in run_job job_wrapper.finish( stdout, stderr ) File "/doolittle/Galaxy/galaxy_dist/lib/galaxy/jobs/__init__.py", line 618, in finish dataset.set_meta( overwrite = False ) File "/doolittle/Galaxy/galaxy_dist/lib/galaxy/model/__init__.py", line 874, in set_meta return self.datatype.set_meta( self, **kwd ) File "/doolittle/Galaxy/galaxy_dist/lib/galaxy/datatypes/binary.py", line 179, in set_meta raise Exception, "Error Setting BAM Metadata: %s" % stderr Exception: Error Setting BAM Metadata: [bam_header_read] EOF marker is absent. The input is probably truncated. [bam_header_read] invalid BAM binary header (this is not a BAM file) I ran bamtools on the unix command line to see if there was anything wrong with the file(s) but nothing. I tried uploading different bam files from other projects and get the same error. I did do an update to the latest release yesterday...if that helps? Thanks in advance, Liisa

2 3

CAGE data - trimming SAM files (functionality suggestion)
by Lukasz Jan Kielpinski 16 Mar '12

16 Mar '12

Hello, I work sometimes with galaxy analysing of short reads but actually I very much lack the function of keeping only first nucleotide of the read in SAM file. This is not so straightforward to do, I need to modifiy sequence, quality scores and CIGAR. It would be very useful for looking e.g on CAGE (cap analysis of gene expression) data where only 5' most of the read indicates end of mRNA molecule and rest of the read is needed only to map it to its location, then is unnecessary. Hope it can be easily implemented Best, Lukasz -- Lukasz J. Kielpinski PhD student Department of Biology, University of Copenhagen RNA Biology Group Ole Maaløes Vej 5, room 3.1.17 2200 Copenhagen N DENMARK

1 0

Local Toolshed misbehaviours?
by Daniel Sobral 16 Mar '12

16 Mar '12

Hello, I've installed a local toolshed server in http://myserver/toolshed/ (served by Apache) I am able to run it and create a repository with a tool. In my local galaxy server (http://myserver/galaxy/) I am able to see the repositories in that toolshed server, but when I try to install the tool, it gives me: abort: HTTP Error 404: Not Found . I can install tools from the main galaxy toolshed without problems. I've also uploaded a test tool to the test toolshed server http://testtoolshed.g2.bx.psu.edu and I could install the tools in my galaxy server. Any ideas of what I may be doing wrong here with my toolshed server? >From the galaxy Toolshed wiki I cannot see anything to give me a clue about what's going on... Thanks, Daniel

2 3

file upload of local data
by Christian Dreischer 16 Mar '12

16 Mar '12

Hello galaxy community, yesterday I installed a new Galaxy instance and followed the recommended steps (creating galaxy user account, galaxy mysql database, changes in universe_wsgi.ini, ...). It works fine except that there's an odd behaviour, concerning the file upload from my local file system. When I select a file for upload, the only thing I see is this blue upload arrow and nothing happens, even for files of a few kb. However, when I select the same file again, I gets uploaded instantely. I have do this for every new file. Do you have any suggestions? Thanks Christian

1 0

Depositing Data in a user's account
by Alex Kalderimis 16 Mar '12

16 Mar '12

Hi all, at InterMine we are looking to improve various aspects of our UI, and also our interaction with Galaxy. To that end, it would be great to find out what ways there are of depositing data in a user's account. The ideal would be to make requests from javascript that either open in another tab, or can be entirely backgrounded. After some reading of the bitbucket repos I note that: a) I can use the current mechanism of a data_source.py tool such as the flymine et al. ones. This would require me to do things like parsing the response headers for the galaxy session though, and I don't see a nice way of depositing data in a user's account without having come originally from galaxy. b) The API provides mechanisms for associating data files with histories, provided that these files have been already uploaded. It is straightforward to operate on a specific account, but I didn't find the API controller that allows individual tools, rather than whole workflows to be run. This would be ok I guess if I could create workflows rather than just run them. So the current situation (please correct me if I am mistaken) is a web-app based workflow that is best originated in galaxy, and an API workflow that lacks certain features to do what I'd like. Again, I'd very much like to be wrong here, and have like just not connected the dots correctly. However, if that is an accurate summation of the situation, I would be happy to contribute to get what I want, rather than expecting others to provide it. If anyone is interested in pull requests and a few questions about internals, I'd be happy to help develop the API so that: a) POST /workflows -> creates a new workflow (from one or more tools), rather than running it. b) GET /jobs -> lists the status of running or completed jobs. c) POST /jobs -> runs a job (from a workflow), returning a job id. d) DELETE /jobs -> cancels a job, or removes it from the history if completed. All best, Alex ------------------ Software Developer www.intermine.org University of Cambridge

1 0

Displaying BED files in IGV
by Ann Black-Ziegelbein 15 Mar '12

15 Mar '12

Hello - I am wanting to be able to display bed files from Galaxy in IGV. I have created a bed.xml display application description, and edited the datatypes_conf.xml to enable the igv display of datasets of bed format. However, when I select the link in Galaxy to "Display with IGV local" I get the following error message in Galaxy: You must import this dataset into your current history before you can view it at the desired display application. I am attaching my bed.xml file I created based off of existing IGV display types. I was hoping someone either has a working display descriptors for bed files for IGV, or might know what my error is. Thanks much, Ann

1 0

Error while uploading
by Luciano Cosme 15 Mar '12

15 Mar '12

Hi, I uploaded 11 files today (1 to 2Gb compressed files) to my local instance without a problem (as admin). Then I got this error message below and it get stuck at "This dataset is uploading" and I could not upload the last files. Is there anything I can do to solve it? What I notice is that I can still upload small files, before I concatenate them. I have 15 time points and some of them have up to 20 files, so it is easier to concatenate them and them upload. Thank you. Luciano 127.0.0.1 - - [10/Mar/2012:17:01:47 -0500] "POST /library_common/upload_library_dataset HTTP/1.1" 500 - " http://127.0.0.1:8080/library_common/upload_library_dataset?library_id=f597…" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" Debug at: http://127.0.0.1:8080/_debug/view/1331420472 ---------------------------------------- Exception happened during processing of request from ('127.0.0.1', 35033) Traceback (most recent call last): File "/home/koala2/galaxy-central/eggs/Paste-1.6-py2.7.egg/paste/httpserver.py", line 1053, in process_request_in_thread self.finish_request(request, client_address) File "/usr/lib/python2.7/SocketServer.py", line 323, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib/python2.7/SocketServer.py", line 639, in __init__ self.handle() File "/home/koala2/galaxy-central/eggs/Paste-1.6-py2.7.egg/paste/httpserver.py", line 432, in handle BaseHTTPRequestHandler.handle(self) File "/usr/lib/python2.7/BaseHTTPServer.py", line 343, in handle self.handle_one_request() File "/home/koala2/galaxy-central/eggs/Paste-1.6-py2.7.egg/paste/httpserver.py", line 427, in handle_one_request self.wsgi_execute() File "/home/koala2/galaxy-central/eggs/Paste-1.6-py2.7.egg/paste/httpserver.py", line 287, in wsgi_execute self.wsgi_start_response) File "/home/koala2/galaxy-central/eggs/Paste-1.6-py2.7.egg/paste/urlmap.py", line 202, in __call__ return app(environ, start_response) File "/home/koala2/galaxy-central/lib/galaxy/web/framework/middleware/xforwardedhost.py", line 21, in __call__ return self.app( environ, start_response ) File "/home/koala2/galaxy-central/eggs/PasteDeploy-1.3.3-py2.7.egg/paste/deploy/config.py", line 164, in __call__ app_iter = self.application(environ, start_response) File "/home/koala2/galaxy-central/lib/galaxy/web/framework/middleware/translogger.py", line 68, in __call__ return self.application(environ, replacement_start_response) File "/home/koala2/galaxy-central/eggs/WebError-0.8a-py2.7.egg/weberror/evalexception/middleware.py", line 226, in __call__ return self.respond(environ, start_response) File "/home/koala2/galaxy-central/eggs/WebError-0.8a-py2.7.egg/weberror/evalexception/middleware.py", line 399, in respond if self.xmlhttp_key in req.params: File "/home/koala2/galaxy-central/eggs/WebOb-0.8.5-py2.7.egg/webob/__init__.py", line 900, in params params = self.str_params File "/home/koala2/galaxy-central/eggs/WebOb-0.8.5-py2.7.egg/webob/__init__.py", line 892, in str_params return NestedMultiDict(self.str_GET, self.str_POST) File "/home/koala2/galaxy-central/eggs/WebOb-0.8.5-py2.7.egg/webob/__init__.py", line 818, in str_POST keep_blank_values=True) File "/usr/lib/python2.7/cgi.py", line 508, in __init__ self.read_multi(environ, keep_blank_values, strict_parsing) File "/usr/lib/python2.7/cgi.py", line 632, in read_multi environ, keep_blank_values, strict_parsing) File "/usr/lib/python2.7/cgi.py", line 510, in __init__ self.read_single() File "/usr/lib/python2.7/cgi.py", line 647, in read_single self.read_lines() File "/home/koala2/galaxy-central/lib/galaxy/web/framework/base.py", line 268, in read_lines self.read_lines_to_outerboundary() File "/usr/lib/python2.7/cgi.py", line 697, in read_lines_to_outerboundary line = self.fp.readline(1<<16) File "/home/koala2/galaxy-central/eggs/Paste-1.6-py2.7.egg/paste/httpserver.py", line 467, in readline data = self.file.readline(self.length - self._consumed) File "/usr/lib/python2.7/socket.py", line 412, in readline bline = buf.readline(size) OverflowError: signed integer is less than minimum ----------------------------------------

2 1

Reading user id before the execution of main tool
by M Khushi 15 Mar '12

15 Mar '12

Dear Galaxy Devs I would like to display some custom data that will load up dynamically based on logged in user and I could get simple dynamic_options working, however how would I grab logged in user id, before the execution of main tool. I hope there would be a way to do this. I have tried dynamic_options="load_dynamic_values($__user_id__) in xml wrapper but did not work. I know $__user_id__ can be passed to the tool as command line arguments, however, my issue is to read user id even before the execution of main tool. Thanks for your help guys. Regards Khushi

1 0