- galaxy-dev - lists.galaxyproject.org

Database migration error
by Bossers, Alex 12 Oct '12

12 Oct '12

After a fresh galaxy-dist pull today (no update for various reasons) the instance was running ok on the default sqlight database. When I migrated back the setting to use the MySQL database it complained during startup about having found an old version of the database and I need to upgrade. Database upgrade started and the first few migrations were successful, but now it hangs every time at 99->100. Any clue what might be wrong? Here is the lengthy trace log. thanks Alex ==================================================== 99 -> 100... Migration script to alter the type of the tool_dependency.version column from TrimmedString(40) to Text. Traceback (most recent call last): File "./scripts/manage_db.py", line 63, in <module> main( repository=repo, url=db_url ) File "/opt/galaxy_dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/shell.py", line 150, in main ret = command_func(**kwargs) File "/opt/galaxy_dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/api.py", line 221, in upgrade return _migrate(url, repository, version, upgrade=True, err=err, **opts) File "/opt/galaxy_dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/api.py", line 349, in _migrate schema.runchange(ver, change, changeset.step) File "/opt/galaxy_dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/schema.py", line 184, in runchange change.run(self.engine, step) File "/opt/galaxy_dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/script/py.py", line 101, in run func() File "lib/galaxy/model/migrate/versions/0100_alter_tool_dependency_table_version_column.py", line 30, in upgrade ToolDependency_table = Table( "tool_dependency", metadata, autoload=True ) File "/opt/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/schema.py", line 108, in __call__ return type.__call__(self, name, metadata, *args, **kwargs) File "/opt/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/schema.py", line 236, in __init__ _bind_or_error(metadata).reflecttable(self, include_columns=include_columns) File "/opt/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/engine/base.py", line 1265, in reflecttable self.dialect.reflecttable(conn, table, include_columns) File "/opt/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/databases/mysql.py", line 1664, in reflecttable sql = self._show_create_table(connection, table, charset) File "/opt/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/databases/mysql.py", line 1835, in _show_create_table raise exc.NoSuchTableError(full_name) sqlalchemy.exc.NoSuchTableError: tool_dependency Error in sys.excepthook: Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/apport_python_hook.py", line 66, in apport_excepthook from apport.fileutils import likely_packaged, get_recent_crashes ImportError: No module named apport.fileutils Original exception was: Traceback (most recent call last): File "./scripts/manage_db.py", line 63, in <module> main( repository=repo, url=db_url ) File "/opt/galaxy_dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/shell.py", line 150, in main ret = command_func(**kwargs) File "/opt/galaxy_dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/api.py", line 221, in upgrade return _migrate(url, repository, version, upgrade=True, err=err, **opts) File "/opt/galaxy_dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/api.py", line 349, in _migrate schema.runchange(ver, change, changeset.step) File "/opt/galaxy_dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/schema.py", line 184, in runchange change.run(self.engine, step) File "/opt/galaxy_dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/script/py.py", line 101, in run func() File "lib/galaxy/model/migrate/versions/0100_alter_tool_dependency_table_version_column.py", line 30, in upgrade ToolDependency_table = Table( "tool_dependency", metadata, autoload=True ) File "/opt/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/schema.py", line 108, in __call__ return type.__call__(self, name, metadata, *args, **kwargs) File "/opt/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/schema.py", line 236, in __init__ _bind_or_error(metadata).reflecttable(self, include_columns=include_columns) File "/opt/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/engine/base.py", line 1265, in reflecttable self.dialect.reflecttable(conn, table, include_columns) File "/opt/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/databases/mysql.py", line 1664, in reflecttable sql = self._show_create_table(connection, table, charset) File "/opt/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/databases/mysql.py", line 1835, in _show_create_table raise exc.NoSuchTableError(full_name) sqlalchemy.exc.NoSuchTableError: tool_dependency ========================================================

1 1

Small bug with "search tools" in the lastest version of Galaxy
by David Roquis 11 Oct '12

11 Oct '12

Dear Galaxy community, I have noticed that since I have upgraded to the latest galaxy distribution (October 5th, b5bda7a5c345), the "search tools" function in the left frame doesn't seem to work properly. For example, in previous distributions, if I typed "fastx", it would display all the tools under the "FASTX-Toolkit for FASTQ data" section. Now, I don't get anything (but the FASTX-Toolkit for FASTQ data and the tools in it are still available if I look manually in the tool list). Similar problem when I search with the keyword "draw". I get some results, but I would miss some. For example, "Draw quality score boxplot for SOLiD data""Draw quality score boxplot", "Draw nucleotides distribution chart" (and some more) does not appear in the search results. I haven't tested all the possible keywords to search, but my guess is that this problem probably happens with other search terms. The same situation occurs on the penn state instance of galaxy (https://main.g2.bx.psu.edu/) so I guess this problem is not related to my local version. This is not a major bug, but I thought it was worth mentioning it. Sincerely David University of Perpignan

2 1

Running a tool from a tool?
by Ganote, Carrie L 11 Oct '12

11 Oct '12

Hi all, I'm trying to make a tool that takes, as input, a dataset of type="data" and format = "axt,fasta". I want to run the axt_to_fasta tool if the input is axt and the fasta_to_tabular if the input is fasta. Basically, I want to wrap all the type conversions into one tool, since it seems like there is no easy way to allow submenus under the <section> tags. My output from this tool will be the same as the input because I just want to pass it along to another tool; my question is, what should my command be? Should I try to call /lib/galaxy/tools/__init__.py or /lib/galaxy/web/controllers/tool_runner.py? So far I'm trying to learn the architecture of galaxy by delving into the code, looking at all the tool configs and running firebug on firefox to step through the javascript. It makes for slow going though and there's a lot I don't get yet. Thanks for any insight, -Carrie

1 0

Method to copy files....
by Neil.Burdett＠csiro.au 10 Oct '12

10 Oct '12

Hi, Can you tell me the part of the Galaxy code or method responsible for actually copying over the data from one machine to another? i.e. it looks like data is first copied over to the ~/galaxy-dist/database/tmp area first before being moved on, which method copies over the data to ~/galaxy-dist/database/tmp ? Or where is a good place to investigate where this may happen? Thanks Neil

1 0

Re: [galaxy-dev] Adding binary datatype: upload problem
by John Chilton 10 Oct '12

10 Oct '12

Comments inline. On Wed, Oct 10, 2012 at 2:35 PM, Mark Johnson <mjohnson(a)ncbi.nlm.nih.gov> wrote: > > Thanks for the quick turnaround. On the contrary, your examples are very > helpful. I'll try your approach. I haven't upgraded Galaxy in about 6 > months, but was hoping that change had already been made. (I haven't worked > on Galaxy in the interim.) Then do I understand right that the wiki is not > up to date with respect to creating new datatypes? I cannot find anything on the wiki about adding new binary datatypes, does the wiki reference modifying upload.py anywhere? > And does this change mean > that new datatypes are something users will be able to distribute through > the toolshed? Hopefully! I haven't tested distributing datatypes at all via the toolshed let alone binary ones, but my guess is that it will work now. Perhaps Greg has a comment on this issue. It seems like it wouldn't work if you had multiple versions of the same datatype installed, but I don't know how Galaxy handles that even in for normal types. > > I have a bit of confusion (being a novice Galaxian) about how conversion > tools work. The NCBI SRA format has a collection of binary runtime programs > (SRA toolkit) for performing various operations on .csra files. So my new > Galaxy tools that rely on those programs will have somehow to access them. > Is there a formalized way to package Galaxy tools with binaries, so both the > Galaxy XML+Python and the binaries are distributed together? Or do you have > some way of telling the Galaxy admin what needs to be installed, and where, > for Galaxy tools to work? Greg has been hard at work over the last several months implementing the ability to package scripts for installing dependencies along with your tool repository. http://wiki.g2.bx.psu.edu/ToolShedToolFeatures Again, I haven't played with that feature - I'm waiting on the ability to script interactions with the tool shed and inter-repository dependencies. -John > > I tried to follow the link that Nate sent to Pieter on the site I mentioned: > http://bitbucket.org/galaxy/galaxy-central/issue/304/make-binary-datatype-u… > but BitBucket says > > You do not have access to the issues. > > Use the links at the top to get back. > > > > :-( They have disabled the issue tracker, but have promised the data will be transplanted somewhere new at some point. -John > > Is there a way to get access to the issues? > > Thanks 10e+06 > > --Mark Johnson > NCBI > > > On 10/10/12 2:45 PM, John Chilton wrote: > > Hey Mark, > > A few things, instead of printing stuff out you could use the global > variable log in the binary.py file. Maybe something like > log.warn("SNIFFING csra"). I think your logging statements will be > more likely to show up then. > > I don't know why what you are doing won't work (it looks like it > should), but a recent dist update included some changes that I > submitted that allow for the addition of binary datatypes without > needing to modify upload.py which I think is a big improvement. I > would recommend updating galaxy and transitioning to this new > mechanism. > > I have been working with Ira Cooke to gather a lot of generally useful > proteomics datatypes into one place, this includes Pieter Neerincx RAW > file datatype from the original dev list e-mail you linked to so it is > a good example of how this has changed. Here is said file: > > https://bitbucket.org/iracooke/protk-toolshed/src/tip/lib/galaxy/datatypes/… > > Basically, after you define your binary type all you need to do is > register it as a sniffable binary format, in your case this would be > adding this line: > > Binary.register_sniffable_binary_format('crsa', 'crsa', Crsa) > > after the last line of this example: http://pastie.org/5030960. Note > there is no indenting this should be at the top level of the python > code. > > As a side note (since this isn't documented anywhere else), in the > proteomics.py example I went a step further to allow for backward > compatibility with Galaxy versions predating the inclusion of these > binary datatype rewrites by using the following construct: > > if hasattr(Binary, 'register_sniffable_binary_format'): > Binary.register_sniffable_binary_format('RAW', 'RAW', RAW) > > I hope this helps, but I imagine my rambling e-mails only ever serve > to confuse issues. > > -John > > ------------------------------------------------ > John Chilton > Senior Software Developer > University of Minnesota Supercomputing Institute > Office: 612-625-0917 > Cell: 612-226-9223 > Bitbucket: https://bitbucket.org/jmchilton > Github: https://github.com/jmchilton > Web: http://jmchilton.net > > On Wed, Oct 10, 2012 at 1:21 PM, Mark Johnson <mjohnson(a)ncbi.nlm.nih.gov> > wrote: > > I'm trying to add CSRA (NCBI Compressed Sequence Read Archive) as a new > datatype for Galaxy. > > I've followed the instructions on the wiki, and the module seems to load OK. > csra shows up as a datatype in the upload view. > > But the upload fails, and the uploaded file size is always 0. The actual > file I upload is 156k. > > Here are my changes: > > In binary.py: > http://pastie.org/5030960 > > In upload.py (following the example at > http://dev.list.galaxyproject.org/Binary-datatypes-td4135969.html) > http://pastie.org/5030967 > > datatypes_conf.xml: > http://pastie.org/5030970 > > I also have a very hard time debugging Galaxy. Where can I look for an error > stream that explains what it's doing? paster.log only tells me the HTTP > traffic. I need to know where it is failing to know where to look. And my > code needs to be able to, at the very least, print debug messages. How do > people generally do that in Galaxy? > > Thanks > > --Mark Johnson > > > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ > >

2 1

file upload error
by Sukha Malladi 10 Oct '12

10 Oct '12

Hi, I was trying to upload a 43Gb BAM file to Galaxy when I encountered this problem. Please let me know how to fix it. I am new to galaxy and python. I have tried uploading files in other formats and they have worked fine. I have not configured error reporting from the web page. I am using galaxy as a local installation. Thanks GalaxyUser =========== Traceback (most recent call last): File "/Data/galaxy/galaxy-dist/tools/data_source/upload.py", line 384, in <module> __main__() File "/Data/galaxy/galaxy-dist/tools/data_source/upload.py", line 373, in __main__ add_file( dataset, registry, json_file, output_path ) File "/Data/galaxy/galaxy-dist/tools/data_source/upload.py", line 312, in add_file if link_data_only == 'copy_files' and datatype.dataset_content_needs_grooming( output_path ): File "/Data/galaxy/galaxy-dist/lib/galaxy/datatypes/binary.py", line 113, in dataset_content_needs_grooming version = self._get_samtools_version() File "/Data/galaxy/galaxy-dist/lib/galaxy/datatypes/binary.py", line 97, in _get_samtools_version output = subprocess.Popen( [ 'samtools' ], stderr=subprocess.PIPE, stdout=subprocess.PIPE ).communicate()[1] File "/Data/galaxy/Python-2.7.1/Lib/subprocess.py", line 672, in __init__ errread, errwrite) File "/Data/galaxy/Python-2.7.1/Lib/subprocess.py", line 1202, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory ============== Report this error to the Galaxy Team The Galaxy team regularly reviews errors that occur in the application. However, if you would like to provide additional information (such as what you were trying to do when the error occurred) and a contact e-mail address, we will be better able to investigate your problem and get back to you.

1 0

comvert solid output to fastq python sqlite 3 error
by Jacob Israel Cervantes Luevano (LANGEBIO) 10 Oct '12

10 Oct '12

Hi, What's happening ? or what do I need to install ? I review my python sqlite 3 and yes it is installed I configured my own local galaxy instance in own server, it works well, I used proftp, postgresql, and Python 2.7 I uploaded my SOLiD colorspace and Quality data via filezilla to my galaxy instance , everything works fine, but when I try convert to fastq format using following options in Galaxy: AB-SOLiD data Convert SOLiD output to fastq crashed , the error and my galaxy history are: 4: test_S1_F3_QV.qual 3: test_S1_F3.csfasta 18: Convert on data 4 and data 3 0 bytes An error occurred running this job: Traceback (most recent call last): File "/state/partition1/results4/apps/galaxy/hg/galaxy-dist/tools/next_gen_conversion/solid2fastq.py", line 7, in <module> import sqlite3 File "/state/partition1/results4/apps/galaxy/hg/python-installed/lib/pytho Error Detail: ================= Traceback (most recent call last): File "/state/partition1/results4/apps/galaxy/hg/galaxy-dist/tools/next_gen_conversion/solid2fastq.py", line 7, in <module> import sqlite3 File "/state/partition1/results4/apps/galaxy/hg/python-installed/lib/python2.7/sqlite3/__init__.py", line 24, in <module> from dbapi2 import * File "/state/partition1/results4/apps/galaxy/hg/python-installed/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module> from _sqlite3 import * ImportError: No module named _sqlite3 View Detail =============== Tool: Convert Name: Convert on data 4 and data 3 Created: Oct 10, 2012 Filesize: 0 bytes Dbkey: borEut13 Format: fastqcssanger Tool Version: Tool Standard Output: stdout Tool Standard Error: stderr Full Path: /state/partition1/results4/apps/galaxy/hg/galaxy-dist/database/files/000/dataset_31.dat Input Parameter Value Select reads 3: test_S1_F3.csfasta Select qualities 4: test_S1_F3_QV.qual Is this a mate-pair run? no Remove reads containing color qualities below this value 0 Trim trailing "_F3" and "_R3" ? Yes Trim first base? No (bowtie) Double encode? No (bowtie) Inheritance Chain Convert on data 4 and data 3 I reviewed python 2.7 sqlite 3 directories .. [jacob@xanini gnuplot-4.6.1]$ cd /state/partition1/results4/apps/galaxy/hg/python-installed/lib/python2.7/sqlite3 [jacob@xanini sqlite3]$ ls dbapi2.py dbapi2.pyc dbapi2.pyo dump.py dump.pyc dump.pyo __init__.py __init__.pyc __init__.pyo test dbapi2.py source code : #-*- coding: ISO-8859-1 -*- # pysqlite2/dbapi2.py: the DB-API 2.0 interface # # Copyright (C) 2004-2005 Gerhard H�ring <gh(a)ghaering.de> # # This file is part of pysqlite. # # This software is provided 'as-is', without any express or implied # warranty. In no event will the authors be held liable for any damages # arising from the use of this software. # # Permission is granted to anyone to use this software for any purpose, # including commercial applications, and to alter it and redistribute it # freely, subject to the following restrictions: # # 1. The origin of this software must not be misrepresented; you must not # claim that you wrote the original software. If you use this software # in a product, an acknowledgment in the product documentation would be # appreciated but is not required. # 2. Altered source versions must be plainly marked as such, and must not be # misrepresented as being the original software. # 3. This notice may not be removed or altered from any source distribution. import datetime import time from _sqlite3 import * What's the problem ? Thanks in advance -- Jacob Cinvvestav Langebio Web site: http://www.langebio.cinvestav.mx/bioinformatica/jacob/ -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.

1 0

Cannot install ncbi_blast_plus or other tools from Tool Shed: HTTP 500 error
by Joel Rosenberg 10 Oct '12

10 Oct '12

(Apologies if this is in duplicate, Peter Cock may have forwarded it on after I posted it to the user list) So I'm trying to install the recently migrated ncbi_blast_plus tool from the Tool Shed to my local galaxy instance running in EC2 and am getting errors at the installation step. After clicking "Install to local Galaxy" I immediately get: "Server Error An error occurred. See the error logs for more information. (Turn debug on to display exception reports here)" and an HTTP 500 error code is returned. Chrome tells me the URL it's attempting to request is: http://toolshed.g2.bx.psu.edu/repository/install_repositories_by_revision?c… I tried the same process for other non-blast related tools that appeared to be simple (no 3rd party binary dependencies) and got the same server error. I restarted galaxy with its log level set to DEBUG and wasn't illuminated further, but that doesn't surprise me since my browser's requesting data from the public toolshed, not my server. I just performed a mercurial update in my cluster, too. Any ideas?

3 2

Velveth wrapper not working with real user jobs
by Oleksandr Moskalenko 10 Oct '12

10 Oct '12

Velveth wrapper passes the location of the database/files/xxx/dataset_yyy_files to the velveth as an argument. Same paths are put into the output html file. This method can't work anymore when jobs are submitted under real user names as a real user doesn't have write permissions outside of the exernal_chown_script, so the velveth wrapper is broken for all installations using real user id jobs at this time. Regards, Alex

2 1

The migrated BWA installed but missing dependencies
by Derrick LIN 10 Oct '12

10 Oct '12

Hi guys, I have updated the galaxy to Oct 5 dist, and BWA has been reinstalled via the migration script. However, I couldn't find any entry has been added in migrated_tools_conf.xml (thus the BWA tools are not display in the tool panel). I check in the admin "Installed tool shed repositories" section and BWA repos status is "Installed, missing dependencies", is that why it didn't get added into the migrated_tools_conf.xml? I have had BWA 0.5.9-r16 in path, and been used by the old BWA wrapper without an issue before. How can I fix this dependencies issue for a Shed Tool? Should I add something like: <tool_dependency_dir>/bowtie/0.12.7/env.sh thanks, Derrick

3 8