December 2011 - galaxy-dev - lists.galaxyproject.org

Galaxy Server on Bio-Linux
by Bicak, Mesude 05 Jan '12

05 Jan '12

Dear Galaxy Developers, We work in Professor Dawn Field's group (Molecular Evolution and Bioinformatics Research Group) at the NERC Environmental Bioinformatics Centre (NEBC) of the Centre for Ecology and Hydrology (CEH) Research Institute based in Oxford. We develop and distribute Bio-Linux (http://nebc.nerc.ac.uk/tools/bio-linux) which is a customised Ubuntu distribution that comes with 500+ bioinformatics packages. Within our research group we also provide bioinformatics analysis for NERC-funded researchers, and recently started looking into Galaxy as well. It didn't take us long to discover its power and we would like to enable Bio-Linux users to install, run and maintain the Galaxy server with minimal effort, also with the aim to spread the word on Galaxy in Europe! Recently we took on a project with Dr. Casey Bergman from University of Manchester as the Principal Investigator, to package all the necessary Galaxy dependencies for Ubuntu/Bio-Linux. As many pre-requisities are already included with Bio-Linux, we are already some way down this path. New packages that we create will appear in a Launchpad PPA (https://launchpad.net/~nebc/+archive/galaxy) We will be happy to hear any comments regarding these efforts and we hope that this will be a useful resource for all Galaxy users. Once the initial packaging is done, we hope to collaborate with Galaxy team in maintaining and improving this resource. Best wishes, Tim, Soon and Mesude -- Dr. Mesude Bicak <mbicak(a)ceh.ac.uk> Bioinformatician & Bio-Linux Developer NERC Biomolecular Analysis Facility (NBAF) <http://nbaf.nerc.ac.uk> NERC Environmental Bioinformatics Centre (NEBC) <http://nebc.nerc.ac.uk> Molecular Evolution and Bioinformatics Group Centre for Ecology and Hydrology <http://ceh.ac.uk> Maclean Building, Benson Lane Crowmarsh Gifford Wallingford, Oxfordshire OX10 8BB Office Tel: +44 1491 69 2705-- This message (and any attachments) is for the recipient only. NERC is subject to the Freedom of Information Act 2000 and the contents of this email and any reply you make may be disclosed by NERC unless it is exempt from release under the Act. Any material supplied to NERC may be stored in an electronic records management system.

4 4

Best practices with data on clusters
by Cittaro Davide 04 Jan '12

04 Jan '12

Hi developers, I have a question that may be an OT, but since galaxy can work in a clustered environment withh queueing system, I'll try to ask here. Is there anibody here who copies data in a local temporary directory before performing any analysis step and copy it back into the "final results"? Thanks d Sent from my iPad

2 3

Problem with uploading SMALL file to Galaxy on EC2 and get "client intended to send too large body"
by Thon deBoer 04 Jan '12

04 Jan '12

I am trying to setup a instance of Galaxy on the EC2 cloud and everything seems to be going OK. Everything installs correctly and I can start Galaxy with no problems. But when I try to upload a SMALL BAM file (~5 MB), it just hangs there and never completes... When I look at the error.log I see the following error message: 2011/12/09 19:27:32 [error] 929#0: *1204 client intended to send too large body: 4719398 bytes, client: 173.195.189.92, server: localhost, request: "POST /tool_runner/index HTTP/1.1", host: "ec2-50-17-113-5.compute-1.amazonaws.com", referrer: "http://ec2-50-17-113-5.compute-1.amazonaws.com/tool_runner?tool_id=upload1" It seems to have a problem with the size of the body... Anyone have an idea how to fix this? I am using ami-da58aab3 Regards, Thon de Boer, Ph.D. Bioinformatics Guru +1-650-799-6839 thondeboer(a)me.com LinkedIn Profile

2 1

Torque / scrambling pbs_python
by Ryan Golhar 03 Jan '12

03 Jan '12

Hi - I've just finished installing a ROCKS 5.4.3 cluster and am now adding Galaxy to it. We are using torque instead of SGE. I see the Wiki docs at http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Cluster state I need to scramble pbs_python by doing: LIBTORQUE_DIR=/path/to/libtorque python scripts/scramble.py -e pbs_python I have libtorque installed in /usr/lib64: [galaxy@bic x86_64]$ ll /usr/lib64/libtorque.* -rw-r--r-- 1 root root 536862 Dec 28 12:32 /usr/lib64/libtorque.a -rwxr-xr-x 1 root root 810 Dec 28 12:32 /usr/lib64/libtorque.la lrwxrwxrwx 1 root root 18 Dec 28 13:50 /usr/lib64/libtorque.so -> libtorque.so.2.0.0 lrwxrwxrwx 1 root root 14 Dec 28 13:50 /usr/lib64/libtorque.so.0 -> libtorque.so.2 lrwxrwxrwx 1 root root 18 Dec 28 13:50 /usr/lib64/libtorque.so.2 -> libtorque.so.2.0.0 -rwxr-xr-x 1 root root 766431 Dec 28 12:32 /usr/lib64/libtorque.so.2.0.0 So I then run (and get the error shown at the bottom): [galaxy@bic x86_64]$ cd [galaxy@bic ~]$ cd galaxy-dist/ [galaxy@bic galaxy-dist]$ LIBTORQUE_DIR=/usr/lib64/libtorque.so.2.0.0 python scripts/scramble.py -e pbs_python fetch_one(): Trying to fetch: http://eggs.g2.bx.psu.edu/pbs_python/pbs_python-4.1.0.tar.gz fetch_one(): Fetched to: /home/galaxy/galaxy-dist/scripts/scramble/archives/pbs_python-4.1.0.tar.gz unpack_source(): Unpacked to: /home/galaxy/galaxy-dist/scripts/scramble/build/py2.6-linux-x86_64-ucs2/pbs_python copy_build_script(): Using build script /home/galaxy/galaxy-dist/scripts/scramble/scripts/pbs_python.py run_scramble_script(): Beginning build run_scramble_script(): Executing in /home/galaxy/galaxy-dist/scripts/scramble/build/py2.6-linux-x86_64-ucs2/pbs_python: /share/apps/Python-2.6.7/bin/python scramble.py --------------------------------------------------------------------------- This script requires setuptools version 0.6c11 to run (even to display help). I will attempt to download it for you (from http://pypi.python.org/packages/2.6/s/setuptools/) but you may need to enable firewall access for this script first. I will start the download in 8 seconds. (Note: if this machine does not have network access, please obtain the file http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg and place it in this directory before rerunning this script.) --------------------------------------------------------------------------- Downloading http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg checking for pbs-config... /usr/bin/pbs-config Found torque version: 2.5.9 checking for python... /share/apps/Python-2.6.7/bin/python checking for python version... 2.6 checking for python platform... linux2 checking for python script directory... ${prefix}/lib/python2.6/site-packages checking for python extension module directory... ${exec_prefix}/lib/python2.6/site-packages configure: creating ./config.status config.status: creating Makefile config.status: creating setup.py scramble(): Patching setup.py running egg_info creating src/pbs_python.egg-info writing src/pbs_python.egg-info/PKG-INFO writing top-level names to src/pbs_python.egg-info/top_level.txt writing dependency_links to src/pbs_python.egg-info/dependency_links.txt writing manifest file 'src/pbs_python.egg-info/SOURCES.txt' reading manifest file 'src/pbs_python.egg-info/SOURCES.txt' writing manifest file 'src/pbs_python.egg-info/SOURCES.txt' running bdist_egg installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build creating build/lib.linux-x86_64-2.6 copying src/pbs.py -> build/lib.linux-x86_64-2.6 copying src/PBSQuery.py -> build/lib.linux-x86_64-2.6 running build_ext building '_pbs' extension creating build/temp.linux-x86_64-2.6 creating build/temp.linux-x86_64-2.6/src gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DTORQUE_2_4 -I/share/apps/Python-2.6.7/include/python2.6 -c src/pbs_wrap.c -o build/temp.linux-x86_64-2.6/src/pbs_wrap.o src/pbs_wrap.c:2622: warning: function declaration isn't a prototype src/pbs_wrap.c:2630: warning: function declaration isn't a prototype gcc -pthread -shared build/temp.linux-x86_64-2.6/src/pbs_wrap.o -L/usr/lib64 -ltorque -o build/lib.linux-x86_64-2.6/_pbs.so creating build/bdist.linux-x86_64 creating build/bdist.linux-x86_64/egg copying build/lib.linux-x86_64-2.6/pbs.py -> build/bdist.linux-x86_64/egg copying build/lib.linux-x86_64-2.6/PBSQuery.py -> build/bdist.linux-x86_64/egg copying build/lib.linux-x86_64-2.6/_pbs.so -> build/bdist.linux-x86_64/egg byte-compiling build/bdist.linux-x86_64/egg/pbs.py to pbs.pyc byte-compiling build/bdist.linux-x86_64/egg/PBSQuery.py to PBSQuery.pyc creating stub loader for _pbs.so byte-compiling build/bdist.linux-x86_64/egg/_pbs.py to _pbs.pyc creating build/bdist.linux-x86_64/egg/EGG-INFO copying src/pbs_python.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying src/pbs_python.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying src/pbs_python.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying src/pbs_python.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt zip_safe flag not set; analyzing archive contents... creating dist creating 'dist/pbs_python-4.1.0-py2.6-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Traceback (most recent call last): File "scripts/scramble.py", line 50, in <module> egg.scramble() File "/home/galaxy/galaxy-dist/lib/galaxy/eggs/scramble.py", line 60, in scramble raise ScrambleFailure( self, "%s(): Egg build for %s did not appear to fail, but no egg found to copy from expected path:\n %s" % ( sys._getframe().f_code.co_name, self.name, new_egg ) ) galaxy.eggs.scramble.ScrambleFailure: scramble(): Egg build for pbs_python did not appear to fail, but no egg found to copy from expected path: /home/galaxy/galaxy-dist/scripts/scramble/build/py2.6-linux-x86_64-ucs2/pbs_python/dist/pbs_python-4.1.0-py2.6-linux-x86_64-ucs2.egg [galaxy@bic galaxy-dist]$ cd scripts/scramble [galaxy@bic scramble]$ ll total 20 drwxrwxr-x 2 galaxy galaxy 4096 Dec 29 16:21 archives drwxrwxr-x 3 galaxy galaxy 4096 Dec 29 16:21 build drwxrwxr-x 2 galaxy galaxy 4096 Dec 29 16:22 lib drwxrwxr-x 7 galaxy galaxy 4096 Dec 29 13:25 patches drwxrwxr-x 2 galaxy galaxy 4096 Dec 29 13:25 scripts [galaxy@bic scramble]$ cd build/ [galaxy@bic build]$ ll total 4 drwxrwxr-x 3 galaxy galaxy 4096 Dec 29 16:21 py2.6-linux-x86_64-ucs2 [galaxy@bic build]$ cd py2.6-linux-x86_64-ucs2/ [galaxy@bic py2.6-linux-x86_64-ucs2]$ ll total 4 drwx------ 7 galaxy galaxy 4096 Dec 29 16:22 pbs_python [galaxy@bic py2.6-linux-x86_64-ucs2]$ cd pbs_python/dist/ [galaxy@bic dist]$ ll total 180 -rw-rw-r-- 1 galaxy galaxy 178869 Dec 29 16:22 pbs_python-4.1.0-py2.6-linux-x86_64.egg I see the egg is created, but not with the name expected by galaxy. How do I fix this?

3 3

unpacking zip files during upload
by Langhorst, Brad 03 Jan '12

03 Jan '12

I'm working on unpacking a zip file into multiple datasets. I think this is the code path Upload.py UploadToolAction upload_common.py: get_uploaded_datesets new_upload new_history_upload or new_library_upload Then a job gets spooled Which calles add_file in data_source/upload.py And does the expansion of the zip I can unpack the zip and create files in the dataset's path there. But I don't know how to create more dataset associations, and I'm not sure that it makes sense to create datasets on the fly in data_source/upload.py . Should I pass some information along with data_source/upload.py about how to create dataset object and associate them with library/history associations? Or maybe I can pass in some kind of a callback that can handle the dataset expansion? (I'm pretty new to python, but it seems similar to ruby) I thought about a composite dataset, but that seems like overloading that concept. Really the files I'm thinking about uplaoding are 8 independent BAMs or fastqs or whatever – not a set of files that are related to each other. Any suggestions? Brad -- Brad Langhorst New England Biolabs langhorst(a)neb.com

2 1

Problem uploading file by an URL
by Marc Bras 03 Jan '12

03 Jan '12

Hi, We have a Galaxy server in production and I have a problem when I want to download a huge file with HTTP URL. Galaxy starts well to download my file, but Galaxy writes this file in "/ tmp" directory. Is it possible to change this directory ? Thank you, Regards, Marc Bras -- Marc Bras ------------------------------------------ Marc.Bras(a)versailles.inra.fr INRA-URGI: Unité de Recherche Génomique Info Centre de Recherche de Versailles-Grignon Route de Saint Cyr 78026 Versailles - FRANCE Tel: +33 1 30 83 34 70 ------------------------------------------

2 1

file upload issue
by weihong yan 03 Jan '12

03 Jan '12

I recently installed galaxy program on our Linux server and it is going to be used for analyzing high-throuput sequencing data. I tried to upload a dataset (a bed format file) from local computer to galaxy server, but no success. The message of "Dataset is uploading" stays forever. The uploaded dataset was shown on the /galaxy-dist/database/tmp directory, but it didn't get transferred to the job-working directory or files directory before it was deleted by the galaxy. The output of run_functional_tests shows that uploading file went successfully. The paster.log file doesn't show error message about the uploading. Any configuration did I miss? Your feedback is highly appreciated. Thank you! Weihong

3 4

how to disable error detection
by Xuebing Wu 03 Jan '12

03 Jan '12

Hi, I'm wondering if there is a way to disable the error detection (output showing red). I was adding some tools to a local Galaxy server. Some of the tool's stdout were directed to a file. The tool ran without an error, but Galaxy treated this standard output as error. best, Xuebing

2 1

manage_db.sh upgrade fails after fresh install of Galaxy
by Iry Witham 03 Jan '12

03 Jan '12

I have installed a new Galaxy server utilizing the November 18,2011 distribution on a SLES 11 VM and cannot get it to launch cleanly. I am running postgreSQL v. 8.3.9 to create my database. Once the database is created and running I launch my Galaxy server and within moments all of the PID fail except for the reports_webapp.pid. When I tail the logs I find the following: tail reports_webapp.log - galaxy.webapps.reports.buildapp DEBUG 2011-12-21 14:20:54,179 Enabling 'httpexceptions' middleware galaxy.webapps.reports.buildapp DEBUG 2011-12-21 14:20:54,194 Enabling 'recursive' middleware /hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/Paste-1.6-py2.6.egg/paste/exceptions/serial_number_generator.py:11: DeprecationWarning: the md5 module is deprecated; use hashlib instead import md5 galaxy.webapps.reports.buildapp DEBUG 2011-12-21 14:20:54,272 Enabling 'error' middleware galaxy.webapps.reports.buildapp DEBUG 2011-12-21 14:20:54,277 Enabling 'trans logger' middleware galaxy.webapps.reports.buildapp DEBUG 2011-12-21 14:20:54,277 Enabling 'config' middleware galaxy.webapps.reports.buildapp DEBUG 2011-12-21 14:20:54,294 Enabling 'x-forwarded-host' middleware Starting server in PID 20411. serving on 0.0.0.0:9001 view at http://127.0.0.1:9001 tail web0.log - galaxy.model.migrate.check DEBUG 2011-12-21 14:20:54,793 psycopg2 egg successfully loaded for postgres dialect Traceback (most recent call last): File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/lib/galaxy/web/buildapp.py", line 82, in app_factory app = UniverseApplication( global_conf = global_conf, **kwargs ) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/lib/galaxy/app.py", line 39, in __init__ create_or_verify_database( db_url, kwargs.get( 'global_conf', {} ).get( '__file__', None ), self.config.database_engine_options ) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/lib/galaxy/model/migrate/check.py", line 105, in create_or_verify_database % ( db_schema.version, migrate_repository.versions.latest, config_arg ) ) Exception: Your database has version '38' but this code expects version '85'. Please backup your database and then migrate the schema by running 'sh manage_db.sh -c ./universe_wsgi.webapp.ini upgrade'. Removing PID file web0.pid tail runner0.log - galaxy.model.migrate.check DEBUG 2011-12-21 14:20:54,789 psycopg2 egg successfully loaded for postgres dialect Traceback (most recent call last): File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/lib/galaxy/web/buildapp.py", line 82, in app_factory app = UniverseApplication( global_conf = global_conf, **kwargs ) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/lib/galaxy/app.py", line 39, in __init__ create_or_verify_database( db_url, kwargs.get( 'global_conf', {} ).get( '__file__', None ), self.config.database_engine_options ) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/lib/galaxy/model/migrate/check.py", line 105, in create_or_verify_database % ( db_schema.version, migrate_repository.versions.latest, config_arg ) ) Exception: Your database has version '38' but this code expects version '85'. Please backup your database and then migrate the schema by running 'sh manage_db.sh -c ./universe_wsgi.runner.ini upgrade'. Removing PID file runner0.pid sh manage_db.sh upgrade - 38 -> 39... Migration script to add a synopsis column to the library table. Traceback (most recent call last): File "./scripts/manage_db.py", line 63, in <module> main( repository=repo, url=db_url ) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/shell.py", line 150, in main ret = command_func(**kwargs) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py", line 221, in upgrade return _migrate(url, repository, version, upgrade=True, err=err, **opts) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py", line 349, in _migrate schema.runchange(ver, change, changeset.step) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/schema.py", line 184, in runchange change.run(self.engine, step) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/py.py", line 101, in run func() File "lib/galaxy/model/migrate/versions/0039_add_synopsis_column_to_library_table.py", line 20, in upgrade c.create( Library_table ) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/changeset/schema.py", line 365, in create engine._run_visitor(visitorcallable, self, *args, **kwargs) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 1158, in _run_visitor visitorcallable(self.dialect, conn, **kwargs).traverse(element) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/sql/visitors.py", line 89, in traverse return traverse(obj, self.__traverse_options__, self._visitor_dict) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/sql/visitors.py", line 200, in traverse return traverse_using(iterate(obj, opts), obj, visitors) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/sql/visitors.py", line 194, in traverse_using meth(target) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/changeset/ansisql.py", line 97, in visit_column self.execute() File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 1812, in execute return self.connection.execute(self.buffer.getvalue()) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 824, in execute return Connection.executors[c](self, object, multiparams, params) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 888, in _execute_text return self.__execute_context(context) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 896, in __execute_context self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 950, in _cursor_execute self._handle_dbapi_exception(e, statement, parameters, cursor, context) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 931, in _handle_dbapi_exception raise exc.DBAPIError.instance(statement, parameters, e, connection_invalidated=is_disconnect) sqlalchemy.exc.ProgrammingError: (ProgrammingError) column "synopsis" of relation "library" already exists '\nALTER TABLE "library" ADD synopsis TEXT' {} I have dropped the database multiple times and recreated it and intentionally left out steps when creating DB users and roles and the manage_db.sh fails at different spots. However, I cannot get past 38 -> 39. Any suggestions? Thanks, IRy

4 5

Stuck at "Job is waiting to run" when using "Get data"
by Liang Zhang 03 Jan '12

03 Jan '12

Dear Galaxy team: I was trying to use the Get Data - Upload File tool yesterday. I uploaded all files through FTP. Then selected them in "Files uploaded via FTP", selected "auto-detect" + "Mouse July 2007 (NCBI37/mm9) (mm9)" and clicked execute. My jobs are registered, but they all stuck at "Job is waiting to run". I waited for over 16 hours and they are still showing "Job is waiting to run". I tried to click "Run the job again" and it says "Could not find the job for this dataset". I tried to delete all the jobs, log out and log in. Then add jobs again, but still no luck, they all still stuck at "Job is waiting to run". My data files are 8 x 3Gb sized fastq files. Please help. Liangz

2 1