January 2012 - galaxy-dev - lists.galaxyproject.org

file upload issue
by weihong yan 03 Jan '12

03 Jan '12

I recently installed galaxy program on our Linux server and it is going to be used for analyzing high-throuput sequencing data. I tried to upload a dataset (a bed format file) from local computer to galaxy server, but no success. The message of "Dataset is uploading" stays forever. The uploaded dataset was shown on the /galaxy-dist/database/tmp directory, but it didn't get transferred to the job-working directory or files directory before it was deleted by the galaxy. The output of run_functional_tests shows that uploading file went successfully. The paster.log file doesn't show error message about the uploading. Any configuration did I miss? Your feedback is highly appreciated. Thank you! Weihong

3 4

how to disable error detection
by Xuebing Wu 03 Jan '12

03 Jan '12

Hi, I'm wondering if there is a way to disable the error detection (output showing red). I was adding some tools to a local Galaxy server. Some of the tool's stdout were directed to a file. The tool ran without an error, but Galaxy treated this standard output as error. best, Xuebing

2 1

Re: [galaxy-dev] Quota will not decrease with permanent delete
by Nate Coraor 03 Jan '12

03 Jan '12

On Dec 6, 2011, at 2:16 PM, Mary Anne Alliegro wrote: > No, it is still reads over 120GB yet the usage is only 26% Hi Mary Anne, Sorry I didn't get back to you sooner. Your usage is currently ~75GB. The difference before may have been due to some other data in your account which was not permanently deleted. If you still find that your disk usage isn't reflecting your actual history sizes, please have a look at this page for details on how to view deleted data: http://wiki.g2.bx.psu.edu/Learn/Managing%20Datasets And let us know if this doesn't answer any remaining questions. Thanks for using Galaxy, --nate > > Mary Anne Alliegro > Marine Biological Laboratory > JOSEPHINE BAY PAUL CENTER > FOR COMPARATIVE MOLECULAR BIOLOGY AND EVOLUTION > 7 MBL Street > Woods Hole, MA 02543 > maalliegro(a)mbl.edu > 508-289-7419 > > From: "Nate Coraor" <nate(a)bx.psu.edu> > To: "Mary Anne Alliegro" <maalliegro(a)mbl.edu> > Cc: galaxy-dev(a)bx.psu.edu > Sent: Tuesday, December 6, 2011 1:14:32 PM > Subject: Re: [galaxy-dev] Quota will not decrease with permanent delete > > On Nov 30, 2011, at 2:24 PM, Mary Anne Alliegro wrote: > > > Hi Galaxy Users, > > I have permanently deleted numerous files. > > My usage % has decreased, but this is NOT reflected in my Gb report (upper right)- it remains the same. > > Am I missing some phantom trash bin? > > If not, will Galaxy recalculate my Gb usage so that I may proceed with my project? > > Thank you, > > Mary Anne > > Hi Mary Anne, > > I believe your usage total is now correct, but if this is not the case, please let us know. > > Thanks, > --nate > > > > > > > <pastedGraphic.pdf> > > > > ___________________________________________________________ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this > > and other Galaxy lists, please use the interface at: > > > > http://lists.bx.psu.edu/ >

1 0

proxy server settings (or documentation amendment)
by Ryan 03 Jan '12

03 Jan '12

I set up my Apache server to act as a proxy server based on the docs at http://wiki.g2.bx.psu.edu/Admin/Config/Apache%20Proxy My pages aren't loading correctly and I discovered that since I have galaxy installed as its own user, the apache user account does not have access to the galaxy files. I need to change the permissions of the galaxy files to make them available to the apache user. Someone may want to update the docs to reflect that this change needs to be made...

2 2

load balancing with Apache
by Ryan 03 Jan '12

03 Jan '12

Hi all - I'm setting up Galaxy under Apache and following the guide for a production environment. I've got the proxy server running properly and am now setting up load balancing. I currently start galaxy (for testing) using 'sh run.sh --reload' so I can stop galaxy using Ctrl-C. I'm following the instructions on http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Web%20Application%20Scal… There is a mistake in the run.sh script provided for Starting and Stopping the web app and runner services. The script right now shows: cd `dirname $0` for i in {0..1}; do python ./scripts/paster.py serve universe_wsgi.webapp.ini --server-name=web$i --pid-file=web$i.pid --log-file=web$i.log $@ done python ./scripts/paster.py serve universe_wsgi.runner.ini --server-name=runner0 --pid-file=runner0.pid --log-file=runner0.log $@ The problem with this is the python processes don't get backgrounded. So when this is run, the first web0 python process is invoked, but that's it. The rest doesn't get run. You need to stop the process with Ctrl-C, but then only the next web1 process gets started. Ctrl-C then stops web1, then runner0 gets started. I changed the script to be: cd `dirname $0` for i in {0..1}; do python ./scripts/paster.py serve universe_wsgi.webapp.ini --server-name=web$i --pid-file=web$i.pid --log-file=web$i.log --daemon $@ done python ./scripts/paster.py serve universe_wsgi.runner.ini --server-name=runner0 --pid-file=runner0.pid --log-file=runner0.log --daemon $@ I added '--daemon'. The only hiccup to this is after doing this, Ctrl-C is no longer an option to stop galaxy. So I discovered using 'sh run.sh --reload' is no longer correct. Instead 'sh run.sh --daemon' and 'sh run.sh --stop-daemon' should be used. This may want to be included in the wiki docs.

2 1

manage_db.sh upgrade fails after fresh install of Galaxy
by Iry Witham 03 Jan '12

03 Jan '12

I have installed a new Galaxy server utilizing the November 18,2011 distribution on a SLES 11 VM and cannot get it to launch cleanly. I am running postgreSQL v. 8.3.9 to create my database. Once the database is created and running I launch my Galaxy server and within moments all of the PID fail except for the reports_webapp.pid. When I tail the logs I find the following: tail reports_webapp.log - galaxy.webapps.reports.buildapp DEBUG 2011-12-21 14:20:54,179 Enabling 'httpexceptions' middleware galaxy.webapps.reports.buildapp DEBUG 2011-12-21 14:20:54,194 Enabling 'recursive' middleware /hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/Paste-1.6-py2.6.egg/paste/exceptions/serial_number_generator.py:11: DeprecationWarning: the md5 module is deprecated; use hashlib instead import md5 galaxy.webapps.reports.buildapp DEBUG 2011-12-21 14:20:54,272 Enabling 'error' middleware galaxy.webapps.reports.buildapp DEBUG 2011-12-21 14:20:54,277 Enabling 'trans logger' middleware galaxy.webapps.reports.buildapp DEBUG 2011-12-21 14:20:54,277 Enabling 'config' middleware galaxy.webapps.reports.buildapp DEBUG 2011-12-21 14:20:54,294 Enabling 'x-forwarded-host' middleware Starting server in PID 20411. serving on 0.0.0.0:9001 view at http://127.0.0.1:9001 tail web0.log - galaxy.model.migrate.check DEBUG 2011-12-21 14:20:54,793 psycopg2 egg successfully loaded for postgres dialect Traceback (most recent call last): File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/lib/galaxy/web/buildapp.py", line 82, in app_factory app = UniverseApplication( global_conf = global_conf, **kwargs ) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/lib/galaxy/app.py", line 39, in __init__ create_or_verify_database( db_url, kwargs.get( 'global_conf', {} ).get( '__file__', None ), self.config.database_engine_options ) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/lib/galaxy/model/migrate/check.py", line 105, in create_or_verify_database % ( db_schema.version, migrate_repository.versions.latest, config_arg ) ) Exception: Your database has version '38' but this code expects version '85'. Please backup your database and then migrate the schema by running 'sh manage_db.sh -c ./universe_wsgi.webapp.ini upgrade'. Removing PID file web0.pid tail runner0.log - galaxy.model.migrate.check DEBUG 2011-12-21 14:20:54,789 psycopg2 egg successfully loaded for postgres dialect Traceback (most recent call last): File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/lib/galaxy/web/buildapp.py", line 82, in app_factory app = UniverseApplication( global_conf = global_conf, **kwargs ) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/lib/galaxy/app.py", line 39, in __init__ create_or_verify_database( db_url, kwargs.get( 'global_conf', {} ).get( '__file__', None ), self.config.database_engine_options ) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/lib/galaxy/model/migrate/check.py", line 105, in create_or_verify_database % ( db_schema.version, migrate_repository.versions.latest, config_arg ) ) Exception: Your database has version '38' but this code expects version '85'. Please backup your database and then migrate the schema by running 'sh manage_db.sh -c ./universe_wsgi.runner.ini upgrade'. Removing PID file runner0.pid sh manage_db.sh upgrade - 38 -> 39... Migration script to add a synopsis column to the library table. Traceback (most recent call last): File "./scripts/manage_db.py", line 63, in <module> main( repository=repo, url=db_url ) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/shell.py", line 150, in main ret = command_func(**kwargs) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py", line 221, in upgrade return _migrate(url, repository, version, upgrade=True, err=err, **opts) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py", line 349, in _migrate schema.runchange(ver, change, changeset.step) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/schema.py", line 184, in runchange change.run(self.engine, step) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/py.py", line 101, in run func() File "lib/galaxy/model/migrate/versions/0039_add_synopsis_column_to_library_table.py", line 20, in upgrade c.create( Library_table ) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/changeset/schema.py", line 365, in create engine._run_visitor(visitorcallable, self, *args, **kwargs) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 1158, in _run_visitor visitorcallable(self.dialect, conn, **kwargs).traverse(element) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/sql/visitors.py", line 89, in traverse return traverse(obj, self.__traverse_options__, self._visitor_dict) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/sql/visitors.py", line 200, in traverse return traverse_using(iterate(obj, opts), obj, visitors) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/sql/visitors.py", line 194, in traverse_using meth(target) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/changeset/ansisql.py", line 97, in visit_column self.execute() File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 1812, in execute return self.connection.execute(self.buffer.getvalue()) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 824, in execute return Connection.executors[c](self, object, multiparams, params) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 888, in _execute_text return self.__execute_context(context) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 896, in __execute_context self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 950, in _cursor_execute self._handle_dbapi_exception(e, statement, parameters, cursor, context) File "/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 931, in _handle_dbapi_exception raise exc.DBAPIError.instance(statement, parameters, e, connection_invalidated=is_disconnect) sqlalchemy.exc.ProgrammingError: (ProgrammingError) column "synopsis" of relation "library" already exists '\nALTER TABLE "library" ADD synopsis TEXT' {} I have dropped the database multiple times and recreated it and intentionally left out steps when creating DB users and roles and the manage_db.sh fails at different spots. However, I cannot get past 38 -> 39. Any suggestions? Thanks, IRy

4 5

Stuck at "Job is waiting to run" when using "Get data"
by Liang Zhang 03 Jan '12

03 Jan '12

Dear Galaxy team: I was trying to use the Get Data - Upload File tool yesterday. I uploaded all files through FTP. Then selected them in "Files uploaded via FTP", selected "auto-detect" + "Mouse July 2007 (NCBI37/mm9) (mm9)" and clicked execute. My jobs are registered, but they all stuck at "Job is waiting to run". I waited for over 16 hours and they are still showing "Job is waiting to run". I tried to click "Run the job again" and it says "Could not find the job for this dataset". I tried to delete all the jobs, log out and log in. Then add jobs again, but still no luck, they all still stuck at "Job is waiting to run". My data files are 8 x 3Gb sized fastq files. Please help. Liangz

2 1

Problem uploading files from filesystem paths
by Liisa Koski 03 Jan '12

03 Jan '12

Hi, I'm trying to upload data to Data Libraries from filesystems paths as Admin. I get the following error. Any ideas? Thanks in advance, Liisa Traceback (most recent call last): File "/data/Galaxy/galaxy-dist/tools/data_source/upload.py", line 394, in __main__() File "/data/Galaxy/galaxy-dist/tools/data_source/upload.py", line 386, in __main__ add_file( dataset, registry, js Job Standard Error Traceback (most recent call last): File "/data/Galaxy/galaxy-dist/tools/data_source/upload.py", line 394, in __main__() File "/data/Galaxy/galaxy-dist/tools/data_source/upload.py", line 386, in __main__ add_file( dataset, registry, json_file, output_path ) File "/data/Galaxy/galaxy-dist/tools/data_source/upload.py", line 300, in add_file if datatype.dataset_content_needs_grooming( dataset.path ): File "/data/Galaxy/galaxy-dist/lib/galaxy/datatypes/binary.py", line 79, in dataset_content_needs_grooming version = self._get_samtools_version() File "/data/Galaxy/galaxy-dist/lib/galaxy/datatypes/binary.py", line 63, in _get_samtools_version output = subprocess.Popen( [ 'samtools' ], stderr=subprocess.PIPE, stdout=subprocess.PIPE ).communicate()[1] File "/usr/lib64/python2.6/subprocess.py", line 633, in __init__ errread, errwrite) File "/usr/lib64/python2.6/subprocess.py", line 1139, in _execute_child raise child_exception OSError: [Errno 13] Permission denied

6 9

links to datafile don't work in Postgres database?
by Thon Deboer 03 Jan '12

03 Jan '12

Hi, I had no problems with providing my galaxy installation with datasets that were links to files in my filesystem, when I was using the SQLite database. But when I switched to a Postgres database, the links result in empty files when I choose links in the dataset upload. When I use actual "Load into galaxy" there is no problem, so it seems I just can't pass links to the Postgres database. Am I missing something? Database runs on the same machine as galaxy... Thanks Regards, Thon de Boer, Ph.D. Bioinformatics Guru LinkedIn Profile

1 0

Is there a way to pass key =value pairs to the output files in a programmatic way? (Tags? Metadata?)
by Thon deBoer 02 Jan '12

02 Jan '12

Hi, I am trying to pass some key-value pairs to the result of a tool (such as the original sample name) but I cannot find a way to do this in the galaxy XML tool definitions. I am simply trying to retain in all the data files I produce, the name of the original sample which the results are related to. Now I have to follow the history of a result just to see which original data file sample my results came from (and when using hidden files, this is almost impossible, since all results refer to some arbitrary number my data set had ("Tophat on data 30" is not going to help me, if I have 140 samples with all the same name)... Is there some way to pass some key-value pairs to a tools output? Can tags be assigned in the Tool definition? Can I set metadata tags programmatically? Any help would be appreciated... Regards, Thon de Boer, Ph.D. Bioinformatics Guru LinkedIn Profile

4 5