Dear list,

Yesterday, I noticed that the user quota setting on our Galaxy had a problem - the list of user email address is missing from the manage quota page. So I thought to give a Galaxy codebase update a try after backing up the Galaxy dir and the database.

Running hg head gives me:
==
postgres@ubuntu:/mnt/galaxyTools/galaxy-2013-05-13$ hg head
changeset:   10003:b4a373d86c51
tag:         tip
parent:      10001:471484ff8be6
user:        greg
date:        Wed Jun 12 11:48:09 2013 -0400
summary:     Add targets to Repository Actions menu items.
==

I updated the DB schema to support the code update. But the restart of Galaxy failed with an error in paster.log:
==
Traceback (most recent call last):
  File "/mnt/galaxyTools/galaxy-2013-05-13/lib/galaxy/webapps/galaxy/buildapp.py", line 35, in app_factory
    app = UniverseApplication( global_conf = global_conf, **kwargs )
  File "/mnt/galaxyTools/galaxy-2013-05-13/lib/galaxy/app.py", line 164, in __init__
    self.job_manager = manager.JobManager( self )
  File "/mnt/galaxyTools/galaxy-2013-05-13/lib/galaxy/jobs/manager.py", line 36, in __init__
    self.job_handler.start()
  File "/mnt/galaxyTools/galaxy-2013-05-13/lib/galaxy/jobs/handler.py", line 34, in start
    self.job_queue.start()
  File "/mnt/galaxyTools/galaxy-2013-05-13/lib/galaxy/jobs/handler.py", line 77, in start
    self.__check_jobs_at_startup()
  File "/mnt/galaxyTools/galaxy-2013-05-13/lib/galaxy/jobs/handler.py", line 125, in __check_jobs_at_startup
    self.dispatcher.recover( job, job_wrapper )
  File "/mnt/galaxyTools/galaxy-2013-05-13/lib/galaxy/jobs/handler.py", line 620, in recover
    self.job_runners[runner_name].recover( job, job_wrapper )
  File "/mnt/galaxyTools/galaxy-2013-05-13/lib/galaxy/jobs/runners/local.py", line 128, in recover
    job_wrapper.change_state( model.Job.states.ERROR, info = "This job was killed when Galaxy was restarted.  Please retry the job." )
  File "/mnt/galaxyTools/galaxy-2013-05-13/lib/galaxy/jobs/__init__.py", line 824, in change_state
    dataset.state = state
  File "/mnt/galaxyTools/galaxy-2013-05-13/lib/galaxy/model/__init__.py", line 1163, in set_dataset_state
    object_session( self ).flush() #flush here, because hda.flush() won't flush the Dataset object
  File "/mnt/galaxyTools/galaxy-2013-05-13/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 1718, in flush
    self._flush(objects)
  File "/mnt/galaxyTools/galaxy-2013-05-13/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 1789, in _flush
    flush_context.execute()
  File "/mnt/galaxyTools/galaxy-2013-05-13/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/unitofwork.py", line 331, in execute
    rec.execute(self)
  File "/mnt/galaxyTools/galaxy-2013-05-13/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/unitofwork.py", line 475, in execute
    uow
  File "/mnt/galaxyTools/galaxy-2013-05-13/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/persistence.py", line 59, in save_obj
    mapper, table, update)
  File "/mnt/galaxyTools/galaxy-2013-05-13/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/persistence.py", line 485, in _emit_update_statements
    execute(statement, params)
  File "/mnt/galaxyTools/galaxy-2013-05-13/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1449, in execute
    params)
  File "/mnt/galaxyTools/galaxy-2013-05-13/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/mnt/galaxyTools/galaxy-2013-05-13/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1698, in _execute_context
    context)
  File "/mnt/galaxyTools/galaxy-2013-05-13/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1691, in _execute_context
    context)
  File "/mnt/galaxyTools/galaxy-2013-05-13/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/default.py", line 331, in do_execute
    cursor.execute(statement, parameters)
InternalError: (InternalError) could not read block 326 in file "base/257536/259291": read only 0 of 8192 bytes
 'UPDATE dataset SET update_time=%(update_time)s, state=%(state)s WHERE dataset.id = %(dataset_id)s' {'update_time': datetime.datetime(2013, 6, 17, 12, 42, 38, 145908), 'state': 'error', 'dataset_id': 68792}
==

I went on to check the Galaxy database (in postgres) which confirmed the error in the paster.log.
==
galaxy=# select * from dataset where id = 68792;                                                                                                                                                               id   |        create_time         |        update_time         | state | deleted | purged | purgable | external_filename | _extra_files_path | file_size | total_size | object_store_id | uuid
-------+----------------------------+----------------------------+-------+---------+--------+----------+-------------------+-------------------+-----------+------------+-----------------+------
 68792 | 2013-06-12 06:55:17.608725 | 2013-06-12 06:55:19.396557 | error | f       | f      | t        |                   |                   |           |            |                 |
(1 row)

galaxy=# UPDATE dataset SET update_time='2013-06-17 12:42:38.145908', state='error' WHERE dataset.id = 68792;
ERROR:  could not read block 326 in file "base/257536/259291": read only 0 of 8192 bytes
galaxy=# UPDATE dataset SET state='error' WHERE dataset.id = 68792;
UPDATE 1
==

When I tried to restore the database from the dumped copy I made yesterday, I got the following error:
==
ERROR:  insert or update on table "galaxy_session_to_history" violates foreign key constraint "galaxy_session_to_history_
session_id_fkey"
DETAIL:  Key (session_id)=(956338) is not present in table "galaxy_session".
ERROR:  insert or update on table "job" violates foreign key constraint "job_session_id_fkey"
DETAIL:  Key (session_id)=(852060) is not present in table "galaxy_session".
ERROR:  insert or update on table "workflow_step_connection" violates foreign key constraint "workflow_step_connection_input_step_id_fkey"
DETAIL:  Key (input_step_id)=(18422) is not present in table "workflow_step".
ERROR:  insert or update on table "workflow_step_connection" violates foreign key constraint "workflow_step_connection_output_step_id_fkey"
DETAIL:  Key (output_step_id)=(18422) is not present in table "workflow_step".
==

It seems our postgres DB got corrupted? Any suggestion on a fix?

Thanks a lot!

Leon

--
Hailiang (Leon) Mei
Netherlands Bioinformatics Center  
BioAssist NGS Taskforce
 - http://ngs.nbic.nl
Skype: leon_mei    Mobile: +31 6 41709231