[galaxy-dev] again on multiple server instances

21 May 2010

      Hi all, after successful configuration of multiple server instances (web and runner) I've placed everything behind apache proxy (actually I'm trying to).
I can see the welcome page and navigate through my history (still I have to delegate the authentication to apache), nevertheless I'm not able to perform any job...
getting data from UCSC is waiting for run after a couple of hours, uploading files is stuck...
Hence I've tried to disable the proxy thing and still have the same issues. I've then checked at the log files for the five runners and two of them report this:

[runner0]
serving on http://127.0.0.1:8100
galaxy.jobs.schedulingpolicy.roundrobin DEBUG 2010-05-21 13:49:09,032 RoundRobin queue: user/session did not exist, created new jobqueue for session = 6galaxy.jobs DEBUG 2010-05-21 13:49:09,032 job 13 put in policy queue
galaxy.jobs.schedulingpolicy.roundrobin DEBUG 2010-05-21 13:49:09,032 RoundRobin queue: retrieving job from job queue for session = 6
galaxy.jobs DEBUG 2010-05-21 13:49:09,032 dispatching job 13 to local runner
galaxy.jobs DEBUG 2010-05-21 13:49:09,135 job 13 dispatched
galaxy.jobs.runners.local DEBUG 2010-05-21 13:49:09,538 executing: python /data/galaxy_dist/tools/data_source/data_source.py /data/galaxy_dist/database/files
/000/dataset_60.dat 0
galaxy.jobs.runners.local ERROR 2010-05-21 13:49:09,761 Uncaught exception running job
Traceback (most recent call last):
  File "/data/galaxy_dist/lib/galaxy/jobs/runners/local.py", line 46, in run_next
    self.run_job( job_wrapper )
  File "/data/galaxy_dist/lib/galaxy/jobs/runners/local.py", line 101, in run_job
    job_wrapper.fail( "failure running job", exception=True )
  File "/data/galaxy_dist/lib/galaxy/jobs/__init__.py", line 413, in fail
    self.sa_session.refresh( job )
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/orm/scoping.py", line 127, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/orm/session.py", line 929, in refresh
    only_load_props=attribute_names) is None:
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/orm/query.py", line 1500, in _get
    return q.all()[0]
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/orm/query.py", line 1267, in all
    return list(self)
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/orm/query.py", line 1361, in __iter__
    return self._execute_and_instances(context)
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/orm/query.py", line 1364, in _execute_and_instances
    result = self.session.execute(querycontext.statement, params=self._params, mapper=self._mapper_zero_or_none())
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/orm/session.py", line 755, in execute
    clause, params or {})
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py", line 824, in execute
    return Connection.executors[c](self, object, multiparams, params)
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py", line 874, in _execute_clauseelement
    return self.__execute_context(context)
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py", line 901, in __execute_context
    return context.get_result_proxy()
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/databases/postgres.py", line 346, in get_result_proxy
    return base.BufferedRowResultProxy(self)
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py", line 1415, in __init__
    self._init_metadata()
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py", line 1704, in _init_metadata
    self.__buffer_rows()
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py", line 1721, in __buffer_rows
    self.__rowbuffer = self.cursor.fetchmany(size)
  File "/usr/lib/python2.5/decimal.py", line 541, in __new__
    m = _parser(value)
OperationalError: could not receive data from server: Error 0

[runner1]
serving on http://127.0.0.1:8101
galaxy.jobs.schedulingpolicy.roundrobin DEBUG 2010-05-21 13:49:09,198 RoundRobin queue: user/session did not exist, created new jobqueue for session = 6
galaxy.jobs DEBUG 2010-05-21 13:49:09,199 job 13 put in policy queue
galaxy.jobs.schedulingpolicy.roundrobin DEBUG 2010-05-21 13:49:09,199 RoundRobin queue: retrieving job from job queue for session = 6
galaxy.jobs DEBUG 2010-05-21 13:49:09,199 dispatching job 13 to local runner
galaxy.jobs DEBUG 2010-05-21 13:49:09,315 job 13 dispatched
galaxy.jobs.runners.local ERROR 2010-05-21 13:49:09,741 Uncaught exception running job
Traceback (most recent call last):
  File "/data/galaxy_dist/lib/galaxy/jobs/runners/local.py", line 46, in run_next
    self.run_job( job_wrapper )
  File "/data/galaxy_dist/lib/galaxy/jobs/runners/local.py", line 58, in run_job
    job_wrapper.fail( "failure preparing job", exception=True )
  File "/data/galaxy_dist/lib/galaxy/jobs/__init__.py", line 413, in fail
    self.sa_session.refresh( job )
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/orm/scoping.py", line 127, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/orm/session.py", line 929, in refresh
    only_load_props=attribute_names) is None:
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/orm/query.py", line 1500, in _get
    return q.all()[0]
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/orm/query.py", line 1267, in all
    return list(self)
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/orm/query.py", line 1361, in __iter__
    return self._execute_and_instances(context)
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/orm/query.py", line 1364, in _execute_and_instances
    result = self.session.execute(querycontext.statement, params=self._params, mapper=self._mapper_zero_or_none())
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/orm/session.py", line 755, in execute
    clause, params or {})
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py", line 824, in execute
    return Connection.executors[c](self, object, multiparams, params)
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py", line 874, in _execute_clauseelement
    return self.__execute_context(context)
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py", line 901, in __execute_context
    return context.get_result_proxy()
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/databases/postgres.py", line 346, in get_result_proxy
    return base.BufferedRowResultProxy(self)
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py", line 1415, in __init__
    self._init_metadata()
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py", line 1704, in _init_metadata
    self.__buffer_rows()
  File "/data/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py", line 1721, in __buffer_rows
    self.__rowbuffer = self.cursor.fetchmany(size)
  File "/usr/lib/python2.5/decimal.py", line 541, in __new__
    m = _parser(value)
OperationalError: could not receive data from server: Error 0

BTW, this is the proxy conf:

<IfModule mod_proxy.c>

        ProxyRequests Off

        ProxyVia Off
        <Proxy balancer://galaxy>
                BalancerMember http://localhost:8080
                BalancerMember http://localhost:8081
        </Proxy>
</IfModule>

and the rewrite rules:

RewriteEngine on
RewriteRule ^/galaxy$ /galaxy/ [R]
RewriteRule ^/galaxy/static/style/(.*) /data/galaxy_dist/static/june_2007_style/blue/$1 [L]
RewriteRule ^/galaxy/static/(.*) /data/galaxy_dist/static/$1 [L]
RewriteRule ^/galaxy/images/(.*) /data/galaxy_dist/static/images/$1 [L]
RewriteRule ^/galaxy/favicon.ico /data/galaxy_dist/static/favicon.ico [L]
RewriteRule ^/galaxy/robots.txt /data/galaxy_dist/static/robots.txt [L]
RewriteRule ^/galaxy(.*) http://127.0.0.1:8080$1 [P]

Any hint is appreciated...

Thanks

d

/*
Davide Cittaro

Cogentech - Consortium for Genomic Technologies
via adamello, 16
20139 Milano
Italy

tel.: +39(02)574303007
e-mail: davide.cittaro@ifom-ieo-campus.it
*/