I've migrated my staging instance to use postgres and ran the functional tests.  I see lots failing.  I know this is known and perhaps expected, but it still bothers me.  I would expect to see all the tests pass on a clean instance.  If the tests aren't passing, to me, that indicates a problem somewhere.  What's the point in having the tests if they always fail?



On Tue, Jun 16, 2015 at 1:12 PM, Ryan G <ngsbioinformatics@gmail.com> wrote:
I am using postgres in production but want to validate my staging instance before pushing it into production.  I do this with the functional tests.  I guess I'll have to set up a postgres database for the staging instance as well.


On Mon, Jun 15, 2015 at 10:32 AM, John Chilton <jmchilton@gmail.com> wrote:
Not exactly - I feel like sqlite should be able to do the kinds of
things Galaxy does without this occurring - it should just be
inefficient. We are probably doing something across multiple threads
that we should not be - presumably related to job handling. After a
recent sqlalchemy upgrade - there is now a new error message in the
logs that should help track down the problem:

sqlalchemy.pool.NullPool ERROR 2015-06-15 09:53:47,547 Exception
during reset or similar
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/sqlalchemy/pool.py", line 571, in
_finalize_fairy
    fairy._reset(pool)
  File "build/bdist.linux-x86_64/egg/sqlalchemy/pool.py", line 705, in _reset
    pool._dialect.do_rollback(self)
  File "build/bdist.linux-x86_64/egg/sqlalchemy/engine/default.py",
line 412, in do_rollback
    dbapi_connection.rollback()
ProgrammingError: SQLite objects created in a thread can only be used
in that same thread.The object was created in thread id
140442561648384 and this is thread id 140442989479680

At any rate - we recommend using postgres in production for a reason -
these sqlite errors are pretty common.

-John


On Fri, Jun 12, 2015 at 11:15 AM, Ryan G <ngsbioinformatics@gmail.com> wrote:
> Is there a known cause for this?
>
> On Fri, Jun 12, 2015 at 10:52 AM, Nicola Soranzo <nsoranzo@tiscali.it>
> wrote:
>>
>> Hi Ryan,
>> there is no know fix for this, apart from moving to PostgreSQL.
>>
>> Cheers,
>> Nicola
>>
>> Il 12.06.2015 16:43 Ryan G ha scritto:
>>
>> Hi all - I'm running functional tests on a set of tools using
>> 'run_tests.sh -sid ...'.  A few of the tests fail.  When I go back and run
>> the tool tests individually, they succeed.  So I looked into why they fail
>> when running them in batch, and it turns out to be a database lock.  I'm
>> using the default sqllite database that run_tests.sh creates.  Does this
>> need to be tuned somehow?
>>
>>
>> galaxy.jobs.handler INFO 2015-06-12 10:28:35,854 (14) Job dispatched
>> Error - : (OperationalError) database is locked u'SELECT api_keys.id AS
>> api_keys_id, api_keys.create_time AS api_keys_create_time, api_keys.user_id
>> AS api_keys_user_id, api_keys."key" AS api_keys_key \nFROM api_keys \nWHERE
>> api_keys."key" = ?' (u'e4fe9bf787edb9ef746714beb6212bc8',)
>> URL:
>> http://localhost:8363/api/histories/cb227cec8ca83994?key=e4fe9bf787edb9ef746714beb6212bc8
>> File
>> '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/middleware/error.py',
>> line 149 in __call__
>>   app_iter = self.application(environ, sr_checker)
>> File
>> '/apps/sys/galaxy/galaxy-v15.05-staging/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py',
>> line 84 in __call__
>>   return self.application(environ, start_response)
>> File
>> '/apps/sys/galaxy/galaxy-v15.05-staging/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py',
>> line 633 in __call__
>>   return self.application(environ, start_response)
>> File
>> '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/base.py',
>> line 132 in __call__
>>   return self.handle_request( environ, start_response )
>> File
>> '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/base.py',
>> line 159 in handle_request
>>   trans = self.transaction_factory( environ )
>> File
>> '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
>> line 71 in
>>   self.set_transaction_factory( lambda e: self.transaction_chooser( e,
>> galaxy_app, session_cookie ) )
>> File
>> '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
>> line 102 in transaction_chooser
>>   return GalaxyWebTransaction( environ, galaxy_app, self, session_cookie )
>> File
>> '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
>> line 194 in __init__
>>   self.error_message = self._authenticate_api( session_cookie )
>> File
>> '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
>> line 283 in _authenticate_api
>>   provided_key = self.sa_session.query( self.app.model.APIKeys ).filter(
>> self.app.model.APIKeys.table.c.key == api_key ).one()
>> File 'build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py', line 2184 in
>> one
>> File 'build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py', line 2227 in
>> __iter__
>> File 'build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py', line 2242 in
>> _execute_and_instances
>> File 'build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py', line 1449
>> in execute
>> File 'build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py', line 1584
>> in _execute_clauseelement
>> File 'build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py', line 1698
>> in _execute_context
>> File 'build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py', line 1691
>> in _execute_context
>> File 'build/bdist.linux-x86_64/egg/sqlalchemy/engine/default.py', line 331
>> in do_execute
>> OperationalError: (OperationalError) database is locked u'SELECT
>> api_keys.id AS api_keys_id, api_keys.create_time AS api_keys_create_time,
>> api_keys.user_id AS api_keys_user_id, api_keys."key" AS api_keys_key \nFROM
>> api_keys \nWHERE api_keys."key" = ?' (u'e4fe9bf787edb9ef746714beb6212bc8',)
>>
>>
>>
>>
>>
>> Connetti gratis il mondo con la nuova indoona: hai la chat, le chiamate,
>> le video chiamate e persino le chiamate di gruppo.
>> E chiami gratis anche i numeri fissi e mobili nel mondo!
>> Scarica subito l’app Vai su https://www.indoona.com/
>>
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/