Job splitters + SQLIte = OperationalError?
Hi all, We are using PostreSQL for our production Galaxy server, but for development I am using SQLite. Typically it is just me is using the development server, so database contention has not generally been a problem. However, while testing job splitting for breaking up tasks for the cluster, I have noticed today multiple problems with the database when queuing or running the sub-jobs: OperationalError: (OperationalError) database is locked My hunch is that all the child-jobs are created and added to the queue and some of this is happening in parallel leading to contention over the SQLite database. Does this sound likely? Is this something that could be more robust in Galaxy, or should I simply switch to a real database (SQLite or PostreSQL) for these tests? Thanks, Peter
On Jun 14, 2012, at 11:37 AM, Peter Cock wrote:
My hunch is that all the child-jobs are created and added to the queue and some of this is happening in parallel leading to contention over the SQLite database. Does this sound likely?
Yep, this is exactly what's happening. It isn't just the job splitting code either, this happens in other circumstances as well.
Is this something that could be more robust in Galaxy, or should I simply switch to a real database (SQLite or PostreSQL) for these tests?
I've never taken a serious look at what it'd take to prevent this happening, but I use (and recommend) Postgres for any of hte parallelism work. -Dannon
participants (2)
-
Dannon Baker -
Peter Cock