Hi all, I've seen several posts for jobs which stay queued and never start with an installation of galaxy on a cluster. But our installation is just a local instance of galaxy in 1 PC and I have more than 100 jobs which stay queued and never start. How can I debug this problem (my version of galaxy is the latest 2june2014). And is it possible to run those jobs without restart galaxy because it would be very long to rerun each job manually after reloading galaxy. Thanks. Julie
On 23/07/2014 11:14, julie dubois wrote:
Hi all, I've seen several posts for jobs which stay queued and never start with an installation of galaxy on a cluster. But our installation is just a local instance of galaxy in 1 PC and I have more than 100 jobs which stay queued and never start.
How can I debug this problem (my version of galaxy is the latest 2june2014).
And is it possible to run those jobs without restart galaxy because it would be very long to rerun each job manually after reloading galaxy.
Thanks.
Julie ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi, What king of cluster are you using ? You should first check that your "galaxy" user is allowed to send jobs on your scheduler. Regards, Thomas -- Thomas Bellembois, Network and System Administrator ENS-Lyon/IGFL - FRANCE +33 4 26 73 13 67 .The IGFL is a member of the "portail-labos" project (https://portail-labos.ens-lyon.fr/site/). .IGFL IT documentation: http://itdoc.igfl.ens-lyon.fr/itdoc
Hi Julie Are there any jobs running at all? What do you get when you click on "Manage jobs" in the admin menu? Maybe you have some old jobs 'running' which are now blocking your queue? Hans-Rudolf On 07/23/2014 11:14 AM, julie dubois wrote:
Hi all, I've seen several posts for jobs which stay queued and never start with an installation of galaxy on a cluster. But our installation is just a local instance of galaxy in 1 PC and I have more than 100 jobs which stay queued and never start.
How can I debug this problem (my version of galaxy is the latest 2june2014).
And is it possible to run those jobs without restart galaxy because it would be very long to rerun each job manually after reloading galaxy.
Thanks.
Julie ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Hans-Rudolf, Thanks for your answer. I've launched lot of jobs since monday. And some jobs were finished and some jobs never started. In "Manage jobs" I can see jobs queued because they need inputs which are queued, so I think it's normal. But other jobs are described like this : 9554 user_login 0 hours ago samtools_flagstat new 23199 ok, 23243 ok None None None The two necessary inputs for this job (23199 and 2343) are ok and the job doesn't run. Another mysterious line is : 9561 user_login 0 hours ago fastqc new Unable to determine inputs None None None After check this job in my history, I'm sure that the input of this job is green and really available. Why can galaxy not determine inputs ? Finally, in "Manage Jobs" there are only jobs queued. No jobs running. Any idea ? Thanks Julie 2014-07-23 11:35 GMT+02:00 Hans-Rudolf Hotz <hrh@fmi.ch>:
Hi Julie
Are there any jobs running at all? What do you get when you click on "Manage jobs" in the admin menu? Maybe you have some old jobs 'running' which are now blocking your queue?
Hans-Rudolf
On 07/23/2014 11:14 AM, julie dubois wrote:
Hi all, I've seen several posts for jobs which stay queued and never start with an installation of galaxy on a cluster. But our installation is just a local instance of galaxy in 1 PC and I have more than 100 jobs which stay queued and never start.
How can I debug this problem (my version of galaxy is the latest 2june2014).
And is it possible to run those jobs without restart galaxy because it would be very long to rerun each job manually after reloading galaxy.
Thanks.
Julie ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
On 07/23/2014 12:05 PM, julie dubois wrote:
Hi Hans-Rudolf, Thanks for your answer. I've launched lot of jobs since monday. And some jobs were finished and some jobs never started. In "Manage jobs" I can see jobs queued because they need inputs which are queued, so I think it's normal.
But other jobs are described like this :
9554 user_login 0 hours ago samtools_flagstat new 23199 ok, 23243 ok None None None The two necessary inputs for this job (23199 and 2343) are ok and the job doesn't run.
Another mysterious line is :
9561 user_login 0 hours ago fastqc new Unable to determine inputs None None None
After check this job in my history, I'm sure that the input of this job is green and really available. Why can galaxy not determine inputs ? Finally, in "Manage Jobs" there are only jobs queued. No jobs running.
Any idea ?
I have one suspicion: which database are you using? If you are just using the built in SQlite you might have run in a database-lock? Because you have submitted too many jobs at once? In this case, I guess, the only way to recover is to re-start... Hans-Rudolf
Thanks Julie
2014-07-23 11:35 GMT+02:00 Hans-Rudolf Hotz <hrh@fmi.ch>:
Hi Julie
Are there any jobs running at all? What do you get when you click on "Manage jobs" in the admin menu? Maybe you have some old jobs 'running' which are now blocking your queue?
Hans-Rudolf
On 07/23/2014 11:14 AM, julie dubois wrote:
Hi all, I've seen several posts for jobs which stay queued and never start with an installation of galaxy on a cluster. But our installation is just a local instance of galaxy in 1 PC and I have more than 100 jobs which stay queued and never start.
How can I debug this problem (my version of galaxy is the latest 2june2014).
And is it possible to run those jobs without restart galaxy because it would be very long to rerun each job manually after reloading galaxy.
Thanks.
Julie ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
I'm using a MySQL database !!! But may be such database-lock can appear because of my universe.ini configuration file. Here it's the lines concerning the database : database_engine_option_pool_size = 10 database_engine_option_max_overflow = 20 database_engine_option_pool_recycle = -1 database_engine_option_strategy = threadlocal Can one of these parameters be at the origin of my problem ? If yes, when I'll restart Galaxy, the queued jobs will be in a staying state with the capacity to become in a running state ? Or they will be in error state and I should restart the process as new jobs ? Thanks. Julie 2014-07-23 12:55 GMT+02:00 Hans-Rudolf Hotz <hrh@fmi.ch>:
On 07/23/2014 12:05 PM, julie dubois wrote:
Hi Hans-Rudolf, Thanks for your answer. I've launched lot of jobs since monday. And some jobs were finished and some jobs never started. In "Manage jobs" I can see jobs queued because they need inputs which are queued, so I think it's normal.
But other jobs are described like this :
9554 user_login 0 hours ago samtools_flagstat new 23199 ok, 23243 ok None None None The two necessary inputs for this job (23199 and 2343) are ok and the job doesn't run.
Another mysterious line is :
9561 user_login 0 hours ago fastqc new Unable to determine inputs None None None
After check this job in my history, I'm sure that the input of this job is green and really available. Why can galaxy not determine inputs ? Finally, in "Manage Jobs" there are only jobs queued. No jobs running.
Any idea ?
I have one suspicion:
which database are you using? If you are just using the built in SQlite you might have run in a database-lock? Because you have submitted too many jobs at once?
In this case, I guess, the only way to recover is to re-start...
Hans-Rudolf
Thanks Julie
2014-07-23 11:35 GMT+02:00 Hans-Rudolf Hotz <hrh@fmi.ch>:
Hi Julie
Are there any jobs running at all? What do you get when you click on "Manage jobs" in the admin menu? Maybe you have some old jobs 'running' which are now blocking your queue?
Hans-Rudolf
On 07/23/2014 11:14 AM, julie dubois wrote:
Hi all, I've seen several posts for jobs which stay queued and never start with an installation of galaxy on a cluster. But our installation is just a local instance of galaxy in 1 PC and I have more than 100 jobs which stay queued and never start.
How can I debug this problem (my version of galaxy is the latest 2june2014).
And is it possible to run those jobs without restart galaxy because it would be very long to rerun each job manually after reloading galaxy.
Thanks.
Julie ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
On 07/23/2014 02:13 PM, julie dubois wrote:
I'm using a MySQL database !!! But may be such database-lock can appear because of my universe.ini configuration file. Here it's the lines concerning the database :
database_engine_option_pool_size = 10 database_engine_option_max_overflow = 20 database_engine_option_pool_recycle = -1 database_engine_option_strategy = threadlocal
Can one of these parameters be at the origin of my problem ?
we are also using MySQL, but we use the suggested setting: database_engine_option_pool_recycle = 7200 though, i don't think this is related to your problem. Or did you get the error: "MySQL server has gone away" in the galaxy log file?
If yes, when I'll restart Galaxy, the queued jobs will be in a staying state with the capacity to become in a running state ? Or they will be in error state and I should restart the process as new jobs ?
I don't know, I don't have enough experience with queued workflow jobs and what is happening with them during a re-start. I am sorry, I can't be of much further help. Maybe someone else on the list? Hans-Rudolf
Thanks.
Julie
2014-07-23 12:55 GMT+02:00 Hans-Rudolf Hotz <hrh@fmi.ch>:
On 07/23/2014 12:05 PM, julie dubois wrote:
Hi Hans-Rudolf, Thanks for your answer. I've launched lot of jobs since monday. And some jobs were finished and some jobs never started. In "Manage jobs" I can see jobs queued because they need inputs which are queued, so I think it's normal.
But other jobs are described like this :
9554 user_login 0 hours ago samtools_flagstat new 23199 ok, 23243 ok None None None The two necessary inputs for this job (23199 and 2343) are ok and the job doesn't run.
Another mysterious line is :
9561 user_login 0 hours ago fastqc new Unable to determine inputs None None None
After check this job in my history, I'm sure that the input of this job is green and really available. Why can galaxy not determine inputs ? Finally, in "Manage Jobs" there are only jobs queued. No jobs running.
Any idea ?
I have one suspicion:
which database are you using? If you are just using the built in SQlite you might have run in a database-lock? Because you have submitted too many jobs at once?
In this case, I guess, the only way to recover is to re-start...
Hans-Rudolf
Thanks Julie
2014-07-23 11:35 GMT+02:00 Hans-Rudolf Hotz <hrh@fmi.ch>:
Hi Julie
Are there any jobs running at all? What do you get when you click on "Manage jobs" in the admin menu? Maybe you have some old jobs 'running' which are now blocking your queue?
Hans-Rudolf
On 07/23/2014 11:14 AM, julie dubois wrote:
Hi all, I've seen several posts for jobs which stay queued and never start with an installation of galaxy on a cluster. But our installation is just a local instance of galaxy in 1 PC and I have more than 100 jobs which stay queued and never start.
How can I debug this problem (my version of galaxy is the latest 2june2014).
And is it possible to run those jobs without restart galaxy because it would be very long to rerun each job manually after reloading galaxy.
Thanks.
Julie ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Hans-Rudolf. Yes, I have this error in my log file :"MySQL server has gone away"!!! So I changed the option database_engine_option_pool and it seems to work fine! Thanks a lot for your help. And for the information, when a job is queued on a local instance not in a cluster, when we restart the server, the job is crashed to the error state. Thanks again. julie 2014-07-23 14:25 GMT+02:00 Hans-Rudolf Hotz <hrh@fmi.ch>:
On 07/23/2014 02:13 PM, julie dubois wrote:
I'm using a MySQL database !!! But may be such database-lock can appear because of my universe.ini configuration file. Here it's the lines concerning the database :
database_engine_option_pool_size = 10 database_engine_option_max_overflow = 20 database_engine_option_pool_recycle = -1 database_engine_option_strategy = threadlocal
Can one of these parameters be at the origin of my problem ?
we are also using MySQL, but we use the suggested setting:
database_engine_option_pool_recycle = 7200
though, i don't think this is related to your problem. Or did you get the error: "MySQL server has gone away" in the galaxy log file?
If yes, when I'll restart Galaxy, the queued jobs will be in a staying state with the capacity to become in a running state ? Or they will be in error state and I should restart the process as new jobs ?
I don't know, I don't have enough experience with queued workflow jobs and what is happening with them during a re-start. I am sorry, I can't be of much further help. Maybe someone else on the list?
Hans-Rudolf
Thanks.
Julie
2014-07-23 12:55 GMT+02:00 Hans-Rudolf Hotz <hrh@fmi.ch>:
On 07/23/2014 12:05 PM, julie dubois wrote:
Hi Hans-Rudolf, Thanks for your answer. I've launched lot of jobs since monday. And some jobs were finished and some jobs never started. In "Manage jobs" I can see jobs queued because they need inputs which are queued, so I think it's normal.
But other jobs are described like this :
9554 user_login 0 hours ago samtools_flagstat new 23199 ok, 23243 ok None None None The two necessary inputs for this job (23199 and 2343) are ok and the job doesn't run.
Another mysterious line is :
9561 user_login 0 hours ago fastqc new Unable to determine inputs None None None
After check this job in my history, I'm sure that the input of this job is green and really available. Why can galaxy not determine inputs ? Finally, in "Manage Jobs" there are only jobs queued. No jobs running.
Any idea ?
I have one suspicion:
which database are you using? If you are just using the built in SQlite you might have run in a database-lock? Because you have submitted too many jobs at once?
In this case, I guess, the only way to recover is to re-start...
Hans-Rudolf
Thanks Julie
2014-07-23 11:35 GMT+02:00 Hans-Rudolf Hotz <hrh@fmi.ch>:
Hi Julie
Are there any jobs running at all? What do you get when you click on "Manage jobs" in the admin menu? Maybe you have some old jobs 'running' which are now blocking your queue?
Hans-Rudolf
On 07/23/2014 11:14 AM, julie dubois wrote:
Hi all, I've seen several posts for jobs which stay queued and never start with an installation of galaxy on a cluster. But our installation is just a local instance of galaxy in 1 PC and I have more than 100 jobs which stay queued and never start.
How can I debug this problem (my version of galaxy is the latest 2june2014).
And is it possible to run those jobs without restart galaxy because it would be very long to rerun each job manually after reloading galaxy.
Thanks.
Julie ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
participants (3)
-
Hans-Rudolf Hotz -
julie dubois -
Thomas Bellembois