Jobs are slow to start on my galaxy instance. Used to be much faster
Hi, Jobs that I start on my in-house Galaxy instance now take up to 3-4 min to go from queued to Running, even though there is nothing much going on on the galaxy server... I have been running this instance since June and use a relatively new version of Galaxy-central (Last update, 22-Aug changeset: 7535:bf6517b2b336) I have noticed that my jobs table in the galaxy Postgres database contains about 60,000 jobs... Could that be the culprit? Does it slow to the complete database to see if there are any jobs that need to run? Could I purge the jobs and related tables to speed up the submissions? Thanks, Thon
On Sep 18, 2012, at 6:31 PM, Anthonius deBoer wrote:
Hi,
Jobs that I start on my in-house Galaxy instance now take up to 3-4 min to go from queued to Running, even though there is nothing much going on on the galaxy server...
I have been running this instance since June and use a relatively new version of Galaxy-central (Last update, 22-Aug changeset: 7535:bf6517b2b336)
I have noticed that my jobs table in the galaxy Postgres database contains about 60,000 jobs... Could that be the culprit? Does it slow to the complete database to see if there are any jobs that need to run?
Could I purge the jobs and related tables to speed up the submissions?
Hi Thon, It's not really possible to remove things from the database since there are a lot of interdependencies. The first thing to check would be whether set_metadata_externally = True in your Galaxy config. --nate
Thanks,
Thon
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Hi Thon Sorry for re-activating an old e-mail thread. I am just curious, whether Nate's suggestion solved your problem? After our upgrade (from the "March 12th changeset 6799:40f1816d6857" to the "October 5th changeset 7824:b5bda7a5c345") we see long delays in the jobs going from queued to running. For our server, switching from "set_metadata_externally = False" to "set_metadata_externally = True" did not speed up anything. Regards, Hans-Rudolf On 09/19/2012 04:27 PM, Nate Coraor wrote:
On Sep 18, 2012, at 6:31 PM, Anthonius deBoer wrote:
Hi,
Jobs that I start on my in-house Galaxy instance now take up to 3-4 min to go from queued to Running, even though there is nothing much going on on the galaxy server...
I have been running this instance since June and use a relatively new version of Galaxy-central (Last update, 22-Aug changeset: 7535:bf6517b2b336)
I have noticed that my jobs table in the galaxy Postgres database contains about 60,000 jobs... Could that be the culprit? Does it slow to the complete database to see if there are any jobs that need to run?
Could I purge the jobs and related tables to speed up the submissions?
Hi Thon,
It's not really possible to remove things from the database since there are a lot of interdependencies.
The first thing to check would be whether set_metadata_externally = True in your Galaxy config.
--nate
Thanks,
Thon
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
I created the index but that did not change anything... I had sent James Taylor the schema as he requested, but I never heard anything so I guess he must not have found anything weird... I am waiting to move my Galaxy server off to a three server configuration, so I am hoping the problem will not return, although your issue does not bode well for this issue... Regards, Thon de Boer, Ph.D. Bioinformatics Guru +1-650-799-6839 thondeboer@me.com LinkedIn Profile On Oct 23, 2012, at 4:14 AM, Hans-Rudolf Hotz <hrh@fmi.ch> wrote:
Hi Thon
Sorry for re-activating an old e-mail thread. I am just curious, whether Nate's suggestion solved your problem?
After our upgrade (from the "March 12th changeset 6799:40f1816d6857" to the "October 5th changeset 7824:b5bda7a5c345") we see long delays in the jobs going from queued to running.
For our server, switching from "set_metadata_externally = False" to "set_metadata_externally = True" did not speed up anything.
Regards, Hans-Rudolf
On 09/19/2012 04:27 PM, Nate Coraor wrote:
On Sep 18, 2012, at 6:31 PM, Anthonius deBoer wrote:
Hi,
Jobs that I start on my in-house Galaxy instance now take up to 3-4 min to go from queued to Running, even though there is nothing much going on on the galaxy server...
I have been running this instance since June and use a relatively new version of Galaxy-central (Last update, 22-Aug changeset: 7535:bf6517b2b336)
I have noticed that my jobs table in the galaxy Postgres database contains about 60,000 jobs... Could that be the culprit? Does it slow to the complete database to see if there are any jobs that need to run?
Could I purge the jobs and related tables to speed up the submissions?
Hi Thon,
It's not really possible to remove things from the database since there are a lot of interdependencies.
The first thing to check would be whether set_metadata_externally = True in your Galaxy config.
--nate
Thanks,
Thon
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
On Tue, Oct 23, 2012 at 12:55 PM, Anthonius deBoer <thondeboer@me.com> wrote:
I had sent James Taylor the schema as he requested, but I never heard anything so I guess he must not have found anything weird
Yes, it appears to have all the right indexes to me.
On Oct 23, 2012, at 12:55 PM, Anthonius deBoer wrote:
I created the index but that did not change anything... I had sent James Taylor the schema as he requested, but I never heard anything so I guess he must not have found anything weird...
I am waiting to move my Galaxy server off to a three server configuration, so I am hoping the problem will not return, although your issue does not bode well for this issue...
Are the Galaxy server processes using a large amount of CPU or memory? Unfortunately, you may need to add a bit of debugging to the job code to see exactly where the delay is taking place. --nate
Regards,
Thon de Boer, Ph.D. Bioinformatics Guru +1-650-799-6839 thondeboer@me.com LinkedIn Profile
On Oct 23, 2012, at 4:14 AM, Hans-Rudolf Hotz <hrh@fmi.ch> wrote:
Hi Thon
Sorry for re-activating an old e-mail thread. I am just curious, whether Nate's suggestion solved your problem?
After our upgrade (from the "March 12th changeset 6799:40f1816d6857" to the "October 5th changeset 7824:b5bda7a5c345") we see long delays in the jobs going from queued to running.
For our server, switching from "set_metadata_externally = False" to "set_metadata_externally = True" did not speed up anything.
Regards, Hans-Rudolf
On 09/19/2012 04:27 PM, Nate Coraor wrote:
On Sep 18, 2012, at 6:31 PM, Anthonius deBoer wrote:
Hi,
Jobs that I start on my in-house Galaxy instance now take up to 3-4 min to go from queued to Running, even though there is nothing much going on on the galaxy server...
I have been running this instance since June and use a relatively new version of Galaxy-central (Last update, 22-Aug changeset: 7535:bf6517b2b336)
I have noticed that my jobs table in the galaxy Postgres database contains about 60,000 jobs... Could that be the culprit? Does it slow to the complete database to see if there are any jobs that need to run?
Could I purge the jobs and related tables to speed up the submissions?
Hi Thon,
It's not really possible to remove things from the database since there are a lot of interdependencies.
The first thing to check would be whether set_metadata_externally = True in your Galaxy config.
--nate
Thanks,
Thon
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Hi Thon Have you made any progress regarding the delays you've experienced? I had some time this morning and was digging in the code a little bit....in '~/lib/galaxy/jobs/manager.py' there is on line 155: time.sleep( 5 ) and I am wondering is this delay really necessary? - needless to say, if I remove it, jobs are executed ~5 seconds faster, and the time now roughly corresponds to the numbers I have seen before the upgrade. I doubt, there is a connection to the delays in minutes you've experienced, but I am pretty happy with the speed up in our case. Now. the big question: how safe is it to remove.... (or at least set to 1 sec)? - that's where my python and/or galaxy code knowledge comes to the end. Any comments from the developers appreciated. Thank you very much Hans-Rudolf On 10/23/2012 01:14 PM, Hans-Rudolf Hotz wrote:
Hi Thon
Sorry for re-activating an old e-mail thread. I am just curious, whether Nate's suggestion solved your problem?
After our upgrade (from the "March 12th changeset 6799:40f1816d6857" to the "October 5th changeset 7824:b5bda7a5c345") we see long delays in the jobs going from queued to running.
For our server, switching from "set_metadata_externally = False" to "set_metadata_externally = True" did not speed up anything.
Regards, Hans-Rudolf
On 09/19/2012 04:27 PM, Nate Coraor wrote:
On Sep 18, 2012, at 6:31 PM, Anthonius deBoer wrote:
Hi,
Jobs that I start on my in-house Galaxy instance now take up to 3-4 min to go from queued to Running, even though there is nothing much going on on the galaxy server...
I have been running this instance since June and use a relatively new version of Galaxy-central (Last update, 22-Aug changeset: 7535:bf6517b2b336)
I have noticed that my jobs table in the galaxy Postgres database contains about 60,000 jobs... Could that be the culprit? Does it slow to the complete database to see if there are any jobs that need to run?
Could I purge the jobs and related tables to speed up the submissions?
Hi Thon,
It's not really possible to remove things from the database since there are a lot of interdependencies.
The first thing to check would be whether set_metadata_externally = True in your Galaxy config.
--nate
Thanks,
Thon
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
participants (4)
-
Anthonius deBoer -
Hans-Rudolf Hotz -
James Taylor -
Nate Coraor