Dear Galaxy managers,
I would like to ask about the queuing rules for the workflows before processed on the server?
I use my customized Galaxy workflow which contains 22 following steps (basically, filtering, trimming and format change of NGS data). I guess the server is generally very busy during last weeks/months (?), so my job was waiting about 24 hours in a queue (which would not be a problem), and then the first step of the workflow was processed, but the following 21 is again/still waiting (already for another couple of hours...). It makes me wondering about the queuing rules because I expected that the whole workflow is queued as one job.. Then my question is if the whole workflow, once submitted, is listed in the queue, or does the following step queue only after the previous step is finished (which would mean to wait the whole queue for each step of the workflow...)?
I routinely used those wrokflows before (months ago) without any problems...
I tried to search similar question in the archive before I posted this one...
Thanks a lot for your answer,
My best
Zuzana Musilova
Hi Zuzana,
Jobs queue individually. But different jobs may queue to different clusters, so if there are no dependencies, you may see jobs execute out of order. Workflows are still a benefit as this places jobs in the queue sooner than waiting for each job to finished (under normal circumstances - not the extended queue times seen now during the server reconfiguration).
The banner on main provides updates about the status of Main, but right now NGS jobs are in hold. We expect these to start again tomorrow and then for the final stages of the upgrades to be complete within the next few weeks.
Apologies for the slow reply to your question,
Jen Galaxy team
On 7/24/13 2:33 PM, zuzmus wrote:
Dear Galaxy managers,
I would like to ask about the queuing rules for the workflows before processed on the server?
I use my customized Galaxy workflow which contains 22 following steps (basically, filtering, trimming and format change of NGS data). I guess the server is generally very busy during last weeks/months (?), so my job was waiting about 24 hours in a queue (which would not be a problem), and then the first step of the workflow was processed, but the following 21 is again/still waiting (already for another couple of hours...). It makes me wondering about the queuing rules because I expected that the whole workflow is queued as one job.. Then my question is if the whole workflow, once submitted, is listed in the queue, or does the following step queue only after the previous step is finished (which would mean to wait the whole queue for each step of the workflow...)?
I routinely used those wrokflows before (months ago) without any problems...
I tried to search similar question in the archive before I posted this one...
Thanks a lot for your answer,
My best
Zuzana Musilova
Zuzana Musilova, PhD. (zuzmus@gmail.com mailto:zuzmus@gmail.com)
Zoological Institute University of Basel Vesalgasse 1, CH-4051 Basel Switzerland - Europe
The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
To search Galaxy mailing lists use the unified search at:
Hi Jennifer,
Things are going better in the main server of Galaxy since the beginning of this week: thanks for these last improvements !
However, I started worflows of NGS analysis last monday (30th of september) for two histories (I guess something similar to Zuzana's worflow). Since I started it, all the steps of the worflow are "waiting to run" ("grey status"). I tried many times to start them by clicking on "Resume paused jobs" (in the history wheel-tool menu), but nothing happened. As you said in your reply to Zuzana, NGS jobs are still in hold so far. Do you confirm me I ought to wait (and avoid cancelling the worflow to start a new one, as it would end up in more delay)?
At the same time, I am doing other NGS analysis with other histories (GATK Base Quality Recalibration and Unified Genotyper) and these jobs are working well. This time, I am doing all the steps sequentially, without worflows. Do individual NGS jobs run better than worflows or is it just variable from tool to tool ?
Thank you for reply and for all your constant efforts to maintain this very useful plateform in shape !
Fabrice
Le 03/10/2013 00:20, Jennifer Jackson a écrit :
Hi Zuzana,
Jobs queue individually. But different jobs may queue to different clusters, so if there are no dependencies, you may see jobs execute out of order. Workflows are still a benefit as this places jobs in the queue sooner than waiting for each job to finished (under normal circumstances
- not the extended queue times seen now during the server reconfiguration).
The banner on main provides updates about the status of Main, but right now NGS jobs are in hold. We expect these to start again tomorrow and then for the final stages of the upgrades to be complete within the next few weeks.
Apologies for the slow reply to your question,
Jen Galaxy team
On 7/24/13 2:33 PM, zuzmus wrote:
Dear Galaxy managers,
I would like to ask about the queuing rules for the workflows before processed on the server?
I use my customized Galaxy workflow which contains 22 following steps (basically, filtering, trimming and format change of NGS data). I guess the server is generally very busy during last weeks/months (?), so my job was waiting about 24 hours in a queue (which would not be a problem), and then the first step of the workflow was processed, but the following 21 is again/still waiting (already for another couple of hours...). It makes me wondering about the queuing rules because I expected that the whole workflow is queued as one job.. Then my question is if the whole workflow, once submitted, is listed in the queue, or does the following step queue only after the previous step is finished (which would mean to wait the whole queue for each step of the workflow...)?
I routinely used those wrokflows before (months ago) without any problems...
I tried to search similar question in the archive before I posted this one...
Thanks a lot for your answer,
My best
Zuzana Musilova
Zuzana Musilova, PhD. (zuzmus@gmail.com mailto:zuzmus@gmail.com)
Zoological Institute University of Basel Vesalgasse 1, CH-4051 Basel Switzerland - Europe
The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
To search Galaxy mailing lists use the unified search at:
The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
To search Galaxy mailing lists use the unified search at:
Hello Fabrice,
On 10/3/13 2:07 AM, Fabrice Besnard wrote:
Hi Jennifer,
Things are going better in the main server of Galaxy since the beginning of this week: thanks for these last improvements !
Glad to hear that, much more is on the way!
However, I started worflows of NGS analysis last monday (30th of september) for two histories (I guess something similar to Zuzana's worflow). Since I started it, all the steps of the worflow are "waiting to run" ("grey status"). I tried many times to start them by clicking on "Resume paused jobs" (in the history wheel-tool menu), but nothing happened. As you said in your reply to Zuzana, NGS jobs are still in hold so far. Do you confirm me I ought to wait (and avoid cancelling the worflow to start a new one, as it would end up in more delay)?
Canceling and re-starting workflows will do nothing but move your work back to the end of the queue. I would leave queued jobs alone unless you no longer want to do that analysis anymore (permanently delete if really do not need).
At the same time, I am doing other NGS analysis with other histories (GATK Base Quality Recalibration and Unified Genotyper) and these jobs are working well. This time, I am doing all the steps sequentially, without worflows. Do individual NGS jobs run better than worflows or is it just variable from tool to tool ?
Worflows queue jobs sooner, so are a good strategy no matter the tool - all these do is add multiple tool jobs to your history at once. From there, it is just the same as if you started the jobs individually (but somehow at the same time!).
It is important to note here that your jobs are distributed "per user". If you have a bunch of workflows running in different histories, then these will be competing for the job slot when it is your turn to have a job run. The job queued earliest will run. When the server is very busy, be sure to queue up what is most important first.
Thank you for reply and for all your constant efforts to maintain this very useful plateform in shape !
Thanks for our great development team and active development community. They keep it all together!
Take care!
Jen Galaxy team
Fabrice
Le 03/10/2013 00:20, Jennifer Jackson a écrit :
Hi Zuzana,
Jobs queue individually. But different jobs may queue to different clusters, so if there are no dependencies, you may see jobs execute out of order. Workflows are still a benefit as this places jobs in the queue sooner than waiting for each job to finished (under normal circumstances
- not the extended queue times seen now during the server reconfiguration).
The banner on main provides updates about the status of Main, but right now NGS jobs are in hold. We expect these to start again tomorrow and then for the final stages of the upgrades to be complete within the next few weeks.
Apologies for the slow reply to your question,
Jen Galaxy team
On 7/24/13 2:33 PM, zuzmus wrote:
Dear Galaxy managers,
I would like to ask about the queuing rules for the workflows before processed on the server?
I use my customized Galaxy workflow which contains 22 following steps (basically, filtering, trimming and format change of NGS data). I guess the server is generally very busy during last weeks/months (?), so my job was waiting about 24 hours in a queue (which would not be a problem), and then the first step of the workflow was processed, but the following 21 is again/still waiting (already for another couple of hours...). It makes me wondering about the queuing rules because I expected that the whole workflow is queued as one job.. Then my question is if the whole workflow, once submitted, is listed in the queue, or does the following step queue only after the previous step is finished (which would mean to wait the whole queue for each step of the workflow...)?
I routinely used those wrokflows before (months ago) without any problems...
I tried to search similar question in the archive before I posted this one...
Thanks a lot for your answer,
My best
Zuzana Musilova
Zuzana Musilova, PhD. (zuzmus@gmail.com mailto:zuzmus@gmail.com)
Zoological Institute University of Basel Vesalgasse 1, CH-4051 Basel Switzerland - Europe
The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-devTo manage your subscriptions to this and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
To search Galaxy mailing lists use the unified search at:
galaxy-user@lists.galaxyproject.org
-
Fabrice Besnard -
Jennifer Jackson -
zuzmus