showing progress on running tasks
Hi all - I know this isn't currently a feature, but wanted to find out about what it would take to implement.... We developed some custom tools that are long running....multiple hours. The tools run through various steps of an analysis. Is there a way to display progress of a task in the history? Possibly as a small progress bar? Or provide some feedback via an html page in the main frame? Ryan
Hi Ryan, This is a fairly complex problem because the length of the run depends most often on input size, job algorithms, and computational capacity all of which are quite difficult to estimate. We have discussed it before and one of the ways to approach this is to gather statistics from as many jobs as possible and then provide the user with some estimates based on this data on similar jobs to theirs. For the way to gather the data please see https://github.com/galaxyproject/galaxy/pull/2455 from Eric Rasche. With the data (or at least data format) we can start working on an algorithm that would make a guess on a new job length. However, there are many other approaches to this problem. Thanks, Martin On Fri, Jul 8, 2016 at 10:34 AM Ryan G <ngsbioinformatics@gmail.com> wrote:
Hi all - I know this isn't currently a feature, but wanted to find out about what it would take to implement....
We developed some custom tools that are long running....multiple hours. The tools run through various steps of an analysis.
Is there a way to display progress of a task in the history? Possibly as a small progress bar? Or provide some feedback via an html page in the main frame?
Ryan
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Martin - I was actually thinking to let the tool itself tell Galaxy about the progress instead of Galaxy estimating that. If instance, if the tool know it has 5 stages, it can report back to Galaxy 20,40,60,80,and 100 % complete. I know this is complex but could be a nice feature. Please excuse any typos -- Sent from my iPhone
On Jul 8, 2016, at 10:47 AM, Martin Čech <marten@bx.psu.edu> wrote:
Hi Ryan,
This is a fairly complex problem because the length of the run depends most often on input size, job algorithms, and computational capacity all of which are quite difficult to estimate. We have discussed it before and one of the ways to approach this is to gather statistics from as many jobs as possible and then provide the user with some estimates based on this data on similar jobs to theirs.
For the way to gather the data please see https://github.com/galaxyproject/galaxy/pull/2455 from Eric Rasche. With the data (or at least data format) we can start working on an algorithm that would make a guess on a new job length.
However, there are many other approaches to this problem.
Thanks,
Martin
On Fri, Jul 8, 2016 at 10:34 AM Ryan G <ngsbioinformatics@gmail.com> wrote: Hi all - I know this isn't currently a feature, but wanted to find out about what it would take to implement....
We developed some custom tools that are long running....multiple hours. The tools run through various steps of an analysis.
Is there a way to display progress of a task in the history? Possibly as a small progress bar? Or provide some feedback via an html page in the main frame?
Ryan
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Ryan, multiple tool stages do not necessarily equally split runtime so you wouldn't have 'time progress bar' but 'unscaled progress bar'. The usefulness of such I would put somewhere in between a loading animation and stats estimate. This would also require every tool to be enhanced as well as the framework as opposed to the proposed stats solution that could work with every used tool building on past data without putting more requirements on the tool developer. Whole another book of problems would be how (and if) is this handled in job schedulers such as Slurm - where most of the big jobs live. To summarize: having tool reporting progress stages is possible to implement but has many drawbacks. Thanks, Martin On Fri, Jul 8, 2016 at 12:41 PM Ryan G <ngsbioinformatics@gmail.com> wrote:
Hi Martin - I was actually thinking to let the tool itself tell Galaxy about the progress instead of Galaxy estimating that.
If instance, if the tool know it has 5 stages, it can report back to Galaxy 20,40,60,80,and 100 % complete.
I know this is complex but could be a nice feature.
Please excuse any typos -- Sent from my iPhone
On Jul 8, 2016, at 10:47 AM, Martin Čech <marten@bx.psu.edu> wrote:
Hi Ryan,
This is a fairly complex problem because the length of the run depends most often on input size, job algorithms, and computational capacity all of which are quite difficult to estimate. We have discussed it before and one of the ways to approach this is to gather statistics from as many jobs as possible and then provide the user with some estimates based on this data on similar jobs to theirs.
For the way to gather the data please see https://github.com/galaxyproject/galaxy/pull/2455 from Eric Rasche. With the data (or at least data format) we can start working on an algorithm that would make a guess on a new job length.
However, there are many other approaches to this problem.
Thanks,
Martin
On Fri, Jul 8, 2016 at 10:34 AM Ryan G <ngsbioinformatics@gmail.com> wrote:
Hi all - I know this isn't currently a feature, but wanted to find out about what it would take to implement....
We developed some custom tools that are long running....multiple hours. The tools run through various steps of an analysis.
Is there a way to display progress of a task in the history? Possibly as a small progress bar? Or provide some feedback via an html page in the main frame?
Ryan
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Ryan, On Sat, Jul 9, 2016 at 12:40 AM, Ryan G <ngsbioinformatics@gmail.com> wrote:
Hi Martin - I was actually thinking to let the tool itself tell Galaxy about the progress instead of Galaxy estimating that.
If instance, if the tool know it has 5 stages, it can report back to Galaxy 20,40,60,80,and 100 % complete.
If you're willing for something less elegant, instead of "reporting back" to Galaxy, you can just use some scripting and a cron job (to run the script periodically) to generate an HTML file. An obvious choice is perhaps static/welcome.html , the page that is shown when a user logs in. It's not perfect, but (IMHO), it's better than nothing. And maybe if you said "stage 1 complete" instead of "20% complete", it might be less misleading (since what you want to say is 20% of the stages have completed and not really 20% of the total running time). I haven't yet tried the static/welcome.html idea, but I'm planning to use it in a slightly different way. I want to allow users to at least see how many jobs are queued by the job scheduler. (It is misleading for many reasons -- least of which, the time for each job is still unknown -- but maybe better than nothing. Sort of like seeing how many people are in line at the bank before joining the line...you have no idea how long each customer ahead of you will take.) Ray
participants (3)
-
Martin Čech -
Raymond Wan -
Ryan G