Keeping a tool from running multiple times at once
Hi, I am currently working on a workflow with some custom tools, and one of these tools has to create very big temporary files (like 45Go big). As this workflow will be used on a lot of file at the same time, I have to keep it from running more than once or twice at the same time (the other execution would wait in the queue). If I don't, I'm afraid that some memory lack or something could cause all the executions to fail and stop. The problem is : I can't find if it's possible to do that, and if it is, how do I do it. Anyone has an idea please ?
On Tue, May 13, 2014 at 11:11 AM, Guillaume Penderia <g.penderia@gmail.com> wrote:
Hi,
I am currently working on a workflow with some custom tools, and one of these tools has to create very big temporary files (like 45Go big). As this workflow will be used on a lot of file at the same time, I have to keep it from running more than once or twice at the same time (the other execution would wait in the queue). If I don't, I'm afraid that some memory lack or something could cause all the executions to fail and stop.
The problem is : I can't find if it's possible to do that, and if it is, how do I do it.
Anyone has an idea please ?
If you are using a cluster, one idea would be to setup a dedicated queue for these big jobs, configured to ensure only one runs at a time. Or at least, only one per cluster node. Peter
Hi Guillaume, you can set a limit for "concurrent_jobs" in job_conf.xml file. Please have a look at the job_conf.xml.sample_advanced file for more information. Cheers, Bjoern Am 13.05.2014 12:11, schrieb Guillaume Penderia:
Hi,
I am currently working on a workflow with some custom tools, and one of these tools has to create very big temporary files (like 45Go big). As this workflow will be used on a lot of file at the same time, I have to keep it from running more than once or twice at the same time (the other execution would wait in the queue). If I don't, I'm afraid that some memory lack or something could cause all the executions to fail and stop.
The problem is : I can't find if it's possible to do that, and if it is, how do I do it.
Anyone has an idea please ?
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Peter I am not using a cluster, I am on a big computer with a lot of RAM. Plus I wouldn't know how to configure this, but thank you anyway ! Björn I didn't know about this file, but it seems really useful for what I'm willing to do. I don't totally understand what I should put in it yet, but I'm going to collect some informations about it to see if I can figure it out by myself before I decide if I have to bother you again. Thank you ! Guillaume 2014-05-13 12:16 GMT+02:00 Björn Grüning <bjoern.gruening@gmail.com>:
Hi Guillaume,
you can set a limit for "concurrent_jobs" in job_conf.xml file. Please have a look at the job_conf.xml.sample_advanced file for more information.
Cheers, Bjoern
Am 13.05.2014 12:11, schrieb Guillaume Penderia:
Hi,
I am currently working on a workflow with some custom tools, and one of these tools has to create very big temporary files (like 45Go big). As this workflow will be used on a lot of file at the same time, I have to keep it from running more than once or twice at the same time (the other execution would wait in the queue). If I don't, I'm afraid that some memory lack or something could cause all the executions to fail and stop.
The problem is : I can't find if it's possible to do that, and if it is, how do I do it.
Anyone has an idea please ?
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
participants (3)
-
Björn Grüning -
Guillaume Penderia -
Peter Cock