Hi,
I am currently working on a workflow with some custom tools, and one of these tools has to create very big temporary files (like 45Go big). As this workflow will be used on a lot of file at the same time, I have to keep it from running more than once or twice at the same time (the other execution would wait in the queue). If I don't, I'm afraid that some memory lack or something could cause all the executions to fail and stop.
The problem is : I can't find if it's possible to do that, and if it is, how do I do it.
Anyone has an idea please ?
On Tue, May 13, 2014 at 11:11 AM, Guillaume Penderia g.penderia@gmail.com wrote:
Hi,
I am currently working on a workflow with some custom tools, and one of these tools has to create very big temporary files (like 45Go big). As this workflow will be used on a lot of file at the same time, I have to keep it from running more than once or twice at the same time (the other execution would wait in the queue). If I don't, I'm afraid that some memory lack or something could cause all the executions to fail and stop.
The problem is : I can't find if it's possible to do that, and if it is, how do I do it.
Anyone has an idea please ?
If you are using a cluster, one idea would be to setup a dedicated queue for these big jobs, configured to ensure only one runs at a time. Or at least, only one per cluster node.
Peter
Hi Guillaume,
you can set a limit for "concurrent_jobs" in job_conf.xml file. Please have a look at the job_conf.xml.sample_advanced file for more information.
Cheers, Bjoern
Am 13.05.2014 12:11, schrieb Guillaume Penderia:
Hi,
I am currently working on a workflow with some custom tools, and one of these tools has to create very big temporary files (like 45Go big). As this workflow will be used on a lot of file at the same time, I have to keep it from running more than once or twice at the same time (the other execution would wait in the queue). If I don't, I'm afraid that some memory lack or something could cause all the executions to fail and stop.
The problem is : I can't find if it's possible to do that, and if it is, how do I do it.
Anyone has an idea please ?
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Peter I am not using a cluster, I am on a big computer with a lot of RAM. Plus I wouldn't know how to configure this, but thank you anyway !
Björn I didn't know about this file, but it seems really useful for what I'm willing to do. I don't totally understand what I should put in it yet, but I'm going to collect some informations about it to see if I can figure it out by myself before I decide if I have to bother you again. Thank you !
Guillaume
2014-05-13 12:16 GMT+02:00 Björn Grüning bjoern.gruening@gmail.com:
Hi Guillaume,
you can set a limit for "concurrent_jobs" in job_conf.xml file. Please have a look at the job_conf.xml.sample_advanced file for more information.
Cheers, Bjoern
Am 13.05.2014 12:11, schrieb Guillaume Penderia:
Hi,
I am currently working on a workflow with some custom tools, and one of these tools has to create very big temporary files (like 45Go big). As this workflow will be used on a lot of file at the same time, I have to keep it from running more than once or twice at the same time (the other execution would wait in the queue). If I don't, I'm afraid that some memory lack or something could cause all the executions to fail and stop.
The problem is : I can't find if it's possible to do that, and if it is, how do I do it.
Anyone has an idea please ?
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
galaxy-dev@lists.galaxyproject.org
-
Björn Grüning -
Guillaume Penderia -
Peter Cock