Hello, If I were to stop and re-start the Galaxy server with the run.sh script would jobs that are aleady running or are in the queue be lost when the server is re-started? If so, is there another method to use which would not lose the running or queued jobs? Thank you.
Unless something has recently changed, you will need to set up a cluster to get jobs to persist across galaxy restarts. You can do this with just a single node acting as both queue master and job runner using sun grid engine (i think, I have not tried this myself) Brad On Aug 27, 2012, at 12:51 PM, kauerbach@comcast.net<mailto:kauerbach@comcast.net> wrote: Hello, If I were to stop and re-start the Galaxy server with the run.sh script would jobs that are aleady running or are in the queue be lost when the server is re-started? If so, is there another method to use which would not lose the running or queued jobs? Thank you. ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Brad Langhorst langhorst@neb.com<mailto:langhorst@neb.com> 978-380-7564
If you refer to using one machine as master AND host simultaneously, which you never tried: let me tell you it does work well with SGE/OGE. I have it running, and it just works out great. It enhances the load control on that machine, due to the great configuration possibilities on queue level (max. cores, queue ranking, user restrictions etc.). I did not connect it to Galaxy yet (which is also installed on the same machine), but I do not see a reason why that should not work out. Indeed, the reason for acting like this was the need to set up a modularized system, using defined interfaces. Currently everything is physically in on one machine, but that will change. Galaxy for example does not care, where and how a cluster is set up physically, it just connects to a given ressource. Regarding the initial question: at least jobs in the cluster queue will indeed finish, because after submission those jobs are decoupled from Galaxy, by definition; it's like submitting from a desktop machine, which can be switched off afterwards. Additionally, I would wonder if Galaxy (which has really great fail safe mechanisms) would not be able to reconnect to the cluster, asking for jobs, which were submitted before restart and not noted as "finished" or "picked up" after restart (noted internally within the DB). I would say test it and get back to the mailing list :). Best, Sebastian Langhorst, Brad schrieb:
Unless something has recently changed, you will need to set up a cluster to get jobs to persist across galaxy restarts.
You can do this with just a single node acting as both queue master and job runner using sun grid engine (i think, I have not tried this myself)
Brad On Aug 27, 2012, at 12:51 PM, kauerbach@comcast.net<mailto:kauerbach@comcast.net> wrote:
Hello,
If I were to stop and re-start the Galaxy server with the run.sh script would jobs that are aleady running or are in the queue be lost when the server is re-started? If so, is there another method to use which would not lose the running or queued jobs?
Thank you.
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Brad Langhorst langhorst@neb.com<mailto:langhorst@neb.com> 978-380-7564
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Sebastian Schaaf, M.Sc. Bioinformatics Chair of Biometry and Bioinformatics Department of Medical Information Sciences, Biometry and Epidemiology University of Munich Marchioninistr. 15, K U1 (postal) Marchioninistr. 17, U 006 (office) D-81377 Munich (Germany) Tel: +49 89 2180-78178
On Aug 27, 2012, at 18:13, "Sebastian Schaaf" <schaaf@ibe.med.uni-muenchen.de> wrote:
If you refer to using one machine as master AND host simultaneously, which you never tried: let me tell you it does work well with SGE/OGE. I have it running, and it just works out great. It enhances the load control on that machine, due to the great configuration possibilities on queue level (max. cores, queue ranking, user restrictions etc.). I did not connect it to Galaxy yet (which is also installed on the same machine), but I do not see a reason why that should not work out. Indeed, the reason for acting like this was the need to set up a modularized system, using defined interfaces. Currently everything is physically in on one machine, but that will change. Galaxy for example does not care, where and how a cluster is set up physically, it just connects to a given ressource.
Regarding the initial question: at least jobs in the cluster queue will indeed finish, because after submission those jobs are decoupled from Galaxy, by definition; it's like submitting from a desktop machine, which can be switched off afterwards. Additionally, I would wonder if Galaxy (which has really great fail safe mechanisms) would not be able to reconnect to the cluster, asking for jobs, which were submitted before restart and not noted as "finished" or "picked up" after restart (noted internally within the DB). I would say test it and get back to the mailing list :).
Hi all, It should work exactly as Sebastian has described. A single node as the entire SGE resource and Galaxy server is perfectly fine. Please let us know if you run in to any problems. --nate
Best, Sebastian
Langhorst, Brad schrieb:
Unless something has recently changed, you will need to set up a cluster to get jobs to persist across galaxy restarts.
You can do this with just a single node acting as both queue master and job runner using sun grid engine (i think, I have not tried this myself)
Brad On Aug 27, 2012, at 12:51 PM, kauerbach@comcast.net<mailto:kauerbach@comcast.net> wrote:
Hello,
If I were to stop and re-start the Galaxy server with the run.sh script would jobs that are aleady running or are in the queue be lost when the server is re-started? If so, is there another method to use which would not lose the running or queued jobs?
Thank you.
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Brad Langhorst langhorst@neb.com<mailto:langhorst@neb.com> 978-380-7564
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Sebastian Schaaf, M.Sc. Bioinformatics Chair of Biometry and Bioinformatics Department of Medical Information Sciences, Biometry and Epidemiology University of Munich Marchioninistr. 15, K U1 (postal) Marchioninistr. 17, U 006 (office) D-81377 Munich (Germany) Tel: +49 89 2180-78178
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
On a related note, it's definitely nice to be able to consolidate the configuration into a single universe_wsi.ini, but --daemon and --stop-daemon are not sufficient for smooth operation of a multi web and job runner instance of Galaxy. I wonder if anyone's already adapted run.sh to restart the runners one by one to accommodate addition of tools and reference datasets without disrupting the user experience. Cheers, Alex On Aug 28, 2012, at 9:03 AM, Nate Coraor <nate@bx.psu.edu> wrote:
On Aug 27, 2012, at 18:13, "Sebastian Schaaf" <schaaf@ibe.med.uni-muenchen.de> wrote:
If you refer to using one machine as master AND host simultaneously, which you never tried: let me tell you it does work well with SGE/OGE. I have it running, and it just works out great. It enhances the load control on that machine, due to the great configuration possibilities on queue level (max. cores, queue ranking, user restrictions etc.). I did not connect it to Galaxy yet (which is also installed on the same machine), but I do not see a reason why that should not work out. Indeed, the reason for acting like this was the need to set up a modularized system, using defined interfaces. Currently everything is physically in on one machine, but that will change. Galaxy for example does not care, where and how a cluster is set up physically, it just connects to a given ressource.
Regarding the initial question: at least jobs in the cluster queue will indeed finish, because after submission those jobs are decoupled from Galaxy, by definition; it's like submitting from a desktop machine, which can be switched off afterwards. Additionally, I would wonder if Galaxy (which has really great fail safe mechanisms) would not be able to reconnect to the cluster, asking for jobs, which were submitted before restart and not noted as "finished" or "picked up" after restart (noted internally within the DB). I would say test it and get back to the mailing list :).
participants (5)
-
kauerbach＠comcast.net -
Langhorst, Brad -
Nate Coraor -
Oleksandr Moskalenko -
Sebastian Schaaf