LSF cluster wierd behaviours!
Hello everyone, I have posted this earlier but I am afraid it did not go through I hope :). I was able to setup galaxy to work with our HPC cluster using the LSF scheduler. So far so good except with few exceptions: 1) I noticed one thing that submitting a job after a long period (for example overnight) the jobs do not get executed and more will not show up as jobs in the queue when I execute the "bjobs" command from the command line. As if the jobs were never submitted to the LSF. However, if I submit a job from the command (i.e >bsub sleep -5), then I check the jobs in the queue using the bjobs command I see this job as well as the other jobs that were submitted and could not see them before. Weird .... Has anyone seen this behavior before? Is this related to galaxy setup? Is there anything I should try out to get rid of such behavior? 2) Also related to LSF setup. Every time I restart galaxy it will not restart rather it will crash. Then if I start it again it will start after that. Here is the error I keep seeing after the first restart "galaxy.jobs.runners.state_handler_factory DEBUG 2015-08-04 08:12:17,484 Loaded 'failure' state handler from module galaxy.jobs.runners.state_handlers.resubmit " Any idea to get rid of this as well? Is this a job still in the database that I need to clean manually? If so can you tell me what table(s) to look into to clear out. 3. Finally, how do I control the resources (i.e cores for a job ) given to a submitted job on Galaxy? Thank you in advance for any tips or hints to resolve these issues. Best regards, Hak Disclaimer: This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient, any reading, printing, storage, disclosure, copying or any other action taken in respect of this e-mail is prohibited and may be unlawful. If you are not the intended recipient, please notify the sender immediately by using the reply function and then permanently delete what you have received. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Sidra Medical and Research Center.
Il 04.08.2015 07:57 Hakeem Almabrazi ha scritto:
Hello everyone,
I was able to setup galaxy to work with our HPC cluster using the LSF scheduler. So far so good except with few exceptions:
...
3. Finally, how do I control the resources (i.e cores for a job ) given to a submitted job on Galaxy?
Hi Hakeem, you need to specify new destinations in config/job_conf.xml, for example if you want to submit jobs asking for 4 cores on the same cluster node, use: <destination id="queue_name_4t" runner="drmaa"> <param id="nativeSpecification">-q queue_name -n 4 -R "span[hosts=1]"</param> </destination> Then in the <tools> section of config/job_conf.xml add <tool id="tool_id" destination="queue_name_4t"/> for each tool that should use 4 cores. More info at: https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster https://wiki.galaxyproject.org/Admin/Config/Jobs Ciao, Nicola Connetti gratis il mondo con la nuova indoona: hai la chat, le chiamate, le video chiamate e persino le chiamate di gruppo. E chiami gratis anche i numeri fissi e mobili nel mondo! Scarica subito l’app Vai su https://www.indoona.com/
Thank you Nicola, I will play with these options and see how it works. Have you played with LSF and Galaxy setup? Thank you, -----Original Message----- From: Nicola Soranzo [mailto:nsoranzo@tiscali.it] Sent: Tuesday, August 04, 2015 2:23 PM To: Hakeem Almabrazi Cc: galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] LSF cluster wierd behaviours! Il 04.08.2015 07:57 Hakeem Almabrazi ha scritto:
Hello everyone,
I was able to setup galaxy to work with our HPC cluster using the LSF scheduler. So far so good except with few exceptions:
...
3. Finally, how do I control the resources (i.e cores for a job ) given to a submitted job on Galaxy?
Hi Hakeem, you need to specify new destinations in config/job_conf.xml, for example if you want to submit jobs asking for 4 cores on the same cluster node, use: <destination id="queue_name_4t" runner="drmaa"> <param id="nativeSpecification">-q queue_name -n 4 -R "span[hosts=1]"</param> </destination> Then in the <tools> section of config/job_conf.xml add <tool id="tool_id" destination="queue_name_4t"/> for each tool that should use 4 cores. More info at: https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster https://wiki.galaxyproject.org/Admin/Config/Jobs Ciao, Nicola Connetti gratis il mondo con la nuova indoona: hai la chat, le chiamate, le video chiamate e persino le chiamate di gruppo. E chiami gratis anche i numeri fissi e mobili nel mondo! Scarica subito l’app Vai su https://www.indoona.com/ Disclaimer: This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient, any reading, printing, storage, disclosure, copying or any other action taken in respect of this e-mail is prohibited and may be unlawful. If you are not the intended recipient, please notify the sender immediately by using the reply function and then permanently delete what you have received. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Sidra Medical and Research Center.
Thank you Nicola,
I will play with
Yes, I'm using LSF with Galaxy at TGAC. Unfortunately I don't have suggestions for your other problems. Ciao, Nicola Il 04.08.2015 13:27 Hakeem Almabrazi ha scritto: these options and see how it works.
Have you played with LSF and
Galaxy setup?
Thank you,
-----Original Message----- From:
Nicola Soranzo [mailto:nsoranzo@tiscali.it [1]]
Sent: Tuesday, August 04, 2015 2:23 PM To: Hakeem Almabrazi Cc: galaxy-dev@lists.galaxyproject.org [2]Subject: Re: [galaxy-dev] LSF cluster wierd behaviours!
Il 04.08.2015 07:57 Hakeem Almabrazi ha scritto:
Hello everyone, I was able to setup galaxy to work with our HPC cluster using the LSF scheduler. So far so good except with few exceptions: ... 3. Finally, how do I control the resources (i.e cores for a job ) given to a submitted job on Galaxy?
Hi Hakeem, you need to specify new destinations in config/job_conf.xml, for example if you want to submit jobs asking for 4 cores on the same cluster node, use:-q queue_name -n 4 -R "span[hosts=1]"
Then in the section of config/job_conf.xml add
for each tool that should use 4 cores.
More info at:
https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster [3]
https://wiki.galaxyproject.org/Admin/Config/Jobs [4]
Ciao,
Nicola
Connetti gratis il mondo con la nuova indoona: hai la chat,
E chiami gratis anche i numeri fissi e mobili nel mondo! Scarica subito l'app Vai su https://www.indoona.com/ [5]
Disclaimer: This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient, any reading, printing, storage, disclosure, copying or any other action taken in respect of this e-mail is prohibited and may be unlawful. If you are not the intended recipient, please notify
le chiamate, le video chiamate e persino le chiamate di gruppo. the sender immediately by using the reply function and then permanently delete what you have received. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Sidra Medical and Research Center. Connetti gratis il mondo con la nuova indoona: hai la chat, le chiamate, le video chiamate e persino le chiamate di gruppo. E chiami gratis anche i numeri fissi e mobili nel mondo! Scarica subito l’app Vai su https://www.indoona.com/
Nicola, I have added your suggestion and seems to work ☺. Is there a way to control these parameters based on a tool? For example, lets say I want to require more cores for certain tools such as bwa. How can I do that? Here is my job_conf.xml. Please let me know if you have better suggestion to it. <job_conf> <plugins> <plugin id="drmaa" type="runner" load="galaxy.jobs.runners.drmaa:DRMAAJobRunner" workers="4"/> </plugins> <handlers default="handlers"> <!--handler id="main" /--> <handler id="handler0" tags="handlers"/> <!--handler id="handler1" tags="handlers"/--> </handlers> <destinations default="lsf"> <!--destination id="lsf" runner="drmaa"/--> <!-- suggested by Nicola--> <destination id="lsf" runner="drmaa"> <param id="nativeSpecification">-n 32 -R "span[hosts=1]"</param> </destination> </destinations> </job_conf> Regards, Hak From: Nicola Soranzo [mailto:nsoranzo@tiscali.it] Sent: Tuesday, August 04, 2015 2:37 PM To: Hakeem Almabrazi Cc: galaxy-dev@lists.galaxyproject.org Subject: RE: [galaxy-dev] LSF cluster wierd behaviours! Yes, I'm using LSF with Galaxy at TGAC. Unfortunately I don't have suggestions for your other problems. Ciao, Nicola Il 04.08.2015 13:27 Hakeem Almabrazi ha scritto: Thank you Nicola, I will play with these options and see how it works. Have you played with LSF and Galaxy setup? Thank you, -----Original Message----- From: Nicola Soranzo [mailto:nsoranzo@tiscali.it<mailto:nsoranzo@tiscali.it>] Sent: Tuesday, August 04, 2015 2:23 PM To: Hakeem Almabrazi Cc: galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org>Subject: Re: [galaxy-dev] LSF cluster wierd behaviours! Il 04.08.2015 07:57 Hakeem Almabrazi ha scritto: Hello everyone, I was able to setup galaxy to work with our HPC cluster using the LSF scheduler. So far so good except with few exceptions: ... 3. Finally, how do I control the resources (i.e cores for a job ) given to a submitted job on Galaxy? Hi Hakeem, you need to specify new destinations in config/job_conf.xml, for example if you want to submit jobs asking for 4 cores on the same cluster node, use:-q queue_name -n 4 -R "span[hosts=1]" Then in the section of config/job_conf.xml add for each tool that should use 4 cores. More info at: https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster https://wiki.galaxyproject.org/Admin/Config/Jobs Ciao, Nicola Connetti gratis il mondo con la nuova indoona: hai la chat, le chiamate, le video chiamate e persino le chiamate di gruppo. E chiami gratis anche i numeri fissi e mobili nel mondo! Scarica subito l’app Vai su https://www.indoona.com/ Disclaimer: This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient, any reading, printing, storage, disclosure, copying or any other action taken in respect of this e-mail is prohibited and may be unlawful. If you are not the intended recipient, please notify the sender immediately by using the reply function and then permanently delete what you have received. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Sidra Medical and Research Center. Connetti gratis il mondo con la nuova indoona: hai la chat, le chiamate, le video chiamate e persino le chiamate di gruppo. E chiami gratis anche i numeri fissi e mobili nel mondo! Scarica subito l’app Vai su https://www.indoona.com/ Disclaimer: This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient, any reading, printing, storage, disclosure, copying or any other action taken in respect of this e-mail is prohibited and may be unlawful. If you are not the intended recipient, please notify the sender immediately by using the reply function and then permanently delete what you have received. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Sidra Medical and Research Center.
Hi Hakeem, you have to _add_ a destination for each number of cores you need, and then specify this destination in the corresponding tools as I said in the previous email, i.e.: <job_conf> <plugins> <plugin id="drmaa" type="runner" load="galaxy.jobs.runners.drmaa:DRMAAJobRunner" workers="4"/> </plugins> <handlers default="handlers"> <!--handler id="main" /--> <handler id="handler0" tags="handlers"/> <!--handler id="handler1" tags="handlers"/--> </handlers> <destinations default="lsf"> <destination id="lsf" runner="drmaa"/> <destination id="lsf_4t" runner="drmaa"> <param id="nativeSpecification">-n 4 -R "span[hosts=1]"</param> </destination> <destination id="lsf_32t" runner="drmaa"> <param id="nativeSpecification">-n 32 -R "span[hosts=1]"</param> </destination> </destinations> <tools> <tool id="spades" destination="lsf_4t"/> <tool id="bwa" destination="lsf_32t"/> </tools> </job_conf> Cheers, Nicola Il 04.08.2015 14:12 Hakeem Almabrazi ha scritto:
Nicola,
I have added your suggestion and seems to work J. Is there a way to control these parameters based on a tool? For example, lets say I want to require more cores for certain tools such as bwa. How can I do that?
Here is my job_conf.xml. Please let me know if you have better suggestion to it.
<job_conf> <plugins> <plugin id="drmaa" type="runner" load="galaxy.jobs.runners.drmaa:DRMAAJobRunner" workers="4"/> </plugins> <handlers default="handlers"> <!--handler id="main" /--> <handler id="handler0" tags="handlers"/> <!--handler id="handler1" tags="handlers"/--> </handlers> <destinations default="lsf"> <!--destination id="lsf" runner="drmaa"/--> <!-- suggested by Nicola--> <destination id="lsf" runner="drmaa"> <param id="nativeSpecification">-n 32 -R "span[hosts=1]"</param> </destination> </destinations> </job_conf>
Regards,
Hak
FROM: Nicola Soranzo [mailto:nsoranzo@tiscali.it] SENT: Tuesday, August 04, 2015 2:37 PM TO: Hakeem Almabrazi CC: galaxy-dev@lists.galaxyproject.org SUBJECT: RE: [galaxy-dev] LSF cluster wierd behaviours!
Yes, I'm using LSF with Galaxy at TGAC.
Unfortunately I don't have suggestions for your other problems.
Ciao, Nicola
Il 04.08.2015 13:27 Hakeem Almabrazi ha scritto:
Thank you Nicola,
I will play with these options and see how it works.
Have you played with LSF and Galaxy setup?
Thank you,
-----Original Message-----
From: Nicola Soranzo [mailto:nsoranzo@tiscali.it [1]]
Sent: Tuesday, August 04, 2015 2:23 PM
To: Hakeem Almabrazi
Cc: galaxy-dev@lists.galaxyproject.org [2]Subject: Re: [galaxy-dev] LSF cluster wierd behaviours!
Il 04.08.2015 07:57 Hakeem Almabrazi ha scritto:
Hello everyone, I was able to setup galaxy to work with our HPC cluster using the LSF scheduler. So far so good except with few exceptions: ... 3. Finally, how do I control the resources (i.e cores for a job ) given to a submitted job on Galaxy?
Hi Hakeem,
you need to specify new destinations in config/job_conf.xml, for example if you want to submit jobs asking for 4 cores on the same cluster node, use:-q queue_name -n 4 -R "span[hosts=1]"
Then in the section of config/job_conf.xml add
for each tool that should use 4 cores.
More info at:
https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster [3]
https://wiki.galaxyproject.org/Admin/Config/Jobs [4]
Ciao,
Nicola
Connetti gratis il mondo con la nuova indoona: hai la chat, le chiamate, le video chiamate e persino le chiamate di gruppo.
E chiami gratis anche i numeri fissi e mobili nel mondo!
Scarica subito l'app Vai su https://www.indoona.com/ [5]
Disclaimer: This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient, any reading, printing, storage, disclosure, copying or any other action taken in respect of this e-mail is prohibited and may be unlawful. If you are not the intended recipient, please notify the sender immediately by using the reply function and then permanently delete what you have received. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Sidra Medical and Research Center.
Connetti gratis il mondo con la nuova indoona: hai la chat, le chiamate, le video chiamate e persino le chiamate di gruppo. E chiami gratis anche i numeri fissi e mobili nel mondo! Scarica subito l'app Vai su https://www.indoona.com/ [6] Disclaimer: This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient, any reading, printing, storage, disclosure, copying or any other action taken in respect of this e-mail is prohibited and may be unlawful. If you are not the intended recipient, please notify the sender immediately by using the reply function and then permanently delete what you have received. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Sidra Medical and Research Center.
Links: ------ [1] mailto:nsoranzo@tiscali.it [2] mailto:galaxy-dev@lists.galaxyproject.org [3] https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster [4] https://wiki.galaxyproject.org/Admin/Config/Jobs [5] https://www.indoona.com/ [6] https://www.indoona.com/ Connetti gratis il mondo con la nuova indoona: hai la chat, le chiamate, le video chiamate e persino le chiamate di gruppo. E chiami gratis anche i numeri fissi e mobili nel mondo! Scarica subito l’app Vai su https://www.indoona.com/
participants (2)
-
Hakeem Almabrazi -
Nicola Soranzo