Dear Galaxy admins, I am contacting you due to multiple problems during the installation of our own server of galaxy. We have been struggling to install galaxy on our HPC system for 1.5 year. We have faced so many problems while installing galaxy that we have lost count. Yet, our main problem was and remains the use of pulsar... Even after following two training sessions, switched to a brand new CentOS 8 cluster (while CentOS 8 will die at the end of the year), we are still not able to use our galaxy instance. We followed the ansible documentation and we are now able to reproduce the training sessions (i.e, execute an analysis with pulsar on our HPC system). Nevertheless, we are not able to connect pulsar with slurm (which makes it unusable). We do not want to use DRMAA because it is not well maintained and compatible with slurm. Instead, we choose to use CLI (as mentioned in the galaxy docs) but we are stuck (see below the pulsarservers.yaml file). Is it possible to use something else than DRMAA to connect pulsar to the scheduler? Thanks, Best regards Luc Cornet - - pulsarservers.yaml - - # Put your Galaxy server's fully qualified domain name (FQDN) (or the FQDN of the RabbitMQ server) above. pulsar_root: /opt/pulsar pulsar_pip_install: true pulsar_pycurl_ssl_library: openssl pulsar_systemd: true pulsar_systemd_runner: webless pulsar_create_user: false pulsar_user: {name: pulsar, shell: /bin/bash} pulsar_optional_dependencies: - pyOpenSSL # For remote transfers initiated on the Pulsar end rather than the Galaxy end - pycurl # drmaa required if connecting to an external DRM using it. - drmaa # kombu needed if using a message queue - kombu # amqp 5.0.3 changes behaviour in an unexpected way, pin for now. - 'amqp==5.0.2' # psutil and pylockfile are optional dependencies but can make Pulsar # more robust in small ways. - psutil pulsar_yaml_config: conda_auto_init: True conda_auto_install: True staging_directory: "{{ pulsar_staging_dir }}" persistence_directory: "{{ pulsar_persistence_dir }}" tool_dependency_dir: "{{ pulsar_dependencies_dir }}" # The following are the settings for the pulsar server to contact the message queue with related timeouts etc. message_queue_url: "pyamqp://galaxy_au:{{ rabbitmq_password_galaxy_au }}@{{ galaxy_server_url }}:5671//pulsar/galaxy_au?ssl=1" managers: _default_: type: queued_cli job_plugin: slurm native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" min_polling_interval: 0.5 amqp_publish_retry: True amqp_publish_retry_max_retries: 5 amqp_publish_retry_interval_start: 10 amqp_publish_retry_interval_step: 10 amqp_publish_retry_interval_max: 60 # We also need to create the dependency resolver file so pulsar knows how to # find and install dependencies for the tools we ask it to run. The simplest # method which covers 99% of the use cases is to use conda auto installs similar # to how Galaxy works. pulsar_dependency_resolvers: - name: conda args: - name: auto_init value: true ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano
Hi Luc, I'm sorry to hear that you're struggling to set up Galaxy to your liking. Let me start by pointing out that usegalaxy.org uses slurm with DRMAA, this is certainly going to be more performant and reliable than the CLI plugin. There is little maintenance necessary, so maybe that is why activity on slurm-drmaa is low (See also https://github.com/natefoo/slurm-drmaa). I would be curious to know how you came to the conclusion that there is some incompatibility between DRMAA and slurm Note that one of the setups we teach during the training submits via DRMAA to slurm. Then I'd like to point out that there are a huge variety of different ways in which you can configure Galaxy and the job submission. We teach the most common ones during the training week, with the aim that you understand how these things work together, as well as giving you a handle on how you can manage these different settings and services using a configuration management system. We cannot tailor a solution to your infrastructure during this week. About your problem specifically, I had asked this on gitter before:
Did you restart pulsar after rolling out the new config ?
to which you've answered that you re-ran the playbook, but that's not a sufficient answer. Every playbook is different, and we cannot know if this includes a restarter service for pulsar. Also please don't assume that everyone that could potentially help you knows ansible and the playbooks that are being taught intimately, and in what ways you have customized your playbook. It is much more helpful to write up the relevant settings you've changed and the logs that go with it. You've also been asked to provide logs of the restart, which as far as I can tell you haven't provided. You had mentioned on gitter that pulsar continues to use DRMAA to submit jobs, so you'll want to double check whether you've really restarted pulsar after the config changes, and look at the startup logs for pulsar, and find out how it is possible for pulsar to submit jobs via drmaa if it is not set up to do so. Best, Marius
Dear Marius, Many thank for your feedback. I join to this email: the playbook, the pulsarservers.yml file and the log of pulsar playbook. CLI plugin is for us the best solution since we have nothing to maintain. DRMAA is not actively developed for slurm, correct ? In the playbook, we use systemd which I think should restart pulsar but It might not be the case: TASK [galaxyproject.pulsar : systemd daemon-reload and enable/start service] **************************************************************************** ok: [HPC] RUNNING HANDLER [galaxyproject.pulsar : default restart pulsar handler] ********************************************************************************* skipping: [HPC] Currently, we never used DRMAA. The job were executed immediately on the cluster with CLI or DRMAA. We had this part in pulsarservers.yml, to activate CLI: managers: _default_: type: queued_cli job_plugin: slurm native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" min_polling_interval: 0.5 amqp_publish_retry: True amqp_publish_retry_max_retries: 5 amqp_publish_retry_interval_start: 10 amqp_publish_retry_interval_step: 10 amqp_publish_retry_interval_max: 60 Thanks for your help, Luc ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano ----- Mail original ----- De: "Marius van den Beek" <m.vandenbeek@gmail.com> À: "Luc Cornet" <luc.cornet@uliege.be> Cc: "HelpGalaxy" <galaxy-dev@lists.galaxyproject.org>, "Baurain Denis" <Denis.Baurain@uliege.be>, "Pierre Becker" <Pierre.Becker@sciensano.be>, "Colignon David" <David.Colignon@uliege.be> Envoyé: Mercredi 30 Juin 2021 16:02:04 Objet: [galaxy-dev] Re: Galaxy install problems Hi Luc, I'm sorry to hear that you're struggling to set up Galaxy to your liking. Let me start by pointing out that [ http://usegalaxy.org/ | usegalaxy.org ] uses slurm with DRMAA, this is certainly going to be more performant and reliable than the CLI plugin. There is little maintenance necessary, so maybe that is why activity on slurm-drmaa is low (See also [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] ). I would be curious to know how you came to the conclusion that there is some incompatibility between DRMAA and slurm Note that one of the setups we teach during the training submits via DRMAA to slurm. Then I'd like to point out that there are a huge variety of different ways in which you can configure Galaxy and the job submission. We teach the most common ones during the training week, with the aim that you understand how these things work together, as well as giving you a handle on how you can manage these different settings and services using a configuration management system. We cannot tailor a solution to your infrastructure during this week. About your problem specifically, I had asked this on gitter before:
Did you restart pulsar after rolling out the new config ?
to which you've answered that you re-ran the playbook, but that's not a sufficient answer. Every playbook is different, and we cannot know if this includes a restarter service for pulsar. Also please don't assume that everyone that could potentially help you knows ansible and the playbooks that are being taught intimately, and in what ways you have customized your playbook. It is much more helpful to write up the relevant settings you've changed and the logs that go with it. You've also been asked to provide logs of the restart, which as far as I can tell you haven't provided. You had mentioned on gitter that pulsar continues to use DRMAA to submit jobs, so you'll want to double check whether you've really restarted pulsar after the config changes, and look at the startup logs for pulsar, and find out how it is possible for pulsar to submit jobs via drmaa if it is not set up to do so. Best, Marius ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Thanks, but these are not the pulsar application logs. Can you provide these please ? On Wed, 30 Jun 2021 at 16:30, Luc Cornet <luc.cornet@uliege.be> wrote:
Dear Marius,
Many thank for your feedback.
I join to this email: the playbook, the pulsarservers.yml file and the log of pulsar playbook.
CLI plugin is for us the best solution since we have nothing to maintain. DRMAA is not actively developed for slurm, correct ?
In the playbook, we use systemd which I think should restart pulsar but It might not be the case:
TASK [galaxyproject.pulsar : systemd daemon-reload and enable/start service] **************************************************************************** ok: [HPC]
RUNNING HANDLER [galaxyproject.pulsar : default restart pulsar handler] ********************************************************************************* skipping: [HPC]
Currently, we never used DRMAA. The job were executed immediately on the cluster with CLI or DRMAA. We had this part in pulsarservers.yml, to activate CLI: managers: _default_: type: queued_cli job_plugin: slurm native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" min_polling_interval: 0.5 amqp_publish_retry: True amqp_publish_retry_max_retries: 5 amqp_publish_retry_interval_start: 10 amqp_publish_retry_interval_step: 10 amqp_publish_retry_interval_max: 60
Thanks for your help, Luc
------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano
----- Mail original ----- De: "Marius van den Beek" <m.vandenbeek@gmail.com> À: "Luc Cornet" <luc.cornet@uliege.be> Cc: "HelpGalaxy" <galaxy-dev@lists.galaxyproject.org>, "Baurain Denis" < Denis.Baurain@uliege.be>, "Pierre Becker" <Pierre.Becker@sciensano.be>, "Colignon David" <David.Colignon@uliege.be> Envoyé: Mercredi 30 Juin 2021 16:02:04 Objet: [galaxy-dev] Re: Galaxy install problems
Hi Luc,
I'm sorry to hear that you're struggling to set up Galaxy to your liking. Let me start by pointing out that [ http://usegalaxy.org/ | usegalaxy.org ] uses slurm with DRMAA, this is certainly going to be more performant and reliable than the CLI plugin. There is little maintenance necessary, so maybe that is why activity on slurm-drmaa is low (See also [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] ). I would be curious to know how you came to the conclusion that there is some incompatibility between DRMAA and slurm Note that one of the setups we teach during the training submits via DRMAA to slurm.
Then I'd like to point out that there are a huge variety of different ways in which you can configure Galaxy and the job submission. We teach the most common ones during the training week, with the aim that you understand how these things work together, as well as giving you a handle on how you can manage these different settings and services using a configuration management system. We cannot tailor a solution to your infrastructure during this week.
About your problem specifically, I had asked this on gitter before:
Did you restart pulsar after rolling out the new config ?
to which you've answered that you re-ran the playbook, but that's not a sufficient answer.
Every playbook is different, and we cannot know if this includes a restarter service for pulsar. Also please don't assume that everyone that could potentially help you knows ansible and the playbooks that are being taught intimately, and in what ways you have customized your playbook. It is much more helpful to write up the relevant settings you've changed and the logs that go with it.
You've also been asked to provide logs of the restart, which as far as I can tell you haven't provided. You had mentioned on gitter that pulsar continues to use DRMAA to submit jobs, so you'll want to double check whether you've really restarted pulsar after the config changes, and look at the startup logs for pulsar, and find out how it is possible for pulsar to submit jobs via drmaa if it is not set up to do so.
Best, Marius
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
And, no I would not recommend the use of the CLI plugin over drmaa. I'm happy to help you set it up, but this does not seem like the right choice. On Wed, 30 Jun 2021 at 16:32, Marius van den Beek <m.vandenbeek@gmail.com> wrote:
Thanks, but these are not the pulsar application logs. Can you provide these please ?
On Wed, 30 Jun 2021 at 16:30, Luc Cornet <luc.cornet@uliege.be> wrote:
Dear Marius,
Many thank for your feedback.
I join to this email: the playbook, the pulsarservers.yml file and the log of pulsar playbook.
CLI plugin is for us the best solution since we have nothing to maintain. DRMAA is not actively developed for slurm, correct ?
In the playbook, we use systemd which I think should restart pulsar but It might not be the case:
TASK [galaxyproject.pulsar : systemd daemon-reload and enable/start service] **************************************************************************** ok: [HPC]
RUNNING HANDLER [galaxyproject.pulsar : default restart pulsar handler] ********************************************************************************* skipping: [HPC]
Currently, we never used DRMAA. The job were executed immediately on the cluster with CLI or DRMAA. We had this part in pulsarservers.yml, to activate CLI: managers: _default_: type: queued_cli job_plugin: slurm native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" min_polling_interval: 0.5 amqp_publish_retry: True amqp_publish_retry_max_retries: 5 amqp_publish_retry_interval_start: 10 amqp_publish_retry_interval_step: 10 amqp_publish_retry_interval_max: 60
Thanks for your help, Luc
------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano
----- Mail original ----- De: "Marius van den Beek" <m.vandenbeek@gmail.com> À: "Luc Cornet" <luc.cornet@uliege.be> Cc: "HelpGalaxy" <galaxy-dev@lists.galaxyproject.org>, "Baurain Denis" < Denis.Baurain@uliege.be>, "Pierre Becker" <Pierre.Becker@sciensano.be>, "Colignon David" <David.Colignon@uliege.be> Envoyé: Mercredi 30 Juin 2021 16:02:04 Objet: [galaxy-dev] Re: Galaxy install problems
Hi Luc,
I'm sorry to hear that you're struggling to set up Galaxy to your liking. Let me start by pointing out that [ http://usegalaxy.org/ | usegalaxy.org ] uses slurm with DRMAA, this is certainly going to be more performant and reliable than the CLI plugin. There is little maintenance necessary, so maybe that is why activity on slurm-drmaa is low (See also [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] ). I would be curious to know how you came to the conclusion that there is some incompatibility between DRMAA and slurm Note that one of the setups we teach during the training submits via DRMAA to slurm.
Then I'd like to point out that there are a huge variety of different ways in which you can configure Galaxy and the job submission. We teach the most common ones during the training week, with the aim that you understand how these things work together, as well as giving you a handle on how you can manage these different settings and services using a configuration management system. We cannot tailor a solution to your infrastructure during this week.
About your problem specifically, I had asked this on gitter before:
Did you restart pulsar after rolling out the new config ?
to which you've answered that you re-ran the playbook, but that's not a sufficient answer.
Every playbook is different, and we cannot know if this includes a restarter service for pulsar. Also please don't assume that everyone that could potentially help you knows ansible and the playbooks that are being taught intimately, and in what ways you have customized your playbook. It is much more helpful to write up the relevant settings you've changed and the logs that go with it.
You've also been asked to provide logs of the restart, which as far as I can tell you haven't provided. You had mentioned on gitter that pulsar continues to use DRMAA to submit jobs, so you'll want to double check whether you've really restarted pulsar after the config changes, and look at the startup logs for pulsar, and find out how it is possible for pulsar to submit jobs via drmaa if it is not set up to do so.
Best, Marius
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
OK, Sure but can you please tell me how to get these pulsar application logs ? Thanks Luc ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano ----- Mail original ----- De: "Marius van den Beek" <m.vandenbeek@gmail.com> À: "Luc Cornet" <luc.cornet@uliege.be> Cc: "HelpGalaxy" <galaxy-dev@lists.galaxyproject.org>, "Baurain Denis" <Denis.Baurain@uliege.be>, "Pierre Becker" <Pierre.Becker@sciensano.be>, "Colignon David" <David.Colignon@uliege.be> Envoyé: Mercredi 30 Juin 2021 16:32:16 Objet: [galaxy-dev] Re: Galaxy install problems Thanks, but these are not the pulsar application logs. Can you provide these please ? On Wed, 30 Jun 2021 at 16:30, Luc Cornet < [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] > wrote: Dear Marius, Many thank for your feedback. I join to this email: the playbook, the pulsarservers.yml file and the log of pulsar playbook. CLI plugin is for us the best solution since we have nothing to maintain. DRMAA is not actively developed for slurm, correct ? In the playbook, we use systemd which I think should restart pulsar but It might not be the case: TASK [galaxyproject.pulsar : systemd daemon-reload and enable/start service] **************************************************************************** ok: [HPC] RUNNING HANDLER [galaxyproject.pulsar : default restart pulsar handler] ********************************************************************************* skipping: [HPC] Currently, we never used DRMAA. The job were executed immediately on the cluster with CLI or DRMAA. We had this part in pulsarservers.yml, to activate CLI: managers: _default_: type: queued_cli job_plugin: slurm native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" min_polling_interval: 0.5 amqp_publish_retry: True amqp_publish_retry_max_retries: 5 amqp_publish_retry_interval_start: 10 amqp_publish_retry_interval_step: 10 amqp_publish_retry_interval_max: 60 Thanks for your help, Luc ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano ----- Mail original ----- De: "Marius van den Beek" < [ mailto:m.vandenbeek@gmail.com | m.vandenbeek@gmail.com ] > À: "Luc Cornet" < [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] > Cc: "HelpGalaxy" < [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] >, "Baurain Denis" < [ mailto:Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] >, "Pierre Becker" < [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] >, "Colignon David" < [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] > Envoyé: Mercredi 30 Juin 2021 16:02:04 Objet: [galaxy-dev] Re: Galaxy install problems Hi Luc, I'm sorry to hear that you're struggling to set up Galaxy to your liking. Let me start by pointing out that [ [ http://usegalaxy.org/ | http://usegalaxy.org/ ] | [ http://usegalaxy.org/ | usegalaxy.org ] ] uses slurm with DRMAA, this is certainly going to be more performant and reliable than the CLI plugin. There is little maintenance necessary, so maybe that is why activity on slurm-drmaa is low (See also [ [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] | [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] ] ). I would be curious to know how you came to the conclusion that there is some incompatibility between DRMAA and slurm Note that one of the setups we teach during the training submits via DRMAA to slurm. Then I'd like to point out that there are a huge variety of different ways in which you can configure Galaxy and the job submission. We teach the most common ones during the training week, with the aim that you understand how these things work together, as well as giving you a handle on how you can manage these different settings and services using a configuration management system. We cannot tailor a solution to your infrastructure during this week. About your problem specifically, I had asked this on gitter before:
Did you restart pulsar after rolling out the new config ?
to which you've answered that you re-ran the playbook, but that's not a sufficient answer. Every playbook is different, and we cannot know if this includes a restarter service for pulsar. Also please don't assume that everyone that could potentially help you knows ansible and the playbooks that are being taught intimately, and in what ways you have customized your playbook. It is much more helpful to write up the relevant settings you've changed and the logs that go with it. You've also been asked to provide logs of the restart, which as far as I can tell you haven't provided. You had mentioned on gitter that pulsar continues to use DRMAA to submit jobs, so you'll want to double check whether you've really restarted pulsar after the config changes, and look at the startup logs for pulsar, and find out how it is possible for pulsar to submit jobs via drmaa if it is not set up to do so. Best, Marius ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s To search Galaxy mailing lists use the unified search at: [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ] ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
If you've set up pulsar to start as a systemd service you should get the logs with journalctl. I assume that's `journalctl -u pulsar.service`. For a quickstart in journalctl check out https://www.linode.com/docs/guides/how-to-use-journalctl/ On Wed, 30 Jun 2021 at 16:38, Luc Cornet <luc.cornet@uliege.be> wrote:
OK,
Sure but can you please tell me how to get these pulsar application logs ?
Thanks Luc
------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano
----- Mail original ----- De: "Marius van den Beek" <m.vandenbeek@gmail.com> À: "Luc Cornet" <luc.cornet@uliege.be> Cc: "HelpGalaxy" <galaxy-dev@lists.galaxyproject.org>, "Baurain Denis" < Denis.Baurain@uliege.be>, "Pierre Becker" <Pierre.Becker@sciensano.be>, "Colignon David" <David.Colignon@uliege.be> Envoyé: Mercredi 30 Juin 2021 16:32:16 Objet: [galaxy-dev] Re: Galaxy install problems
Thanks, but these are not the pulsar application logs. Can you provide these please ?
On Wed, 30 Jun 2021 at 16:30, Luc Cornet < [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] > wrote:
Dear Marius,
Many thank for your feedback.
I join to this email: the playbook, the pulsarservers.yml file and the log of pulsar playbook.
CLI plugin is for us the best solution since we have nothing to maintain. DRMAA is not actively developed for slurm, correct ?
In the playbook, we use systemd which I think should restart pulsar but It might not be the case:
TASK [galaxyproject.pulsar : systemd daemon-reload and enable/start service] ****************************************************************************
ok: [HPC]
RUNNING HANDLER [galaxyproject.pulsar : default restart pulsar handler] *********************************************************************************
skipping: [HPC]
Currently, we never used DRMAA. The job were executed immediately on the cluster with CLI or DRMAA. We had this part in pulsarservers.yml, to activate CLI: managers: _default_: type: queued_cli job_plugin: slurm native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" min_polling_interval: 0.5 amqp_publish_retry: True amqp_publish_retry_max_retries: 5 amqp_publish_retry_interval_start: 10 amqp_publish_retry_interval_step: 10 amqp_publish_retry_interval_max: 60
Thanks for your help, Luc
------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano
----- Mail original ----- De: "Marius van den Beek" < [ mailto:m.vandenbeek@gmail.com | m.vandenbeek@gmail.com ] > À: "Luc Cornet" < [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ]
Cc: "HelpGalaxy" < [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] >, "Baurain Denis" < [ mailto: Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] >, "Pierre Becker" < [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] >, "Colignon David" < [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] > Envoyé: Mercredi 30 Juin 2021 16:02:04 Objet: [galaxy-dev] Re: Galaxy install problems
Hi Luc,
I'm sorry to hear that you're struggling to set up Galaxy to your liking. Let me start by pointing out that [ [ http://usegalaxy.org/ | http://usegalaxy.org/ ] | [ http://usegalaxy.org/ | usegalaxy.org ] ] uses slurm with DRMAA, this is certainly going to be more performant and reliable than the CLI plugin. There is little maintenance necessary, so maybe that is why activity on slurm-drmaa is low (See also [ [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] | [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] ] ). I would be curious to know how you came to the conclusion that there is some incompatibility between DRMAA and slurm Note that one of the setups we teach during the training submits via DRMAA to slurm.
Then I'd like to point out that there are a huge variety of different ways in which you can configure Galaxy and the job submission. We teach the most common ones during the training week, with the aim that you understand how these things work together, as well as giving you a handle on how you can manage these different settings and services using a configuration management system. We cannot tailor a solution to your infrastructure during this week.
About your problem specifically, I had asked this on gitter before:
Did you restart pulsar after rolling out the new config ?
to which you've answered that you re-ran the playbook, but that's not a sufficient answer.
Every playbook is different, and we cannot know if this includes a restarter service for pulsar. Also please don't assume that everyone that could potentially help you knows ansible and the playbooks that are being taught intimately, and in what ways you have customized your playbook. It is much more helpful to write up the relevant settings you've changed and the logs that go with it.
You've also been asked to provide logs of the restart, which as far as I can tell you haven't provided. You had mentioned on gitter that pulsar continues to use DRMAA to submit jobs, so you'll want to double check whether you've really restarted pulsar after the config changes, and look at the startup logs for pulsar, and find out how it is possible for pulsar to submit jobs via drmaa if it is not set up to do so.
Best, Marius
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s
To search Galaxy mailing lists use the unified search at: [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ]
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Re, This is the log of pulsar. As pointed by Keith Superman, the pulsar don't restart since the step is skipped. How can I force pulsar to restart ? Thanks, Luc ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano ----- Mail original ----- De: "Marius van den Beek" <m.vandenbeek@gmail.com> À: "Luc Cornet" <luc.cornet@uliege.be> Cc: "HelpGalaxy" <galaxy-dev@lists.galaxyproject.org>, "Baurain Denis" <Denis.Baurain@uliege.be>, "Pierre Becker" <Pierre.Becker@sciensano.be>, "Colignon David" <David.Colignon@uliege.be> Envoyé: Mercredi 30 Juin 2021 16:44:50 Objet: Re: [galaxy-dev] Re: Galaxy install problems If you've set up pulsar to start as a systemd service you should get the logs with journalctl. I assume that's `journalctl -u pulsar.service`. For a quickstart in journalctl check out https://www.linode.com/docs/guides/how-to-use-journalctl/ On Wed, 30 Jun 2021 at 16:38, Luc Cornet <luc.cornet@uliege.be> wrote:
OK,
Sure but can you please tell me how to get these pulsar application logs ?
Thanks Luc
------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano
----- Mail original ----- De: "Marius van den Beek" <m.vandenbeek@gmail.com> À: "Luc Cornet" <luc.cornet@uliege.be> Cc: "HelpGalaxy" <galaxy-dev@lists.galaxyproject.org>, "Baurain Denis" < Denis.Baurain@uliege.be>, "Pierre Becker" <Pierre.Becker@sciensano.be>, "Colignon David" <David.Colignon@uliege.be> Envoyé: Mercredi 30 Juin 2021 16:32:16 Objet: [galaxy-dev] Re: Galaxy install problems
Thanks, but these are not the pulsar application logs. Can you provide these please ?
On Wed, 30 Jun 2021 at 16:30, Luc Cornet < [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] > wrote:
Dear Marius,
Many thank for your feedback.
I join to this email: the playbook, the pulsarservers.yml file and the log of pulsar playbook.
CLI plugin is for us the best solution since we have nothing to maintain. DRMAA is not actively developed for slurm, correct ?
In the playbook, we use systemd which I think should restart pulsar but It might not be the case:
TASK [galaxyproject.pulsar : systemd daemon-reload and enable/start service] ****************************************************************************
ok: [HPC]
RUNNING HANDLER [galaxyproject.pulsar : default restart pulsar handler] *********************************************************************************
skipping: [HPC]
Currently, we never used DRMAA. The job were executed immediately on the cluster with CLI or DRMAA. We had this part in pulsarservers.yml, to activate CLI: managers: _default_: type: queued_cli job_plugin: slurm native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" min_polling_interval: 0.5 amqp_publish_retry: True amqp_publish_retry_max_retries: 5 amqp_publish_retry_interval_start: 10 amqp_publish_retry_interval_step: 10 amqp_publish_retry_interval_max: 60
Thanks for your help, Luc
------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano
----- Mail original ----- De: "Marius van den Beek" < [ mailto:m.vandenbeek@gmail.com | m.vandenbeek@gmail.com ] > À: "Luc Cornet" < [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ]
Cc: "HelpGalaxy" < [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] >, "Baurain Denis" < [ mailto: Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] >, "Pierre Becker" < [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] >, "Colignon David" < [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] > Envoyé: Mercredi 30 Juin 2021 16:02:04 Objet: [galaxy-dev] Re: Galaxy install problems
Hi Luc,
I'm sorry to hear that you're struggling to set up Galaxy to your liking. Let me start by pointing out that [ [ http://usegalaxy.org/ | http://usegalaxy.org/ ] | [ http://usegalaxy.org/ | usegalaxy.org ] ] uses slurm with DRMAA, this is certainly going to be more performant and reliable than the CLI plugin. There is little maintenance necessary, so maybe that is why activity on slurm-drmaa is low (See also [ [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] | [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] ] ). I would be curious to know how you came to the conclusion that there is some incompatibility between DRMAA and slurm Note that one of the setups we teach during the training submits via DRMAA to slurm.
Then I'd like to point out that there are a huge variety of different ways in which you can configure Galaxy and the job submission. We teach the most common ones during the training week, with the aim that you understand how these things work together, as well as giving you a handle on how you can manage these different settings and services using a configuration management system. We cannot tailor a solution to your infrastructure during this week.
About your problem specifically, I had asked this on gitter before:
Did you restart pulsar after rolling out the new config ?
to which you've answered that you re-ran the playbook, but that's not a sufficient answer.
Every playbook is different, and we cannot know if this includes a restarter service for pulsar. Also please don't assume that everyone that could potentially help you knows ansible and the playbooks that are being taught intimately, and in what ways you have customized your playbook. It is much more helpful to write up the relevant settings you've changed and the logs that go with it.
You've also been asked to provide logs of the restart, which as far as I can tell you haven't provided. You had mentioned on gitter that pulsar continues to use DRMAA to submit jobs, so you'll want to double check whether you've really restarted pulsar after the config changes, and look at the startup logs for pulsar, and find out how it is possible for pulsar to submit jobs via drmaa if it is not set up to do so.
Best, Marius
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s
To search Galaxy mailing lists use the unified search at: [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ]
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
That should be `systemctl restart pulsar.service` Marius On Wed, 30 Jun 2021 at 17:05, Luc Cornet <luc.cornet@uliege.be> wrote:
Re,
This is the log of pulsar.
As pointed by Keith Superman, the pulsar don't restart since the step is skipped. How can I force pulsar to restart ?
Thanks, Luc
------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano
----- Mail original ----- De: "Marius van den Beek" <m.vandenbeek@gmail.com> À: "Luc Cornet" <luc.cornet@uliege.be> Cc: "HelpGalaxy" <galaxy-dev@lists.galaxyproject.org>, "Baurain Denis" < Denis.Baurain@uliege.be>, "Pierre Becker" <Pierre.Becker@sciensano.be>, "Colignon David" <David.Colignon@uliege.be> Envoyé: Mercredi 30 Juin 2021 16:44:50 Objet: Re: [galaxy-dev] Re: Galaxy install problems
If you've set up pulsar to start as a systemd service you should get the logs with journalctl. I assume that's `journalctl -u pulsar.service`. For a quickstart in journalctl check out https://www.linode.com/docs/guides/how-to-use-journalctl/
On Wed, 30 Jun 2021 at 16:38, Luc Cornet <luc.cornet@uliege.be> wrote:
OK,
Sure but can you please tell me how to get these pulsar application logs ?
Thanks Luc
------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano
----- Mail original ----- De: "Marius van den Beek" <m.vandenbeek@gmail.com> À: "Luc Cornet" <luc.cornet@uliege.be> Cc: "HelpGalaxy" <galaxy-dev@lists.galaxyproject.org>, "Baurain Denis" < Denis.Baurain@uliege.be>, "Pierre Becker" <Pierre.Becker@sciensano.be>, "Colignon David" <David.Colignon@uliege.be> Envoyé: Mercredi 30 Juin 2021 16:32:16 Objet: [galaxy-dev] Re: Galaxy install problems
Thanks, but these are not the pulsar application logs. Can you provide these please ?
On Wed, 30 Jun 2021 at 16:30, Luc Cornet < [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] > wrote:
Dear Marius,
Many thank for your feedback.
I join to this email: the playbook, the pulsarservers.yml file and the log of pulsar playbook.
CLI plugin is for us the best solution since we have nothing to maintain. DRMAA is not actively developed for slurm, correct ?
In the playbook, we use systemd which I think should restart pulsar but It might not be the case:
TASK [galaxyproject.pulsar : systemd daemon-reload and enable/start service]
****************************************************************************
ok: [HPC]
RUNNING HANDLER [galaxyproject.pulsar : default restart pulsar handler]
*********************************************************************************
skipping: [HPC]
Currently, we never used DRMAA. The job were executed immediately on the cluster with CLI or DRMAA. We had this part in pulsarservers.yml, to activate CLI: managers: _default_: type: queued_cli job_plugin: slurm native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" min_polling_interval: 0.5 amqp_publish_retry: True amqp_publish_retry_max_retries: 5 amqp_publish_retry_interval_start: 10 amqp_publish_retry_interval_step: 10 amqp_publish_retry_interval_max: 60
Thanks for your help, Luc
------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano
----- Mail original ----- De: "Marius van den Beek" < [ mailto:m.vandenbeek@gmail.com | m.vandenbeek@gmail.com ] > À: "Luc Cornet" < [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ]
Cc: "HelpGalaxy" < [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] >, "Baurain Denis" < [ mailto: Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] >, "Pierre Becker" < [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] >, "Colignon David" < [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] > Envoyé: Mercredi 30 Juin 2021 16:02:04 Objet: [galaxy-dev] Re: Galaxy install problems
Hi Luc,
I'm sorry to hear that you're struggling to set up Galaxy to your liking. Let me start by pointing out that [ [ http://usegalaxy.org/ | http://usegalaxy.org/ ] | [ http://usegalaxy.org/ | usegalaxy.org ] ] uses slurm with DRMAA, this is certainly going to be more performant and reliable than the CLI plugin. There is little maintenance necessary, so maybe that is why activity on slurm-drmaa is low (See also [ [ https://github.com/natefoo/slurm-drmaa
|
https://github.com/natefoo/slurm-drmaa ] | [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] ] ). I would be curious to know how you came to the conclusion that there is some incompatibility between DRMAA and slurm Note that one of the setups we teach during the training submits via DRMAA to slurm.
Then I'd like to point out that there are a huge variety of different ways in which you can configure Galaxy and the job submission. We teach the most common ones during the training week, with the aim that you understand how these things work together, as well as giving you a handle on how you can manage these different settings and services using a configuration management system. We cannot tailor a solution to your infrastructure during this week.
About your problem specifically, I had asked this on gitter before:
Did you restart pulsar after rolling out the new config ?
to which you've answered that you re-ran the playbook, but that's not a sufficient answer.
Every playbook is different, and we cannot know if this includes a restarter service for pulsar. Also please don't assume that everyone that could potentially help you knows ansible and the playbooks that are being taught intimately, and in what ways you have customized your playbook. It is much more helpful to write up the relevant settings you've changed and the logs that go with it.
You've also been asked to provide logs of the restart, which as far as I can tell you haven't provided. You had mentioned on gitter that pulsar continues to use DRMAA to submit jobs, so you'll want to double check whether you've really restarted pulsar after the config changes, and look at the startup logs for pulsar, and find out how it is possible for pulsar to submit jobs via drmaa if it is not set up to do so.
Best, Marius
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s
To search Galaxy mailing lists use the unified search at: [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ]
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Re, We shot down pulsar, run the playbook again, restart pulsar: We I try to launch an analysis, I have this error for the job (still CLI): Failed to find job_plugin of type slurm, available types include ['LSF', 'Slurm', 'SlurmTorque', 'Torque'] Luc, ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano ----- Mail original ----- De: "Marius van den Beek" <m.vandenbeek@gmail.com> À: "Luc Cornet" <luc.cornet@uliege.be> Cc: "HelpGalaxy" <galaxy-dev@lists.galaxyproject.org>, "Baurain Denis" <Denis.Baurain@uliege.be>, "Pierre Becker" <Pierre.Becker@sciensano.be>, "Colignon David" <David.Colignon@uliege.be> Envoyé: Mercredi 30 Juin 2021 17:24:13 Objet: [galaxy-dev] Re: Galaxy install problems That should be `systemctl restart pulsar.service` Marius On Wed, 30 Jun 2021 at 17:05, Luc Cornet < [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] > wrote: Re, This is the log of pulsar. As pointed by Keith Superman, the pulsar don't restart since the step is skipped. How can I force pulsar to restart ? Thanks, Luc ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano ----- Mail original ----- De: "Marius van den Beek" < [ mailto:m.vandenbeek@gmail.com | m.vandenbeek@gmail.com ] > À: "Luc Cornet" < [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] > Cc: "HelpGalaxy" < [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] >, "Baurain Denis" < [ mailto:Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] >, "Pierre Becker" < [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] >, "Colignon David" < [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] > Envoyé: Mercredi 30 Juin 2021 16:44:50 Objet: Re: [galaxy-dev] Re: Galaxy install problems If you've set up pulsar to start as a systemd service you should get the logs with journalctl. I assume that's `journalctl -u pulsar.service`. For a quickstart in journalctl check out [ https://www.linode.com/docs/guides/how-to-use-journalctl/ | https://www.linode.com/docs/guides/how-to-use-journalctl/ ] On Wed, 30 Jun 2021 at 16:38, Luc Cornet < [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] > wrote:
OK,
Sure but can you please tell me how to get these pulsar application logs ?
Thanks Luc
------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano
----- Mail original ----- De: "Marius van den Beek" < [ mailto:m.vandenbeek@gmail.com | m.vandenbeek@gmail.com ] > À: "Luc Cornet" < [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] > Cc: "HelpGalaxy" < [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] >, "Baurain Denis" < [ mailto:Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] >, "Pierre Becker" < [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] >, "Colignon David" < [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] > Envoyé: Mercredi 30 Juin 2021 16:32:16 Objet: [galaxy-dev] Re: Galaxy install problems
Thanks, but these are not the pulsar application logs. Can you provide these please ?
On Wed, 30 Jun 2021 at 16:30, Luc Cornet < [ mailto: [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] | [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] ] > wrote:
Dear Marius,
Many thank for your feedback.
I join to this email: the playbook, the pulsarservers.yml file and the log of pulsar playbook.
CLI plugin is for us the best solution since we have nothing to maintain. DRMAA is not actively developed for slurm, correct ?
In the playbook, we use systemd which I think should restart pulsar but It might not be the case:
TASK [galaxyproject.pulsar : systemd daemon-reload and enable/start service] ****************************************************************************
ok: [HPC]
RUNNING HANDLER [galaxyproject.pulsar : default restart pulsar handler] *********************************************************************************
skipping: [HPC]
Currently, we never used DRMAA. The job were executed immediately on the cluster with CLI or DRMAA. We had this part in pulsarservers.yml, to activate CLI: managers: _default_: type: queued_cli job_plugin: slurm native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" min_polling_interval: 0.5 amqp_publish_retry: True amqp_publish_retry_max_retries: 5 amqp_publish_retry_interval_start: 10 amqp_publish_retry_interval_step: 10 amqp_publish_retry_interval_max: 60
Thanks for your help, Luc
------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano
----- Mail original ----- De: "Marius van den Beek" < [ mailto: [ mailto:m.vandenbeek@gmail.com | m.vandenbeek@gmail.com ] | [ mailto:m.vandenbeek@gmail.com | m.vandenbeek@gmail.com ] ] > À: "Luc Cornet" < [ mailto: [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] | [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] ]
Cc: "HelpGalaxy" < [ mailto: [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] | [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] ] >, "Baurain Denis" < [ mailto: [ mailto:Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] | [ mailto:Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] ] >, "Pierre Becker" < [ mailto: [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] | [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] ] >, "Colignon David" < [ mailto: [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] | [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] ] > Envoyé: Mercredi 30 Juin 2021 16:02:04 Objet: [galaxy-dev] Re: Galaxy install problems
Hi Luc,
I'm sorry to hear that you're struggling to set up Galaxy to your liking. Let me start by pointing out that [ [ [ http://usegalaxy.org/ | http://usegalaxy.org/ ] | [ http://usegalaxy.org/ | http://usegalaxy.org/ ] ] | [ [ http://usegalaxy.org/ | http://usegalaxy.org/ ] | [ http://usegalaxy.org/ | usegalaxy.org ] ] ] uses slurm with DRMAA, this is certainly going to be more performant and reliable than the CLI plugin. There is little maintenance necessary, so maybe that is why activity on slurm-drmaa is low (See also [ [ [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] | [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] ] | [ [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] | [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] ] ] ). I would be curious to know how you came to the conclusion that there is some incompatibility between DRMAA and slurm Note that one of the setups we teach during the training submits via DRMAA to slurm.
Then I'd like to point out that there are a huge variety of different ways in which you can configure Galaxy and the job submission. We teach the most common ones during the training week, with the aim that you understand how these things work together, as well as giving you a handle on how you can manage these different settings and services using a configuration management system. We cannot tailor a solution to your infrastructure during this week.
About your problem specifically, I had asked this on gitter before:
Did you restart pulsar after rolling out the new config ?
to which you've answered that you re-ran the playbook, but that's not a sufficient answer.
Every playbook is different, and we cannot know if this includes a restarter service for pulsar. Also please don't assume that everyone that could potentially help you knows ansible and the playbooks that are being taught intimately, and in what ways you have customized your playbook. It is much more helpful to write up the relevant settings you've changed and the logs that go with it.
You've also been asked to provide logs of the restart, which as far as I can tell you haven't provided. You had mentioned on gitter that pulsar continues to use DRMAA to submit jobs, so you'll want to double check whether you've really restarted pulsar after the config changes, and look at the startup logs for pulsar, and find out how it is possible for pulsar to submit jobs via drmaa if it is not set up to do so.
Best, Marius
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s
To search Galaxy mailing lists use the unified search at: [ [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ] | [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ] ]
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s
To search Galaxy mailing lists use the unified search at: [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ]
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Hi Luc, you need to capitalize the first letter of slurm in your config: pulsar_yaml_config: conda_auto_init: True conda_auto_install: True staging_directory: "{{ pulsar_staging_dir }}" persistence_directory: "{{ pulsar_persistence_dir }}" tool_dependency_dir: "{{ pulsar_dependencies_dir }}" # The following are the settings for the pulsar server to contact the message queue with related timeouts etc. message_queue_url: "pyamqp://galaxy_au:{{ rabbitmq_password_galaxy_au }}@{{ galaxy_server_url }}:5671//pulsar/galaxy_au?ssl=1" managers: _default_: type: queued_cli job_plugin: Slurm native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" min_polling_interval: 0.5 amqp_publish_retry: True amqp_publish_retry_max_retries: 5 amqp_publish_retry_interval_start: 10 amqp_publish_retry_interval_step: 10 amqp_publish_retry_interval_max: 60 Best, Marius On Wed, 30 Jun 2021 at 18:35, Luc Cornet <luc.cornet@uliege.be> wrote:
Re,
We shot down pulsar, run the playbook again, restart pulsar: We I try to launch an analysis, I have this error for the job (still CLI):
Failed to find job_plugin of type slurm, available types include ['LSF', 'Slurm', 'SlurmTorque', 'Torque']
Luc,
------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano
----- Mail original ----- De: "Marius van den Beek" <m.vandenbeek@gmail.com> À: "Luc Cornet" <luc.cornet@uliege.be> Cc: "HelpGalaxy" <galaxy-dev@lists.galaxyproject.org>, "Baurain Denis" < Denis.Baurain@uliege.be>, "Pierre Becker" <Pierre.Becker@sciensano.be>, "Colignon David" <David.Colignon@uliege.be> Envoyé: Mercredi 30 Juin 2021 17:24:13 Objet: [galaxy-dev] Re: Galaxy install problems
That should be `systemctl restart pulsar.service`
Marius
On Wed, 30 Jun 2021 at 17:05, Luc Cornet < [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] > wrote:
Re,
This is the log of pulsar.
As pointed by Keith Superman, the pulsar don't restart since the step is skipped. How can I force pulsar to restart ?
Thanks, Luc
------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano
----- Mail original ----- De: "Marius van den Beek" < [ mailto:m.vandenbeek@gmail.com | m.vandenbeek@gmail.com ] > À: "Luc Cornet" < [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ]
Cc: "HelpGalaxy" < [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] >, "Baurain Denis" < [ mailto: Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] >, "Pierre Becker" < [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] >, "Colignon David" < [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] > Envoyé: Mercredi 30 Juin 2021 16:44:50 Objet: Re: [galaxy-dev] Re: Galaxy install problems
If you've set up pulsar to start as a systemd service you should get the logs with journalctl. I assume that's `journalctl -u pulsar.service`. For a quickstart in journalctl check out [ https://www.linode.com/docs/guides/how-to-use-journalctl/ | https://www.linode.com/docs/guides/how-to-use-journalctl/ ]
On Wed, 30 Jun 2021 at 16:38, Luc Cornet < [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] > wrote:
OK,
Sure but can you please tell me how to get these pulsar application logs ?
Thanks Luc
------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano
----- Mail original ----- De: "Marius van den Beek" < [ mailto:m.vandenbeek@gmail.com | m.vandenbeek@gmail.com ] > À: "Luc Cornet" < [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] > Cc: "HelpGalaxy" < [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] >, "Baurain Denis" < [ mailto:Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] >, "Pierre Becker" < [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] >, "Colignon David" < [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] > Envoyé: Mercredi 30 Juin 2021 16:32:16 Objet: [galaxy-dev] Re: Galaxy install problems
Thanks, but these are not the pulsar application logs. Can you provide these please ?
On Wed, 30 Jun 2021 at 16:30, Luc Cornet < [ mailto: [ mailto: luc.cornet@uliege.be | luc.cornet@uliege.be ] | [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] ] > wrote:
Dear Marius,
Many thank for your feedback.
I join to this email: the playbook, the pulsarservers.yml file and the log of pulsar playbook.
CLI plugin is for us the best solution since we have nothing to maintain. DRMAA is not actively developed for slurm, correct ?
In the playbook, we use systemd which I think should restart pulsar but It might not be the case:
TASK [galaxyproject.pulsar : systemd daemon-reload and enable/start service]
****************************************************************************
ok: [HPC]
RUNNING HANDLER [galaxyproject.pulsar : default restart pulsar handler]
*********************************************************************************
skipping: [HPC]
Currently, we never used DRMAA. The job were executed immediately on the cluster with CLI or DRMAA. We had this part in pulsarservers.yml, to activate CLI: managers: _default_: type: queued_cli job_plugin: slurm native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" min_polling_interval: 0.5 amqp_publish_retry: True amqp_publish_retry_max_retries: 5 amqp_publish_retry_interval_start: 10 amqp_publish_retry_interval_step: 10 amqp_publish_retry_interval_max: 60
Thanks for your help, Luc
------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano
----- Mail original ----- De: "Marius van den Beek" < [ mailto: [ mailto:m.vandenbeek@gmail.com |
[ mailto:m.vandenbeek@gmail.com | m.vandenbeek@gmail.com ] ] > À: "Luc Cornet" < [ mailto: [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] | [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] ]
Cc: "HelpGalaxy" < [ mailto: [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] | [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] ] >, "Baurain Denis" < [ mailto: [ mailto:Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] | [ mailto: Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] ] >, "Pierre Becker" < [ mailto: [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] | [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] ] >, "Colignon David" < [ mailto: [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] | [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] ] > Envoyé: Mercredi 30 Juin 2021 16:02:04 Objet: [galaxy-dev] Re: Galaxy install problems
Hi Luc,
I'm sorry to hear that you're struggling to set up Galaxy to your
Let me start by pointing out that [ [ [ http://usegalaxy.org/ | http://usegalaxy.org/ ] | [ http://usegalaxy.org/ | http://usegalaxy.org/ ] ] | [ [ http://usegalaxy.org/ | http://usegalaxy.org/ ] | [ http://usegalaxy.org/ | usegalaxy.org ] ] ] uses slurm with DRMAA, this is certainly going to be more performant and reliable than the CLI plugin. There is little maintenance necessary, so maybe that is why activity on slurm-drmaa is low (See also [ [ [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] | [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] ] | [ [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] | [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] ] ] ). I would be curious to know how you came to the conclusion that there is some incompatibility between DRMAA and slurm Note that one of the setups we teach during the training submits via DRMAA to slurm.
Then I'd like to point out that there are a huge variety of different ways in which you can configure Galaxy and the job submission. We teach the most common ones during the training week, with the aim
m.vandenbeek@gmail.com ] | liking. that
you understand how these things work together, as well as giving you a handle on how you can manage these different settings and services using a configuration management system. We cannot tailor a solution to your infrastructure during this week.
About your problem specifically, I had asked this on gitter before:
Did you restart pulsar after rolling out the new config ?
to which you've answered that you re-ran the playbook, but that's not a sufficient answer.
Every playbook is different, and we cannot know if this includes a restarter service for pulsar. Also please don't assume that everyone that could potentially help you knows ansible and the playbooks that are being taught intimately, and in what ways you have customized your playbook. It is much more helpful to write up the relevant settings you've changed and the logs that go with it.
You've also been asked to provide logs of the restart, which as far as I can tell you haven't provided. You had mentioned on gitter that pulsar continues to use DRMAA to submit jobs, so you'll want to double check whether you've really restarted pulsar after the config changes, and look at the startup logs for pulsar, and find out how it is possible for pulsar to submit jobs via drmaa if it is not set up to do so.
Best, Marius
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s
To search Galaxy mailing lists use the unified search at: [ [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ] | [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ] ]
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s
To search Galaxy mailing lists use the unified search at: [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ]
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Dear Marius, the slurm typo is now fixed and the job crash without any error reported. When I look at the pulsar log(attached), I see this: Jun 30 19:20:24 nic5-login1 pulsar[210900]: 2021-06-30 19:20:24,122 WARNI [pulsar.managers.util.cli.job.slurm][[manager=_default_]-[action=preprocess]-[job=11]] Unrecognized long argument passed to Slurm CLI plugin: directory_mode Do you have an idea ? Thanks Luc ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano ----- Mail original ----- De: "Marius van den Beek" <m.vandenbeek@gmail.com> À: "Luc Cornet" <luc.cornet@uliege.be> Cc: "HelpGalaxy" <galaxy-dev@lists.galaxyproject.org>, "Baurain Denis" <Denis.Baurain@uliege.be>, "Pierre Becker" <Pierre.Becker@sciensano.be>, "Colignon David" <David.Colignon@uliege.be> Envoyé: Mercredi 30 Juin 2021 18:38:24 Objet: [galaxy-dev] Re: Galaxy install problems Hi Luc, you need to capitalize the first letter of slurm in your config: pulsar_yaml_config: conda_auto_init: True conda_auto_install: True staging_directory: "{ { pulsar_staging_dir }}" persistence_directory: "{ { pulsar_persistence_dir }}" tool_dependency_dir: "{ { pulsar_dependencies_dir }}" # The following are the settings for the pulsar server to contact the message queue with related timeouts etc. message_queue_url: "pyamqp://galaxy_au:{ { rabbitmq_password_galaxy_au }}@{ { galaxy_server_url }}:5671//pulsar/galaxy_au?ssl=1" managers: _default_: type: queued_cli job_plugin: Slurm native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" min_polling_interval: 0.5 amqp_publish_retry: True amqp_publish_retry_max_retries: 5 amqp_publish_retry_interval_start: 10 amqp_publish_retry_interval_step: 10 amqp_publish_retry_interval_max: 60 Best, Marius On Wed, 30 Jun 2021 at 18:35, Luc Cornet < [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] > wrote: Re, We shot down pulsar, run the playbook again, restart pulsar: We I try to launch an analysis, I have this error for the job (still CLI): Failed to find job_plugin of type slurm, available types include ['LSF', 'Slurm', 'SlurmTorque', 'Torque'] Luc, ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano ----- Mail original ----- De: "Marius van den Beek" < [ mailto:m.vandenbeek@gmail.com | m.vandenbeek@gmail.com ] > À: "Luc Cornet" < [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] > Cc: "HelpGalaxy" < [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] >, "Baurain Denis" < [ mailto:Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] >, "Pierre Becker" < [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] >, "Colignon David" < [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] > Envoyé: Mercredi 30 Juin 2021 17:24:13 Objet: [galaxy-dev] Re: Galaxy install problems That should be `systemctl restart pulsar.service` Marius On Wed, 30 Jun 2021 at 17:05, Luc Cornet < [ mailto: [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] | [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] ] > wrote: Re, This is the log of pulsar. As pointed by Keith Superman, the pulsar don't restart since the step is skipped. How can I force pulsar to restart ? Thanks, Luc ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano ----- Mail original ----- De: "Marius van den Beek" < [ mailto: [ mailto:m.vandenbeek@gmail.com | m.vandenbeek@gmail.com ] | [ mailto:m.vandenbeek@gmail.com | m.vandenbeek@gmail.com ] ] > À: "Luc Cornet" < [ mailto: [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] | [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] ] > Cc: "HelpGalaxy" < [ mailto: [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] | [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] ] >, "Baurain Denis" < [ mailto: [ mailto:Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] | [ mailto:Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] ] >, "Pierre Becker" < [ mailto: [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] | [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] ] >, "Colignon David" < [ mailto: [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] | [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] ] > Envoyé: Mercredi 30 Juin 2021 16:44:50 Objet: Re: [galaxy-dev] Re: Galaxy install problems If you've set up pulsar to start as a systemd service you should get the logs with journalctl. I assume that's `journalctl -u pulsar.service`. For a quickstart in journalctl check out [ [ https://www.linode.com/docs/guides/how-to-use-journalctl/ | https://www.linode.com/docs/guides/how-to-use-journalctl/ ] | [ https://www.linode.com/docs/guides/how-to-use-journalctl/ | https://www.linode.com/docs/guides/how-to-use-journalctl/ ] ] On Wed, 30 Jun 2021 at 16:38, Luc Cornet < [ mailto: [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] | [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] ] > wrote:
OK,
Sure but can you please tell me how to get these pulsar application logs ?
Thanks Luc
------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano
----- Mail original ----- De: "Marius van den Beek" < [ mailto: [ mailto:m.vandenbeek@gmail.com | m.vandenbeek@gmail.com ] | [ mailto:m.vandenbeek@gmail.com | m.vandenbeek@gmail.com ] ] > À: "Luc Cornet" < [ mailto: [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] | [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] ] > Cc: "HelpGalaxy" < [ mailto: [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] | [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] ] >, "Baurain Denis" < [ mailto: [ mailto:Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] | [ mailto:Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] ] >, "Pierre Becker" < [ mailto: [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] | [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] ] >, "Colignon David" < [ mailto: [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] | [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] ] > Envoyé: Mercredi 30 Juin 2021 16:32:16 Objet: [galaxy-dev] Re: Galaxy install problems
Thanks, but these are not the pulsar application logs. Can you provide these please ?
On Wed, 30 Jun 2021 at 16:30, Luc Cornet < [ mailto: [ mailto: [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] | [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] ] | [ mailto: [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] | [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] ] ] > wrote:
Dear Marius,
Many thank for your feedback.
I join to this email: the playbook, the pulsarservers.yml file and the log of pulsar playbook.
CLI plugin is for us the best solution since we have nothing to maintain. DRMAA is not actively developed for slurm, correct ?
In the playbook, we use systemd which I think should restart pulsar but It might not be the case:
TASK [galaxyproject.pulsar : systemd daemon-reload and enable/start service] ****************************************************************************
ok: [HPC]
RUNNING HANDLER [galaxyproject.pulsar : default restart pulsar handler] *********************************************************************************
skipping: [HPC]
Currently, we never used DRMAA. The job were executed immediately on the cluster with CLI or DRMAA. We had this part in pulsarservers.yml, to activate CLI: managers: _default_: type: queued_cli job_plugin: slurm native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" min_polling_interval: 0.5 amqp_publish_retry: True amqp_publish_retry_max_retries: 5 amqp_publish_retry_interval_start: 10 amqp_publish_retry_interval_step: 10 amqp_publish_retry_interval_max: 60
Thanks for your help, Luc
------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano
----- Mail original ----- De: "Marius van den Beek" < [ mailto: [ mailto: [ mailto:m.vandenbeek@gmail.com | m.vandenbeek@gmail.com ] | [ mailto:m.vandenbeek@gmail.com | m.vandenbeek@gmail.com ] ] | [ mailto: [ mailto:m.vandenbeek@gmail.com | m.vandenbeek@gmail.com ] | [ mailto:m.vandenbeek@gmail.com | m.vandenbeek@gmail.com ] ] ] > À: "Luc Cornet" < [ mailto: [ mailto: [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] | [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] ] | [ mailto: [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] | [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] ] ]
Cc: "HelpGalaxy" < [ mailto: [ mailto: [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] | [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] ] | [ mailto: [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] | [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] ] ] >, "Baurain Denis" < [ mailto: [ mailto: [ mailto:Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] | [ mailto:Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] ] | [ mailto: [ mailto:Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] | [ mailto:Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] ] ] >, "Pierre Becker" < [ mailto: [ mailto: [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] | [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] ] | [ mailto: [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] | [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] ] ] >, "Colignon David" < [ mailto: [ mailto: [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] | [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] ] | [ mailto: [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] | [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] ] ] > Envoyé: Mercredi 30 Juin 2021 16:02:04 Objet: [galaxy-dev] Re: Galaxy install problems
Hi Luc,
I'm sorry to hear that you're struggling to set up Galaxy to your liking. Let me start by pointing out that [ [ [ [ http://usegalaxy.org/ | http://usegalaxy.org/ ] | [ http://usegalaxy.org/ | http://usegalaxy.org/ ] ] | [ [ http://usegalaxy.org/ | http://usegalaxy.org/ ] | [ http://usegalaxy.org/ | http://usegalaxy.org/ ] ] ] | [ [ [ http://usegalaxy.org/ | http://usegalaxy.org/ ] | [ http://usegalaxy.org/ | http://usegalaxy.org/ ] ] | [ [ http://usegalaxy.org/ | http://usegalaxy.org/ ] | [ http://usegalaxy.org/ | usegalaxy.org ] ] ] ] uses slurm with DRMAA, this is certainly going to be more performant and reliable than the CLI plugin. There is little maintenance necessary, so maybe that is why activity on slurm-drmaa is low (See also [ [ [ [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] | [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] ] | [ [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] | [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] ] ] | [ [ [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] | [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] ] | [ [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] | [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] ] ] ] ). I would be curious to know how you came to the conclusion that there is some incompatibility between DRMAA and slurm Note that one of the setups we teach during the training submits via DRMAA to slurm.
Then I'd like to point out that there are a huge variety of different ways in which you can configure Galaxy and the job submission. We teach the most common ones during the training week, with the aim that you understand how these things work together, as well as giving you a handle on how you can manage these different settings and services using a configuration management system. We cannot tailor a solution to your infrastructure during this week.
About your problem specifically, I had asked this on gitter before:
Did you restart pulsar after rolling out the new config ?
to which you've answered that you re-ran the playbook, but that's not a sufficient answer.
Every playbook is different, and we cannot know if this includes a restarter service for pulsar. Also please don't assume that everyone that could potentially help you knows ansible and the playbooks that are being taught intimately, and in what ways you have customized your playbook. It is much more helpful to write up the relevant settings you've changed and the logs that go with it.
You've also been asked to provide logs of the restart, which as far as I can tell you haven't provided. You had mentioned on gitter that pulsar continues to use DRMAA to submit jobs, so you'll want to double check whether you've really restarted pulsar after the config changes, and look at the startup logs for pulsar, and find out how it is possible for pulsar to submit jobs via drmaa if it is not set up to do so.
Best, Marius
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s
To search Galaxy mailing lists use the unified search at: [ [ [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ] | [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ] ] | [ [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ] | [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ] ] ]
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s
To search Galaxy mailing lists use the unified search at: [ [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ] | [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ] ]
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s To search Galaxy mailing lists use the unified search at: [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ] ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Dear Marius, I am contacting you back since I am not able to run job with pulsar. I still face the error below: Jun 30 19:20:24 nic5-login1 pulsar[210900]: 2021-06-30 19:20:24,122 WARNI [pulsar.managers.util.cli.job.slurm][[manager=_default_]-[action=preprocess]-[job=11]] Unrecognized long argument passed to Slurm CLI plugin: directory_mode Do you have an idea of the problem ? Thanks, Luc ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano ----- Mail original ----- De: "Luc Cornet" <luc.cornet@uliege.be> À: "Marius van den Beek" <m.vandenbeek@gmail.com> Cc: "HelpGalaxy" <galaxy-dev@lists.galaxyproject.org>, "Baurain Denis" <Denis.Baurain@uliege.be>, "Pierre Becker" <Pierre.Becker@sciensano.be>, "Colignon David" <David.Colignon@uliege.be> Envoyé: Mercredi 30 Juin 2021 18:35:34 Objet: Re: [galaxy-dev] Re: Galaxy install problems Re, We shot down pulsar, run the playbook again, restart pulsar: We I try to launch an analysis, I have this error for the job (still CLI): Failed to find job_plugin of type slurm, available types include ['LSF', 'Slurm', 'SlurmTorque', 'Torque'] Luc, ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano ----- Mail original ----- De: "Marius van den Beek" <m.vandenbeek@gmail.com> À: "Luc Cornet" <luc.cornet@uliege.be> Cc: "HelpGalaxy" <galaxy-dev@lists.galaxyproject.org>, "Baurain Denis" <Denis.Baurain@uliege.be>, "Pierre Becker" <Pierre.Becker@sciensano.be>, "Colignon David" <David.Colignon@uliege.be> Envoyé: Mercredi 30 Juin 2021 17:24:13 Objet: [galaxy-dev] Re: Galaxy install problems That should be `systemctl restart pulsar.service` Marius On Wed, 30 Jun 2021 at 17:05, Luc Cornet < [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] > wrote: Re, This is the log of pulsar. As pointed by Keith Superman, the pulsar don't restart since the step is skipped. How can I force pulsar to restart ? Thanks, Luc ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano ----- Mail original ----- De: "Marius van den Beek" < [ mailto:m.vandenbeek@gmail.com | m.vandenbeek@gmail.com ] > À: "Luc Cornet" < [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] > Cc: "HelpGalaxy" < [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] >, "Baurain Denis" < [ mailto:Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] >, "Pierre Becker" < [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] >, "Colignon David" < [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] > Envoyé: Mercredi 30 Juin 2021 16:44:50 Objet: Re: [galaxy-dev] Re: Galaxy install problems If you've set up pulsar to start as a systemd service you should get the logs with journalctl. I assume that's `journalctl -u pulsar.service`. For a quickstart in journalctl check out [ https://www.linode.com/docs/guides/how-to-use-journalctl/ | https://www.linode.com/docs/guides/how-to-use-journalctl/ ] On Wed, 30 Jun 2021 at 16:38, Luc Cornet < [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] > wrote:
OK,
Sure but can you please tell me how to get these pulsar application logs ?
Thanks Luc
------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano
----- Mail original ----- De: "Marius van den Beek" < [ mailto:m.vandenbeek@gmail.com | m.vandenbeek@gmail.com ] > À: "Luc Cornet" < [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] > Cc: "HelpGalaxy" < [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] >, "Baurain Denis" < [ mailto:Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] >, "Pierre Becker" < [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] >, "Colignon David" < [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] > Envoyé: Mercredi 30 Juin 2021 16:32:16 Objet: [galaxy-dev] Re: Galaxy install problems
Thanks, but these are not the pulsar application logs. Can you provide these please ?
On Wed, 30 Jun 2021 at 16:30, Luc Cornet < [ mailto: [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] | [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] ] > wrote:
Dear Marius,
Many thank for your feedback.
I join to this email: the playbook, the pulsarservers.yml file and the log of pulsar playbook.
CLI plugin is for us the best solution since we have nothing to maintain. DRMAA is not actively developed for slurm, correct ?
In the playbook, we use systemd which I think should restart pulsar but It might not be the case:
TASK [galaxyproject.pulsar : systemd daemon-reload and enable/start service] ****************************************************************************
ok: [HPC]
RUNNING HANDLER [galaxyproject.pulsar : default restart pulsar handler] *********************************************************************************
skipping: [HPC]
Currently, we never used DRMAA. The job were executed immediately on the cluster with CLI or DRMAA. We had this part in pulsarservers.yml, to activate CLI: managers: _default_: type: queued_cli job_plugin: slurm native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" min_polling_interval: 0.5 amqp_publish_retry: True amqp_publish_retry_max_retries: 5 amqp_publish_retry_interval_start: 10 amqp_publish_retry_interval_step: 10 amqp_publish_retry_interval_max: 60
Thanks for your help, Luc
------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano
----- Mail original ----- De: "Marius van den Beek" < [ mailto: [ mailto:m.vandenbeek@gmail.com | m.vandenbeek@gmail.com ] | [ mailto:m.vandenbeek@gmail.com | m.vandenbeek@gmail.com ] ] > À: "Luc Cornet" < [ mailto: [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] | [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] ]
Cc: "HelpGalaxy" < [ mailto: [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] | [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] ] >, "Baurain Denis" < [ mailto: [ mailto:Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] | [ mailto:Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] ] >, "Pierre Becker" < [ mailto: [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] | [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] ] >, "Colignon David" < [ mailto: [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] | [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] ] > Envoyé: Mercredi 30 Juin 2021 16:02:04 Objet: [galaxy-dev] Re: Galaxy install problems
Hi Luc,
I'm sorry to hear that you're struggling to set up Galaxy to your liking. Let me start by pointing out that [ [ [ http://usegalaxy.org/ | http://usegalaxy.org/ ] | [ http://usegalaxy.org/ | http://usegalaxy.org/ ] ] | [ [ http://usegalaxy.org/ | http://usegalaxy.org/ ] | [ http://usegalaxy.org/ | usegalaxy.org ] ] ] uses slurm with DRMAA, this is certainly going to be more performant and reliable than the CLI plugin. There is little maintenance necessary, so maybe that is why activity on slurm-drmaa is low (See also [ [ [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] | [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] ] | [ [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] | [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] ] ] ). I would be curious to know how you came to the conclusion that there is some incompatibility between DRMAA and slurm Note that one of the setups we teach during the training submits via DRMAA to slurm.
Then I'd like to point out that there are a huge variety of different ways in which you can configure Galaxy and the job submission. We teach the most common ones during the training week, with the aim that you understand how these things work together, as well as giving you a handle on how you can manage these different settings and services using a configuration management system. We cannot tailor a solution to your infrastructure during this week.
About your problem specifically, I had asked this on gitter before:
Did you restart pulsar after rolling out the new config ?
to which you've answered that you re-ran the playbook, but that's not a sufficient answer.
Every playbook is different, and we cannot know if this includes a restarter service for pulsar. Also please don't assume that everyone that could potentially help you knows ansible and the playbooks that are being taught intimately, and in what ways you have customized your playbook. It is much more helpful to write up the relevant settings you've changed and the logs that go with it.
You've also been asked to provide logs of the restart, which as far as I can tell you haven't provided. You had mentioned on gitter that pulsar continues to use DRMAA to submit jobs, so you'll want to double check whether you've really restarted pulsar after the config changes, and look at the startup logs for pulsar, and find out how it is possible for pulsar to submit jobs via drmaa if it is not set up to do so.
Best, Marius
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s
To search Galaxy mailing lists use the unified search at: [ [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ] | [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ] ]
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s
To search Galaxy mailing lists use the unified search at: [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ]
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Hi, I just wanted to jump in and point out real quick:
Did you restart pulsar after rolling out the new config ?
to which you've answered that you re-ran the playbook, but that's not a sufficient answer.
RUNNING HANDLER [galaxyproject.pulsar : default restart pulsar handler] ********************************************************************************* skipping: [HPC]
The "restart Pulsar" step in the playbook is being skipped, so it looks like the Pulsar server has not, in fact, been restarted. Cheers, Keith
Currently, we never used DRMAA. The job were executed immediately on the cluster with CLI or DRMAA. We had this part in pulsarservers.yml, to activate CLI: managers: _default_: type: queued_cli job_plugin: slurm native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" min_polling_interval: 0.5 amqp_publish_retry: True amqp_publish_retry_max_retries: 5 amqp_publish_retry_interval_start: 10 amqp_publish_retry_interval_step: 10 amqp_publish_retry_interval_max: 60
Thanks for your help, Luc
------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano
----- Mail original ----- De: "Marius van den Beek" <m.vandenbeek@gmail.com> À: "Luc Cornet" <luc.cornet@uliege.be> Cc: "HelpGalaxy" <galaxy-dev@lists.galaxyproject.org>, "Baurain Denis" <Denis.Baurain@uliege.be>, "Pierre Becker" <Pierre.Becker@sciensano.be>, "Colignon David" <David.Colignon@uliege.be> Envoyé: Mercredi 30 Juin 2021 16:02:04 Objet: [galaxy-dev] Re: Galaxy install problems
Hi Luc,
I'm sorry to hear that you're struggling to set up Galaxy to your liking. Let me start by pointing out that [ http://usegalaxy.org/ | usegalaxy.org ] uses slurm with DRMAA, this is certainly going to be more performant and reliable than the CLI plugin. There is little maintenance necessary, so maybe that is why activity on slurm-drmaa is low (See also [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] ). I would be curious to know how you came to the conclusion that there is some incompatibility between DRMAA and slurm Note that one of the setups we teach during the training submits via DRMAA to slurm.
Then I'd like to point out that there are a huge variety of different ways in which you can configure Galaxy and the job submission. We teach the most common ones during the training week, with the aim that you understand how these things work together, as well as giving you a handle on how you can manage these different settings and services using a configuration management system. We cannot tailor a solution to your infrastructure during this week.
About your problem specifically, I had asked this on gitter before:
Did you restart pulsar after rolling out the new config ?
to which you've answered that you re-ran the playbook, but that's not a sufficient answer.
Every playbook is different, and we cannot know if this includes a restarter service for pulsar. Also please don't assume that everyone that could potentially help you knows ansible and the playbooks that are being taught intimately, and in what ways you have customized your playbook. It is much more helpful to write up the relevant settings you've changed and the logs that go with it.
You've also been asked to provide logs of the restart, which as far as I can tell you haven't provided. You had mentioned on gitter that pulsar continues to use DRMAA to submit jobs, so you'll want to double check whether you've really restarted pulsar after the config changes, and look at the startup logs for pulsar, and find out how it is possible for pulsar to submit jobs via drmaa if it is not set up to do so.
Best, Marius
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/<pulsarservers.yml><pulsar.yml><log_puls_2.log>___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
On Wed, Jun 30, 2021 at 10:30 AM Luc Cornet <luc.cornet@uliege.be> wrote:
Dear Marius,
Many thank for your feedback.
I join to this email: the playbook, the pulsarservers.yml file and the log of pulsar playbook.
CLI plugin is for us the best solution since we have nothing to maintain. DRMAA is not actively developed for slurm, correct ?
Just to clarify, I am actively creating slurm-drmaa releases from updates that I've done plus many community contributions at https://github.com/natefoo/slurm-drmaa. We have fixed incompatibilities and bugs in slurm-drmaa with newer versions of Slurm, and have added support for new features in newer versions of Slurm. It was never a part of Slurm, if that is what you're asking, but as Marius said, it is in active use on usegalaxy.org and many other Galaxy servers, as well as in other applications.
In the playbook, we use systemd which I think should restart pulsar but It might not be the case:
TASK [galaxyproject.pulsar : systemd daemon-reload and enable/start service] **************************************************************************** ok: [HPC]
RUNNING HANDLER [galaxyproject.pulsar : default restart pulsar handler] ********************************************************************************* skipping: [HPC]
Currently, we never used DRMAA. The job were executed immediately on the cluster with CLI or DRMAA. We had this part in pulsarservers.yml, to activate CLI: managers: _default_: type: queued_cli job_plugin: slurm native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" min_polling_interval: 0.5 amqp_publish_retry: True amqp_publish_retry_max_retries: 5 amqp_publish_retry_interval_start: 10 amqp_publish_retry_interval_step: 10 amqp_publish_retry_interval_max: 60
Thanks for your help, Luc
------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano
----- Mail original ----- De: "Marius van den Beek" <m.vandenbeek@gmail.com> À: "Luc Cornet" <luc.cornet@uliege.be> Cc: "HelpGalaxy" <galaxy-dev@lists.galaxyproject.org>, "Baurain Denis" < Denis.Baurain@uliege.be>, "Pierre Becker" <Pierre.Becker@sciensano.be>, "Colignon David" <David.Colignon@uliege.be> Envoyé: Mercredi 30 Juin 2021 16:02:04 Objet: [galaxy-dev] Re: Galaxy install problems
Hi Luc,
I'm sorry to hear that you're struggling to set up Galaxy to your liking. Let me start by pointing out that [ http://usegalaxy.org/ | usegalaxy.org ] uses slurm with DRMAA, this is certainly going to be more performant and reliable than the CLI plugin. There is little maintenance necessary, so maybe that is why activity on slurm-drmaa is low (See also [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] ). I would be curious to know how you came to the conclusion that there is some incompatibility between DRMAA and slurm Note that one of the setups we teach during the training submits via DRMAA to slurm.
Then I'd like to point out that there are a huge variety of different ways in which you can configure Galaxy and the job submission. We teach the most common ones during the training week, with the aim that you understand how these things work together, as well as giving you a handle on how you can manage these different settings and services using a configuration management system. We cannot tailor a solution to your infrastructure during this week.
About your problem specifically, I had asked this on gitter before:
Did you restart pulsar after rolling out the new config ?
to which you've answered that you re-ran the playbook, but that's not a sufficient answer.
Every playbook is different, and we cannot know if this includes a restarter service for pulsar. Also please don't assume that everyone that could potentially help you knows ansible and the playbooks that are being taught intimately, and in what ways you have customized your playbook. It is much more helpful to write up the relevant settings you've changed and the logs that go with it.
You've also been asked to provide logs of the restart, which as far as I can tell you haven't provided. You had mentioned on gitter that pulsar continues to use DRMAA to submit jobs, so you'll want to double check whether you've really restarted pulsar after the config changes, and look at the startup logs for pulsar, and find out how it is possible for pulsar to submit jobs via drmaa if it is not set up to do so.
Best, Marius
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/ ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Thanks for the infos. On our HPC system, slurm is a container pre-installed by the company who installed the system. slurm-drmaa is not included in the container and installing would be difficult. This is why we choose CLI instead. In order to use drmaa with galaxy, we should install slurm-drmaa in the pre-installed slurm container. Is it correct ? best, Luc ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano ----- Mail original ----- De: "Nate Coraor" <nate@bx.psu.edu> À: "Luc Cornet" <luc.cornet@uliege.be> Cc: "HelpGalaxy" <galaxy-dev@lists.galaxyproject.org>, "Baurain Denis" <Denis.Baurain@uliege.be>, "Pierre Becker" <Pierre.Becker@sciensano.be>, "Colignon David" <David.Colignon@uliege.be> Envoyé: Mercredi 30 Juin 2021 16:50:13 Objet: [galaxy-dev] Re: Galaxy install problems On Wed, Jun 30, 2021 at 10:30 AM Luc Cornet < [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] > wrote: Dear Marius, Many thank for your feedback. I join to this email: the playbook, the pulsarservers.yml file and the log of pulsar playbook. CLI plugin is for us the best solution since we have nothing to maintain. DRMAA is not actively developed for slurm, correct ? Just to clarify, I am actively creating slurm-drmaa releases from updates that I've done plus many community contributions at [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] . We have fixed incompatibilities and bugs in slurm-drmaa with newer versions of Slurm, and have added support for new features in newer versions of Slurm. It was never a part of Slurm, if that is what you're asking, but as Marius said, it is in active use on [ http://usegalaxy.org/ | usegalaxy.org ] and many other Galaxy servers, as well as in other applications. In the playbook, we use systemd which I think should restart pulsar but It might not be the case: TASK [galaxyproject.pulsar : systemd daemon-reload and enable/start service] **************************************************************************** ok: [HPC] RUNNING HANDLER [galaxyproject.pulsar : default restart pulsar handler] ********************************************************************************* skipping: [HPC] Currently, we never used DRMAA. The job were executed immediately on the cluster with CLI or DRMAA. We had this part in pulsarservers.yml, to activate CLI: managers: _default_: type: queued_cli job_plugin: slurm native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" min_polling_interval: 0.5 amqp_publish_retry: True amqp_publish_retry_max_retries: 5 amqp_publish_retry_interval_start: 10 amqp_publish_retry_interval_step: 10 amqp_publish_retry_interval_max: 60 Thanks for your help, Luc ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano ----- Mail original ----- De: "Marius van den Beek" < [ mailto:m.vandenbeek@gmail.com | m.vandenbeek@gmail.com ] > À: "Luc Cornet" < [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ] > Cc: "HelpGalaxy" < [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] >, "Baurain Denis" < [ mailto:Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] >, "Pierre Becker" < [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] >, "Colignon David" < [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] > Envoyé: Mercredi 30 Juin 2021 16:02:04 Objet: [galaxy-dev] Re: Galaxy install problems Hi Luc, I'm sorry to hear that you're struggling to set up Galaxy to your liking. Let me start by pointing out that [ [ http://usegalaxy.org/ | http://usegalaxy.org/ ] | [ http://usegalaxy.org/ | usegalaxy.org ] ] uses slurm with DRMAA, this is certainly going to be more performant and reliable than the CLI plugin. There is little maintenance necessary, so maybe that is why activity on slurm-drmaa is low (See also [ [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] | [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] ] ). I would be curious to know how you came to the conclusion that there is some incompatibility between DRMAA and slurm Note that one of the setups we teach during the training submits via DRMAA to slurm. Then I'd like to point out that there are a huge variety of different ways in which you can configure Galaxy and the job submission. We teach the most common ones during the training week, with the aim that you understand how these things work together, as well as giving you a handle on how you can manage these different settings and services using a configuration management system. We cannot tailor a solution to your infrastructure during this week. About your problem specifically, I had asked this on gitter before:
Did you restart pulsar after rolling out the new config ?
to which you've answered that you re-ran the playbook, but that's not a sufficient answer. Every playbook is different, and we cannot know if this includes a restarter service for pulsar. Also please don't assume that everyone that could potentially help you knows ansible and the playbooks that are being taught intimately, and in what ways you have customized your playbook. It is much more helpful to write up the relevant settings you've changed and the logs that go with it. You've also been asked to provide logs of the restart, which as far as I can tell you haven't provided. You had mentioned on gitter that pulsar continues to use DRMAA to submit jobs, so you'll want to double check whether you've really restarted pulsar after the config changes, and look at the startup logs for pulsar, and find out how it is possible for pulsar to submit jobs via drmaa if it is not set up to do so. Best, Marius ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s To search Galaxy mailing lists use the unified search at: [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ] ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s To search Galaxy mailing lists use the unified search at: [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ] ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
On Wed, Jun 30, 2021 at 11:00 AM Luc Cornet <luc.cornet@uliege.be> wrote:
Thanks for the infos.
On our HPC system, slurm is a container pre-installed by the company who installed the system. slurm-drmaa is not included in the container and installing would be difficult. This is why we choose CLI instead.
In order to use drmaa with galaxy, we should install slurm-drmaa in the pre-installed slurm container. Is it correct ?
Whichever application (Galaxy or Pulsar - you said Pulsar originally but Galaxy in your most recent message) is going to interact with Slurm needs to be installed on a system configured as a Slurm client. This means that you should be able to run `squeue`, `sinfo`, etc. from the command line on that system. Once that is the case, there are 3 steps: 1. Install slurm-drmaa on the Galaxy or Pulsar server, *not* in the slurm Controller container. 2. `pip install drmaa` into Galaxy or Pulsar's virtualenv. This is already done in the case of your Pulsar server since you have `drmaa` included in `pulsar_optional_dependencies`. For Galaxy, the `galaxyproject.galaxy` role will do this for you automatically if you have enabled a DRMAA-based job runner plugin (e.g DRMAAJobRunner or SlurmJobRunner) in job_conf.xml. 3. Configure the Python drmaa library to find slurm-drmaa's libdrmaa.so, either via the DRMAA_LIBRARY_PATH environment variable, or in the case of Galaxy, in the `drmaa_library_path` runner plugin param, as shown in this example: https://github.com/galaxyproject/galaxy/blob/e74239e010ece4a4b22d7a6fe0f0f3d... --nate
best, Luc
------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano
----- Mail original ----- De: "Nate Coraor" <nate@bx.psu.edu> À: "Luc Cornet" <luc.cornet@uliege.be> Cc: "HelpGalaxy" <galaxy-dev@lists.galaxyproject.org>, "Baurain Denis" < Denis.Baurain@uliege.be>, "Pierre Becker" <Pierre.Becker@sciensano.be>, "Colignon David" <David.Colignon@uliege.be> Envoyé: Mercredi 30 Juin 2021 16:50:13 Objet: [galaxy-dev] Re: Galaxy install problems
On Wed, Jun 30, 2021 at 10:30 AM Luc Cornet < [ mailto: luc.cornet@uliege.be | luc.cornet@uliege.be ] > wrote:
Dear Marius,
Many thank for your feedback.
I join to this email: the playbook, the pulsarservers.yml file and the log of pulsar playbook.
CLI plugin is for us the best solution since we have nothing to maintain. DRMAA is not actively developed for slurm, correct ?
Just to clarify, I am actively creating slurm-drmaa releases from updates that I've done plus many community contributions at [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] . We have fixed incompatibilities and bugs in slurm-drmaa with newer versions of Slurm, and have added support for new features in newer versions of Slurm. It was never a part of Slurm, if that is what you're asking, but as Marius said, it is in active use on [ http://usegalaxy.org/ | usegalaxy.org ] and many other Galaxy servers, as well as in other applications.
In the playbook, we use systemd which I think should restart pulsar but It might not be the case:
TASK [galaxyproject.pulsar : systemd daemon-reload and enable/start service] ****************************************************************************
ok: [HPC]
RUNNING HANDLER [galaxyproject.pulsar : default restart pulsar handler] *********************************************************************************
skipping: [HPC]
Currently, we never used DRMAA. The job were executed immediately on the cluster with CLI or DRMAA. We had this part in pulsarservers.yml, to activate CLI: managers: _default_: type: queued_cli job_plugin: slurm native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" min_polling_interval: 0.5 amqp_publish_retry: True amqp_publish_retry_max_retries: 5 amqp_publish_retry_interval_start: 10 amqp_publish_retry_interval_step: 10 amqp_publish_retry_interval_max: 60
Thanks for your help, Luc
------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano
----- Mail original ----- De: "Marius van den Beek" < [ mailto:m.vandenbeek@gmail.com | m.vandenbeek@gmail.com ] > À: "Luc Cornet" < [ mailto:luc.cornet@uliege.be | luc.cornet@uliege.be ]
Cc: "HelpGalaxy" < [ mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org ] >, "Baurain Denis" < [ mailto: Denis.Baurain@uliege.be | Denis.Baurain@uliege.be ] >, "Pierre Becker" < [ mailto:Pierre.Becker@sciensano.be | Pierre.Becker@sciensano.be ] >, "Colignon David" < [ mailto:David.Colignon@uliege.be | David.Colignon@uliege.be ] > Envoyé: Mercredi 30 Juin 2021 16:02:04 Objet: [galaxy-dev] Re: Galaxy install problems
Hi Luc,
I'm sorry to hear that you're struggling to set up Galaxy to your liking. Let me start by pointing out that [ [ http://usegalaxy.org/ | http://usegalaxy.org/ ] | [ http://usegalaxy.org/ | usegalaxy.org ] ] uses slurm with DRMAA, this is certainly going to be more performant and reliable than the CLI plugin. There is little maintenance necessary, so maybe that is why activity on slurm-drmaa is low (See also [ [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] | [ https://github.com/natefoo/slurm-drmaa | https://github.com/natefoo/slurm-drmaa ] ] ). I would be curious to know how you came to the conclusion that there is some incompatibility between DRMAA and slurm Note that one of the setups we teach during the training submits via DRMAA to slurm.
Then I'd like to point out that there are a huge variety of different ways in which you can configure Galaxy and the job submission. We teach the most common ones during the training week, with the aim that you understand how these things work together, as well as giving you a handle on how you can manage these different settings and services using a configuration management system. We cannot tailor a solution to your infrastructure during this week.
About your problem specifically, I had asked this on gitter before:
Did you restart pulsar after rolling out the new config ?
to which you've answered that you re-ran the playbook, but that's not a sufficient answer.
Every playbook is different, and we cannot know if this includes a restarter service for pulsar. Also please don't assume that everyone that could potentially help you knows ansible and the playbooks that are being taught intimately, and in what ways you have customized your playbook. It is much more helpful to write up the relevant settings you've changed and the logs that go with it.
You've also been asked to provide logs of the restart, which as far as I can tell you haven't provided. You had mentioned on gitter that pulsar continues to use DRMAA to submit jobs, so you'll want to double check whether you've really restarted pulsar after the config changes, and look at the startup logs for pulsar, and find out how it is possible for pulsar to submit jobs via drmaa if it is not set up to do so.
Best, Marius
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s
To search Galaxy mailing lists use the unified search at: [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ] ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s
To search Galaxy mailing lists use the unified search at: [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ]
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
participants (4)
-
Luc Cornet -
Marius van den Beek -
Nate Coraor -
Suderman Keith