New subject: Running slurm job with pulsar

18 Jun 2021


      Dear Gianmauro,
Thanks for your answer. I think that my question was not clear enough. 
Below you will find some answer to your suggestions and maybe it will be more clear for you to guide me.
* check the Pulsar' log for error messages
->I look into staging directory but as my test analysis is executed with success, I have no error log.
When a lunch a test from the galaxy GUI, it is executed with success on the cluster.
My problem is that is not executed in a job, not using srun or sbatch command (so not using the scheduler).
I would like pulsar to be able to submit a job on the cluster, just like others users, and not execute analysis directly "in the terminal".
* verify if your Pulsar server can reach the cluster trough ssh
->The pulsar server is on the HPC cluster. The connection between galaxy (rabbitmq) and the cluster (pulsar) is fine.
* in the staging directory of your job should be a command.sh file. You 
can try to run it manually (sbatch command.sh or something similar) and
Yes indeed I have this file in staging directory (see below).
We i execute the command.sh file with sbatch command.sh, it fails immediately which is normal since command.sh is not a slurm job.
total 48
-rw-r--r-- 1 pulsar pulsar    4 Jun 17 14:15 use_metadata_directory
-rw-r--r-- 1 pulsar pulsar   10 Jun 17 14:15 tool_version
-rw-r--r-- 1 pulsar pulsar   59 Jun 17 14:15 tool_id
drwxr-xr-x 2 pulsar pulsar    6 Jun 17 14:15 tool_files
drwxr-xr-x 2 pulsar pulsar    6 Jun 17 14:15 metadata
drwxr-xr-x 2 pulsar pulsar    6 Jun 17 14:15 configs
drwxr-xr-x 5 pulsar pulsar   30 Jun 17 14:15 ..
-rw-r--r-- 1 pulsar pulsar 2551 Jun 17 14:15 launch_config
drwxr-xr-x 2 pulsar pulsar   46 Jun 17 14:15 inputs
-rw-r--r-- 1 pulsar pulsar    4 Jun 17 14:15 preprocessed
-rwx------ 1 pulsar pulsar 5441 Jun 17 14:15 command.sh
-rw-r--r-- 1 pulsar pulsar    0 Jun 17 14:15 stdout
drwxr-xr-x 2 pulsar pulsar    6 Jun 17 14:15 home
drwxr-xr-x 2 pulsar pulsar   58 Jun 17 14:15 working
-rw-r--r-- 1 pulsar pulsar    4 Jun 17 14:15 running
-rw-r--r-- 1 pulsar pulsar  546 Jun 17 14:15 stderr
drwxr-xr-x 2 pulsar pulsar   26 Jun 17 14:15 outputs
-rw-r--r-- 1 pulsar pulsar    1 Jun 17 14:15 return_code
-rw-r--r-- 1 pulsar pulsar   10 Jun 17 14:15 final_status
-rw-r--r-- 1 pulsar pulsar    0 Jun 17 14:15 postprocessed
drwxr-xr-x 9 pulsar pulsar 4096 Jun 17 14:15 .
------------
Luc Cornet, PhD
Bio-informatician 
Mycology and Aerobiology
Sciensano
----- Mail original -----
De: "Gianmauro Cuccuru" gmauro@informatik.uni-freiburg.de
À: "Luc Cornet" luc.cornet@uliege.be, "HelpGalaxy" galaxy-dev@lists.galaxyproject.org
Cc: "Colignon David" David.Colignon@uliege.be, "Baurain Denis" Denis.Baurain@uliege.be, "Pierre Becker" Pierre.Becker@sciensano.be
Envoyé: Vendredi 18 Juin 2021 11:48:30
Objet: Re: [galaxy-dev] Running slurm job with pulsar
Hi Luc,
I am not a Slurm expert but I can suggest you several things:
* check the Pulsar' log for error messages
* verify if your Pulsar server can reach the cluster trough ssh
* in the staging directory of your job should be a command.sh file. You 
can try to run it manually (sbatch command.sh or something similar) and 
see if it works
Cheers,
Gianmauro
On 17.06.21 20:18, Luc Cornet wrote:
...
Dear all,
I am trying to launch a slurm job with pulsar using CLI (instead of drmaa).
The pulsar playbook below pass without problem but the analyses is still run out of slurm.
The analyses is excited with success but not in a slrum job.
Can you help me to launch slurm with pulsar ?
What did I miss?
Thanks,
Luc
# Put your Galaxy server's fully qualified domain name (FQDN) (or the FQDN of the RabbitMQ server) above.

pulsar_root: /opt/pulsar

pulsar_pip_install: true
pulsar_pycurl_ssl_library: openssl
pulsar_systemd: true
pulsar_systemd_runner: webless

pulsar_create_user: false
pulsar_user: {name: pulsar, shell: /bin/bash}

pulsar_optional_dependencies:
   - pyOpenSSL
   # For remote transfers initiated on the Pulsar end rather than the Galaxy end
   - pycurl
   # drmaa required if connecting to an external DRM using it.
   - drmaa
   # kombu needed if using a message queue
   - kombu
   # amqp 5.0.3 changes behaviour in an unexpected way, pin for now.
   - 'amqp==5.0.2'
   # psutil and pylockfile are optional dependencies but can make Pulsar
   # more robust in small ways.
   - psutil

pulsar_yaml_config:
   conda_auto_init: True
   conda_auto_install: True
   staging_directory: "{{ pulsar_staging_dir }}"
   persistence_directory: "{{ pulsar_persistence_dir }}"
   tool_dependency_dir: "{{ pulsar_dependencies_dir }}"
   # The following are the settings for the pulsar server to contact the message queue with related timeouts etc.
   message_queue_url: "pyamqp://galaxy_au:{{ rabbitmq_password_galaxy_au }}@{{ galaxy_server_url }}:5671//pulsar/galaxy_au?ssl=1"
   managers:
     _default_:
       type: queued_cli
       job_plugin: slurm
       native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00"
       min_polling_interval: 0.5
       amqp_publish_retry: True
       amqp_publish_retry_max_retries: 5
       amqp_publish_retry_interval_start: 10
       amqp_publish_retry_interval_step: 10
       amqp_publish_retry_interval_max: 60

# We also need to create the dependency resolver file so pulsar knows how to
# find and install dependencies for the tools we ask it to run. The simplest
# method which covers 99% of the use cases is to use conda auto installs similar
# to how Galaxy works.
pulsar_dependency_resolvers:
   - name: conda
     args:
       - name: auto_init
         value: true


Luc Cornet, PhD
Bio-informatician
Mycology and Aerobiology
Sciensano
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   %(web_page_url)s
To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/
-- 
Gianmauro Cuccuru

UseGalaxy.eu
Bioinformatics Group
Department of Computer Science
Albert-Ludwigs-University Freiburg
Georges-Köhler-Allee 106
79110 Freiburg, Germany

Re: Running slurm job with pulsar