- galaxy-dev - lists.galaxyproject.org

November 2021 Galaxy Newsletter
by Dave Clements 01 Nov '21

01 Nov '21

Happy November from Galaxy! The November newsletter is thin, *but powerful*: - Galaxy India Launches this Month <https://galaxyproject.org/news/2021-11-galaxy-update/#community-news> - 7 new openings (13 total) <https://galaxyproject.org/news/2021-11-galaxy-update/#community-news> - 9 upcoming Events <https://galaxyproject.org/news/2021-11-galaxy-update/#event-news> - Blog posts on SARS-CoV-2 training in Africa; Sunsetting CloudLaundh; and TIaaS <https://galaxyproject.org/news/2021-11-galaxy-update/#galactic-blog-posts> - UseGalaxy.* Updates <https://galaxyproject.org/news/2021-11-galaxy-update/#galaxy-platforms-news> - Other News too: BY-COVID Launches <https://galaxyproject.org/news/2021-11-galaxy-update/#other-news> Have something for the next newsletter? Please let us know and we will spread the word. Beatriz Serrano-Solano and Dave Clements, Editors -- https://galaxyproject.org/

1 0

October Galaxy News
by Dave Clements 11 Oct '21

11 Oct '21

Hello all, The October Newsletter is out <https://galaxyproject.org/news/2021-10-galaxy-update/>. and it covers Galaxy involvement in Hacktoberfest, and Outreachy, plus a nice batch of trainings, talks and a Galaxy Papercuts CoFest day too. It also brings news of new job openings on two continents, two new platforms, blog posts (where My Little Pony makes an appearance, really), GTN updates (few in number but big in their coolness), publications, and releases. In other words, another active month in the Galaxy. Beatriz Serrano-Solano and Dave Clements, Editors -- https://galaxyproject.org/

1 0

September 2021 News: Papercuts resumes; Misconceptions; Hub updated; microGalaxy Community, ...
by Dave Clements 16 Sep '21

16 Sep '21

Hi all, The September newsletter <https://galaxyproject.org/news/2021-09-galaxy-update/> brings lots of news, including - 12 recent use cases for UseGalaxy.eu, covering a broad spectrum of research. - Want to help address misconceptions about Galaxy? - The Galaxy Community Hub is now using Gridsome. - ELIXIR community updates - JXTX Foundation 2021 Awards - 7 open positions in Belgium, Italy, and the US - The Galaxy YouTube channel is growing up - Upcoming events - Galaxy Platforms News - Recent Blog Posts - Doc, Hub, and Training Updates - Recent Publications - and other news too. Sit back, get comfortable, and explore the Galaxy... Beatriz Serrano-Solano and Dave Clements, Editors -- https://galaxyproject.org/

1 0

PaperCuts is Resuming this Thursday, Septermber 16
by Dave Clements 15 Sep '21

15 Sep '21

Hello all, After taking the last 3 months off, PaperCuts is back <https://galaxyproject.org/events/2021-09-papercuts/>. Galaxy PaperCuts is a monthly online gathering where we address irritating but relatively easy to fix bugs in anything related to Galaxy. This can be code, documentation, training materials, Ansible recipes, and do on. We will be on Gitter <https://gitter.im/galaxyproject/Lobby> for chat all day long, and on 3 calls spread across the day. Please take advantage of both to communicate with your collaborators around the world. - *Call 1: Oceania, Australia, Asia* - 14:00 Australia Eastern time. See your time <https://www.timeanddate.com/worldclock/fixedtime.html?msg=14%3A00+Melbourne…> . - Zoom Link <https://zoom.us/j/98189403566?pwd=SWhueXkzNHlKSG1QMU0xUG1ESFFsUT09> - *Call 2: Middle East, Europe, Africa* - 10:00 Central European Summer time. See your time <https://www.timeanddate.com/worldclock/fixedtime.html?msg=10%3A00+Freiburg+…> . - Zoom Link <https://us02web.zoom.us/j/89108992104?pwd=eHhWdFFUQThnVVFwZ3RRall4WjBXQT09> - *Call 3: Americas* - 12:00 US Eastern time. See your time <https://www.timeanddate.com/worldclock/fixedtime.html?msg=12%3A00+Penn+Stat…> . - Zoom Link <https://zoom.us/j/98189403566?pwd=SWhueXkzNHlKSG1QMU0xUG1ESFFsUT09> If you have any ideas for papercut issues, please add them to the appropriate repo. Thanks, and hope to see you soon, Dave C -- https://galaxyproject.org/

1 0

Galaxy Newsletter: August 2021
by Beatriz Serrano-Solano 06 Aug '21

06 Aug '21

Hello all, The August newsletter brings news of GCC2021 (just finished) and GCC2022 (in Minneapolis); the launch of the Galaxy India community, and open positions in 7 countries <https://galaxyproject.org/news/2021-08-galaxy-update/#community-news>; event news <https://galaxyproject.org/news/2021-08-galaxy-update/#event-news>; Galaxy platform news <https://galaxyproject.org/news/2021-08-galaxy-update/#galaxy-platforms-news>; Galactic blog posts <https://galaxyproject.org/news/2021-08-galaxy-update/#galactic-blog-activity>; training and doc updates <https://galaxyproject.org/news/2021-08-galaxy-update/#doc-hub-and-training-…>; recent open-access Galaxy-related publications <https://galaxyproject.org/news/2021-08-galaxy-update/#publications>; and other news too <https://galaxyproject.org/news/2021-08-galaxy-update/#other-news>.** Dave Clements and Beatriz Serrano-Solano, Editors PS: Subscribe to the Galaxy Announce mailing list <https://lists.galaxyproject.org/?count=100&all-lists=> and receive an email whenever this newsletter is published. -- Dr Beatriz Serrano-Solano (she/her) <https://www.hrc.org/resources/talking-about-pronouns-in-the-workplace> Bioinformatics Group Dept. Computer Science Albert-Ludwigs-Universität Freiburg Georges-Köhler-Allee, 106 79110 Freiburg Germany

1 0

Galaxy install problems
by Luc Cornet 09 Jul '21

09 Jul '21

Dear Galaxy admins, I am contacting you due to multiple problems during the installation of our own server of galaxy. We have been struggling to install galaxy on our HPC system for 1.5 year. We have faced so many problems while installing galaxy that we have lost count. Yet, our main problem was and remains the use of pulsar... Even after following two training sessions, switched to a brand new CentOS 8 cluster (while CentOS 8 will die at the end of the year), we are still not able to use our galaxy instance. We followed the ansible documentation and we are now able to reproduce the training sessions (i.e, execute an analysis with pulsar on our HPC system). Nevertheless, we are not able to connect pulsar with slurm (which makes it unusable). We do not want to use DRMAA because it is not well maintained and compatible with slurm. Instead, we choose to use CLI (as mentioned in the galaxy docs) but we are stuck (see below the pulsarservers.yaml file). Is it possible to use something else than DRMAA to connect pulsar to the scheduler? Thanks, Best regards Luc Cornet - - pulsarservers.yaml - - # Put your Galaxy server's fully qualified domain name (FQDN) (or the FQDN of the RabbitMQ server) above. pulsar_root: /opt/pulsar pulsar_pip_install: true pulsar_pycurl_ssl_library: openssl pulsar_systemd: true pulsar_systemd_runner: webless pulsar_create_user: false pulsar_user: {name: pulsar, shell: /bin/bash} pulsar_optional_dependencies: - pyOpenSSL # For remote transfers initiated on the Pulsar end rather than the Galaxy end - pycurl # drmaa required if connecting to an external DRM using it. - drmaa # kombu needed if using a message queue - kombu # amqp 5.0.3 changes behaviour in an unexpected way, pin for now. - 'amqp==5.0.2' # psutil and pylockfile are optional dependencies but can make Pulsar # more robust in small ways. - psutil pulsar_yaml_config: conda_auto_init: True conda_auto_install: True staging_directory: "{{ pulsar_staging_dir }}" persistence_directory: "{{ pulsar_persistence_dir }}" tool_dependency_dir: "{{ pulsar_dependencies_dir }}" # The following are the settings for the pulsar server to contact the message queue with related timeouts etc. message_queue_url: "pyamqp://galaxy_au:{{ rabbitmq_password_galaxy_au }}@{{ galaxy_server_url }}:5671//pulsar/galaxy_au?ssl=1" managers: _default_: type: queued_cli job_plugin: slurm native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" min_polling_interval: 0.5 amqp_publish_retry: True amqp_publish_retry_max_retries: 5 amqp_publish_retry_interval_start: 10 amqp_publish_retry_interval_step: 10 amqp_publish_retry_interval_max: 60 # We also need to create the dependency resolver file so pulsar knows how to # find and install dependencies for the tools we ask it to run. The simplest # method which covers 99% of the use cases is to use conda auto installs similar # to how Galaxy works. pulsar_dependency_resolvers: - name: conda args: - name: auto_init value: true ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano

4 16

Galaxy Newsletter: July 2021
by Dave Clements 02 Jul '21

02 Jul '21

Hello all, The big news for July is that the 2021 Galaxy Community Conference <https://gcc2021.sched.com/> is under way. With GCC going on, this is one of our thinner newsletters <https://galaxyproject.org/news/2021-07-galaxy-update/>, but there are still *open positions <https://galaxyproject.org/news/2021-07-galaxy-update/#community-news>*, *event news <https://galaxyproject.org/news/2021-07-galaxy-update/#event-news>*, *Galaxy platform news <https://galaxyproject.org/news/2021-07-galaxy-update/#galaxy-platforms-news>*, *Galactic blog posts <https://galaxyproject.org/news/2021-07-galaxy-update/#galactic-blog-activity>*, *training and doc updates <https://galaxyproject.org/news/2021-07-galaxy-update/#doc-hub-and-training-…>*, recent *open-access Galaxy-related publications <https://galaxyproject.org/news/2021-07-galaxy-update/#publications>*, *new releases <https://galaxyproject.org/news/2021-07-galaxy-update/#releases>, and other news too <https://galaxyproject.org/news/2021-07-galaxy-update/#other-news>.* We hope to see you at GCC events this week and next. Dave Clements and Beatriz Serrano-Solano, Editors PS: It is too late to register for the 3 day conference next week. It is also officially too late to sign up for the CoFest on 9-10 July, but if you follow this link <https://docs.google.com/forms/d/e/1FAIpQLScdJHTzw7LWVxFaqGI30COZlcxGPksjGL_…>, you just mind find yourself invited anyway. -- https://gaaxyproject.org/ <https://galaxyproject.org/>

1 0

GCC202 Registration Ends Tomorrow
by Beatriz Serrano-Solano 24 Jun '21

24 Jun '21

Hello All, The final registration for the 2021 Galaxy Community Conference is closing tomorrow, Friday 25th of June, at midnight CEST, so you have still some time to register! https://www.vibconferences.be/events/gcc2021-virtual-edition We will meet in Remo (https://remo.co/) and the schedule is already available at https://gcc2021.sched.com/ We are looking forward to seeing you online! Bea on behalf of the GCC2021 Organizing Committee -- Dr Beatriz Serrano-Solano (she/her) Bioinformatics Group Dept. Computer Science Albert-Ludwigs-Universität Freiburg Georges-Köhler-Allee, 106 79110 Freiburg Germany

1 0

Re: Running slurm job with pulsar
by Luc Cornet 18 Jun '21

18 Jun '21

Dear Gianmauro, Thanks for your answer. I think that my question was not clear enough. Below you will find some answer to your suggestions and maybe it will be more clear for you to guide me. * check the Pulsar' log for error messages ->I look into staging directory but as my test analysis is executed with success, I have no error log. When a lunch a test from the galaxy GUI, it is executed with success on the cluster. My problem is that is not executed in a job, not using srun or sbatch command (so not using the scheduler). I would like pulsar to be able to submit a job on the cluster, just like others users, and not execute analysis directly "in the terminal". * verify if your Pulsar server can reach the cluster trough ssh ->The pulsar server is on the HPC cluster. The connection between galaxy (rabbitmq) and the cluster (pulsar) is fine. * in the staging directory of your job should be a command.sh file. You can try to run it manually (sbatch command.sh or something similar) and Yes indeed I have this file in staging directory (see below). We i execute the command.sh file with sbatch command.sh, it fails immediately which is normal since command.sh is not a slurm job. total 48 -rw-r--r-- 1 pulsar pulsar 4 Jun 17 14:15 use_metadata_directory -rw-r--r-- 1 pulsar pulsar 10 Jun 17 14:15 tool_version -rw-r--r-- 1 pulsar pulsar 59 Jun 17 14:15 tool_id drwxr-xr-x 2 pulsar pulsar 6 Jun 17 14:15 tool_files drwxr-xr-x 2 pulsar pulsar 6 Jun 17 14:15 metadata drwxr-xr-x 2 pulsar pulsar 6 Jun 17 14:15 configs drwxr-xr-x 5 pulsar pulsar 30 Jun 17 14:15 .. -rw-r--r-- 1 pulsar pulsar 2551 Jun 17 14:15 launch_config drwxr-xr-x 2 pulsar pulsar 46 Jun 17 14:15 inputs -rw-r--r-- 1 pulsar pulsar 4 Jun 17 14:15 preprocessed -rwx------ 1 pulsar pulsar 5441 Jun 17 14:15 command.sh -rw-r--r-- 1 pulsar pulsar 0 Jun 17 14:15 stdout drwxr-xr-x 2 pulsar pulsar 6 Jun 17 14:15 home drwxr-xr-x 2 pulsar pulsar 58 Jun 17 14:15 working -rw-r--r-- 1 pulsar pulsar 4 Jun 17 14:15 running -rw-r--r-- 1 pulsar pulsar 546 Jun 17 14:15 stderr drwxr-xr-x 2 pulsar pulsar 26 Jun 17 14:15 outputs -rw-r--r-- 1 pulsar pulsar 1 Jun 17 14:15 return_code -rw-r--r-- 1 pulsar pulsar 10 Jun 17 14:15 final_status -rw-r--r-- 1 pulsar pulsar 0 Jun 17 14:15 postprocessed drwxr-xr-x 9 pulsar pulsar 4096 Jun 17 14:15 . ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano ----- Mail original ----- De: "Gianmauro Cuccuru" <gmauro(a)informatik.uni-freiburg.de> À: "Luc Cornet" <luc.cornet(a)uliege.be>, "HelpGalaxy" <galaxy-dev(a)lists.galaxyproject.org> Cc: "Colignon David" <David.Colignon(a)uliege.be>, "Baurain Denis" <Denis.Baurain(a)uliege.be>, "Pierre Becker" <Pierre.Becker(a)sciensano.be> Envoyé: Vendredi 18 Juin 2021 11:48:30 Objet: Re: [galaxy-dev] Running slurm job with pulsar Hi Luc, I am not a Slurm expert but I can suggest you several things: * check the Pulsar' log for error messages * verify if your Pulsar server can reach the cluster trough ssh * in the staging directory of your job should be a command.sh file. You can try to run it manually (sbatch command.sh or something similar) and see if it works Cheers, Gianmauro On 17.06.21 20:18, Luc Cornet wrote: > Dear all, > > I am trying to launch a slurm job with pulsar using CLI (instead of drmaa). > The pulsar playbook below pass without problem but the analyses is still run out of slurm. > The analyses is excited with success but not in a slrum job. > > Can you help me to launch slurm with pulsar ? > What did I miss? > > Thanks, > Luc > > > ``` > # Put your Galaxy server's fully qualified domain name (FQDN) (or the FQDN of the RabbitMQ server) above. > > pulsar_root: /opt/pulsar > > pulsar_pip_install: true > pulsar_pycurl_ssl_library: openssl > pulsar_systemd: true > pulsar_systemd_runner: webless > > pulsar_create_user: false > pulsar_user: {name: pulsar, shell: /bin/bash} > > pulsar_optional_dependencies: > - pyOpenSSL > # For remote transfers initiated on the Pulsar end rather than the Galaxy end > - pycurl > # drmaa required if connecting to an external DRM using it. > - drmaa > # kombu needed if using a message queue > - kombu > # amqp 5.0.3 changes behaviour in an unexpected way, pin for now. > - 'amqp==5.0.2' > # psutil and pylockfile are optional dependencies but can make Pulsar > # more robust in small ways. > - psutil > > pulsar_yaml_config: > conda_auto_init: True > conda_auto_install: True > staging_directory: "{{ pulsar_staging_dir }}" > persistence_directory: "{{ pulsar_persistence_dir }}" > tool_dependency_dir: "{{ pulsar_dependencies_dir }}" > # The following are the settings for the pulsar server to contact the message queue with related timeouts etc. > message_queue_url: "pyamqp://galaxy_au:{{ rabbitmq_password_galaxy_au }}@{{ galaxy_server_url }}:5671//pulsar/galaxy_au?ssl=1" > managers: > _default_: > type: queued_cli > job_plugin: slurm > native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" > min_polling_interval: 0.5 > amqp_publish_retry: True > amqp_publish_retry_max_retries: 5 > amqp_publish_retry_interval_start: 10 > amqp_publish_retry_interval_step: 10 > amqp_publish_retry_interval_max: 60 > > # We also need to create the dependency resolver file so pulsar knows how to > # find and install dependencies for the tools we ask it to run. The simplest > # method which covers 99% of the use cases is to use conda auto installs similar > # to how Galaxy works. > pulsar_dependency_resolvers: > - name: conda > args: > - name: auto_init > value: true > ``` > > ------------ > Luc Cornet, PhD > Bio-informatician > Mycology and Aerobiology > Sciensano > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > %(web_page_url)s > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/ -- Gianmauro Cuccuru UseGalaxy.eu Bioinformatics Group Department of Computer Science Albert-Ludwigs-University Freiburg Georges-Köhler-Allee 106 79110 Freiburg, Germany

2 2

Running slurm job with pulsar
by Luc Cornet 17 Jun '21

17 Jun '21

Dear all, I am trying to launch a slurm job with pulsar using CLI (instead of drmaa). The pulsar playbook below pass without problem but the analyses is still run out of slurm. The analyses is excited with success but not in a slrum job. Can you help me to launch slurm with pulsar ? What did I miss? Thanks, Luc ``` # Put your Galaxy server's fully qualified domain name (FQDN) (or the FQDN of the RabbitMQ server) above. pulsar_root: /opt/pulsar pulsar_pip_install: true pulsar_pycurl_ssl_library: openssl pulsar_systemd: true pulsar_systemd_runner: webless pulsar_create_user: false pulsar_user: {name: pulsar, shell: /bin/bash} pulsar_optional_dependencies: - pyOpenSSL # For remote transfers initiated on the Pulsar end rather than the Galaxy end - pycurl # drmaa required if connecting to an external DRM using it. - drmaa # kombu needed if using a message queue - kombu # amqp 5.0.3 changes behaviour in an unexpected way, pin for now. - 'amqp==5.0.2' # psutil and pylockfile are optional dependencies but can make Pulsar # more robust in small ways. - psutil pulsar_yaml_config: conda_auto_init: True conda_auto_install: True staging_directory: "{{ pulsar_staging_dir }}" persistence_directory: "{{ pulsar_persistence_dir }}" tool_dependency_dir: "{{ pulsar_dependencies_dir }}" # The following are the settings for the pulsar server to contact the message queue with related timeouts etc. message_queue_url: "pyamqp://galaxy_au:{{ rabbitmq_password_galaxy_au }}@{{ galaxy_server_url }}:5671//pulsar/galaxy_au?ssl=1" managers: _default_: type: queued_cli job_plugin: slurm native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" min_polling_interval: 0.5 amqp_publish_retry: True amqp_publish_retry_max_retries: 5 amqp_publish_retry_interval_start: 10 amqp_publish_retry_interval_step: 10 amqp_publish_retry_interval_max: 60 # We also need to create the dependency resolver file so pulsar knows how to # find and install dependencies for the tools we ask it to run. The simplest # method which covers 99% of the use cases is to use conda auto installs similar # to how Galaxy works. pulsar_dependency_resolvers: - name: conda args: - name: auto_init value: true ``` ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano

1 0