- galaxy-dev - lists.galaxyproject.org

Running slurm job with pulsar
by Luc Cornet 18 Jun '21

18 Jun '21

Dear all, I am trying to launch a slurm job with pulsar using CLI (instead of drmaa). The pulsar playbook below pass without problem but the analyses is still run out of slurm. The analyses is excited with success but not in a slrum job. Can you help me to launch slurm with pulsar ? What did I miss? Thanks, Luc ``` # Put your Galaxy server's fully qualified domain name (FQDN) (or the FQDN of the RabbitMQ server) above. pulsar_root: /opt/pulsar pulsar_pip_install: true pulsar_pycurl_ssl_library: openssl pulsar_systemd: true pulsar_systemd_runner: webless pulsar_create_user: false pulsar_user: {name: pulsar, shell: /bin/bash} pulsar_optional_dependencies: - pyOpenSSL # For remote transfers initiated on the Pulsar end rather than the Galaxy end - pycurl # drmaa required if connecting to an external DRM using it. - drmaa # kombu needed if using a message queue - kombu # amqp 5.0.3 changes behaviour in an unexpected way, pin for now. - 'amqp==5.0.2' # psutil and pylockfile are optional dependencies but can make Pulsar # more robust in small ways. - psutil pulsar_yaml_config: conda_auto_init: True conda_auto_install: True staging_directory: "{{ pulsar_staging_dir }}" persistence_directory: "{{ pulsar_persistence_dir }}" tool_dependency_dir: "{{ pulsar_dependencies_dir }}" # The following are the settings for the pulsar server to contact the message queue with related timeouts etc. message_queue_url: "pyamqp://galaxy_au:{{ rabbitmq_password_galaxy_au }}@{{ galaxy_server_url }}:5671//pulsar/galaxy_au?ssl=1" managers: _default_: type: queued_cli job_plugin: slurm native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" min_polling_interval: 0.5 amqp_publish_retry: True amqp_publish_retry_max_retries: 5 amqp_publish_retry_interval_start: 10 amqp_publish_retry_interval_step: 10 amqp_publish_retry_interval_max: 60 # We also need to create the dependency resolver file so pulsar knows how to # find and install dependencies for the tools we ask it to run. The simplest # method which covers 99% of the use cases is to use conda auto installs similar # to how Galaxy works. pulsar_dependency_resolvers: - name: conda args: - name: auto_init value: true ``` ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano

1 0

Release of Galaxy 21.05
by Marius van den Beek 11 Jun '21

11 Jun '21

Dear Community, The Galaxy Committers team is pleased to announce the release of Galaxy 21.05. The release announcement for developers and admins can be found at https://docs.galaxyproject.org/en/master/releases/21.05_announce.html and user facing release notes are at https://docs.galaxyproject.org/en/master/releases/21.05_announce_user.html. A few release highlights are: ¡Galaxy, ahora en español! ------------------------------------ Thanks to Wendi Bacon, the Spanish language translation of Galaxy has been finalised and merged, so if you prefer to use Galaxy in Spanish, now you can! This update will be part of an ongoing project from Spanish speakers within the Galaxy community to keep the Galaxy interface localisation up to date, and to produce some Spanish language training materials in the GTN. Bugfixes and Stability ----------------------------- This release of Galaxy features fewer user-facing changes, as a huge amount of developer time went into making this a maintenance release with better testing, better stability, and more bugfixes. But watch out, this is all in preparation for the next release of Galaxy, 21.09, which will have some of the biggest changes in years! New development stack -------------------------------- Galaxy release 21.09 will ship with a new web framework (fastAPI <https://fastapi.tiangolo.com/>), Celery <https://docs.celeryproject.org/en/stable/index.html> task queue and process management using Circus <https://circus.readthedocs.io/en/latest/>. You can preview this new stack now by running *APP_WEBSERVER=dev ./run.sh*. Celery for background tasks -------------------------------------- Galaxy can now run certain tasks in the background. The Celery workers are currently not required, but if activated can perform certain long-running tasks, such as creating history export archives. Celery tasks will bridge the gap between rapid requests that can be handled during a web request and jobs that require extensive and relatively slow setup. More robust selection of job handlers ------------------------------------------------- Job throughput can be increased by starting Galaxy with multiple external job handler processes. Jobs were traditionally assigned to a job handler process by the web handler or workflow handler process that created the job. Since Release 19.01 Galaxy has supported additional mechanisms that use database serialization techniques to let job handlers assign processes to themselves. This mechanism is more robust and doesn't require that all job handler processes be alive and known by the web handler process. Galaxy now determines the best method for assigning jobs based on the database in use, if the assignment method is not set explicitly. Older job assignment methods will be removed in Galaxy release 21.09. For more details see the Job Handler Assignment Methods section <https://docs.galaxyproject.org/en/release_21.05/admin/scaling.html#job-hand…> of the Galaxy documentation. Check out the full release notes <https://docs.galaxyproject.org/en/master/releases/21.05_announce.html> for a lot more details, there are many more enhancements, new visualizations and bugfixes as well as instructions for upgrading your Galaxy installation. Thanks for using Galaxy!

1 0

User Data for Research Purpose
by Shamse Tasnim Cynthia 10 Jun '21

10 Jun '21

Hello, I am Shamse Tasnim Cynthia, currently pursuing my M.Sc in Computer Science at the University of Saskatchewan, Canada, under the supervision of Dr. Banani Roy. My current thesis work involves working on adaptive user interfaces for Software Workflow Management Systems. It has been found that existing SWFMSs do not provide special interaction techniques or visual elements to assist with composing complex workflows and do not consider scientists' preferences based on domain expertise, gender, or other individual preferences. To overcome this, we want to investigate how user preferences will affect modelling in different environments. In this regard, I need user data to analyze user preferences and investigate usage logs to refine the user model. Galaxy, being the most popular SWfMS which is installed on over 168 research servers worldwide, certainly holds a huge amount of user data that can be used in my research. Therefore, it will be very helpful for me if I can get the data to complete my research work and it will be a great opportunity for me to contribute to the Software Workflow developing community. Thank you. Regards Shamse Tasnim Cynthia iSE Lab Department of Computer Science, University of Saskatchewan, Saskatoon, Saskatchewan, Canada. Contact: uji657(a)usask.ca tasnim.cynthia99(a)gmail.com

1 0

Pulsar connection
by Luc Cornet 08 Jun '21

08 Jun '21

Dear Galaxy admins, We have a problem of connection with pulsar on our HPC cluster. We use ansible-galaxy to install the couple galaxy-pulsar, like in the training admin session. The two playbook run with success but when we launch an analysis (simple bwa test), the job remains pending without any error message. The same two playbooks work very well on a virtual machine (centos8) and the bwa test works too. We simply modify the pulsar servers to use ou HPC centos8 instead of the VM and we don't understand why it doesn't work anymore. From the cluster, we can see that there is no connection from the pulsar user. Can you, please, help us to fix that? Thanks Luc ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano

1 0

GCC2021 Deadlines Approaching
by Dave Clements 04 Jun '21

04 Jun '21

Hello all, The early registration deadline for the 2021 Galaxy Community Conference has been moved back to June 8. This gives you more time to take advantage of the 50% discount off full registration prices, so make sure you sign up now so you don't miss out! Register here <https://www.vibconferences.be/events/gcc2021-virtual-edition#attendees>. The draft program <https://gcc2021.sched.com/> and the speakers are now also available online. Please bear in mind that the times are set for the Europe/Middle East/Africa - Asia/Pacific window, and that the same schedule will run 9 hours later for the Americas. The availability of the speakers for the Q&A session will depend on the time zone in which they are located. Posters and demo sessions can still be submitted <https://www.vibconferences.be/events/gcc2021-virtual-edition#abstracts> until the 14th of June. Feel free to distribute this message further in your network. We hope to meet you online! Dave Clements on behalf of the GCC2021 Organizing Committee -- https://galaxyproject.org/

1 0

June 2021 Galaxy Newsletter: All about GCC2021!
by Beatriz Serrano-Solano 31 May '21

31 May '21

Hello all, Welcome to the June 2021 edition of Galaxy News. GCC2021 is upon us, *with deadlines and events starting this month <https://galaxyproject.org/news/2021-06-galaxy-update/#featured-news>*. It's not too late to register (register by June 8 and get 50% off) and to submit Poster/Demo abstracts. Training week starts on June 28. We also started getting confirmations from speakers and are *publishing confirmed presenters <https://galaxyproject.org/news/2021-06-galaxy-update/#featured-news>* as we get them. We are also pleased to *announce our first three sponsors for GCC2021: F1000Research, GigaScience Press and KWS-SAAT <https://galaxyproject.org/news/2021-06-galaxy-update/#gcc2021-sponsorships>* . In addition to GCC news, we also released the *James Taylor birthday video <https://galaxyproject.org/news/2021-06-galaxy-update/#community-news>*, and have several *new open positions <https://galaxyproject.org/news/2021-06-galaxy-update/#community-news>*, lots of *event news <https://galaxyproject.org/news/2021-06-galaxy-update/#event-news>*, *Galaxy platform news <https://galaxyproject.org/news/2021-06-galaxy-update/#galaxy-platforms-news>*, *Galactic blog posts <https://galaxyproject.org/news/2021-06-galaxy-update/#galactic-blog-activity>*, *training and doc updates <https://galaxyproject.org/news/2021-06-galaxy-update/#doc-hub-and-training-…>*, recent *open-access Galaxy-related publications <https://galaxyproject.org/news/2021-06-galaxy-update/#publications>*, and *new releases <https://galaxyproject.org/news/2021-06-galaxy-update/#releases>.* As always, thanks for using Galaxy, Dave Clements and Beatriz Serrano-Solano, Editors PS: Subscribe to the Galaxy Announce mailing list <https://lists.galaxyproject.org/?count=100&all-lists=> to receive an email whenever this newsletter is published. -- Dr Beatriz Serrano-Solano (she/her) <https://www.hrc.org/resources/talking-about-pronouns-in-the-workplace> Bioinformatics Group Dept. Computer Science Albert-Ludwigs-Universität Freiburg Georges-Köhler-Allee, 106 70110 Freiburg Germany

1 0

Webinar, Wednesday May 26: Galaxy Resources for Admins and Infrastructure Providers
by Dave Clements 26 May '21

26 May '21

Hello all, We are closing the webinar series *"Galaxy Resources for..."* tomorrow with > Gianmauro Cuccuru and Lucille Delisle. They will talk about the relevant > resources for *Galaxy admins and infrastructure providers*. Please feel > free to register and share this link: > > https://galaxyproject.org/events/2021-05-gr4-admins/ > We hope to see you there, Dave "Blatantly copying Beatriz Serrano-Solano who reminded me to post this here" C PS: Also, some GCC2021 news. The early registration deadline for GCC2021 is Monday, June 1. We may not have a full talk schedule published by then, but we do hope to have published a long list of confirmed topics and presenters by then. Watch this space <https://gcc2021.sched.com/>. -- https://galaxyproject.org/

1 0

rabbitmq erlang-crypto
by Luc Cornet 18 May '21

18 May '21

Dear all, I have a strange error with rabbitmq on ubuntu 20.04. The playbook was working previously on this OS but now I have this error: TASK [usegalaxy_eu.rabbitmq : install rabbitmq] **************************************************************************************************************************** task path: /home/galaxyluc/galaxy/roles/usegalaxy_eu.rabbitmq/tasks/Debian/install.yml:23 <galaxy.inbios.uliege.be> ESTABLISH LOCAL CONNECTION FOR USER: root <galaxy.inbios.uliege.be> EXEC /bin/sh -c 'echo ~root && sleep 0' <galaxy.inbios.uliege.be> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1620999798.916904-12818219648644 `" && echo ansible-tmp-1620999798.916904-12818219648644="` echo /root/.ansible/tmp/ansible-tmp-1620999798.916904-12818219648644 `" ) && sleep 0' Using module file /usr/lib/python3/dist-packages/ansible/modules/packaging/os/apt.py <galaxy.inbios.uliege.be> PUT /root/.ansible/tmp/ansible-local-1239270zkwwo8m/tmpd5o4vaj2 TO /root/.ansible/tmp/ansible-tmp-1620999798.916904-12818219648644/AnsiballZ_apt.py <galaxy.inbios.uliege.be> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1620999798.916904-12818219648644/ /root/.ansible/tmp/ansible-tmp-1620999798.916904-12818219648644/AnsiballZ_apt.py && sleep 0' <galaxy.inbios.uliege.be> EXEC /bin/sh -c '/usr/bin/python3 /root/.ansible/tmp/ansible-tmp-1620999798.916904-12818219648644/AnsiballZ_apt.py && sleep 0' <galaxy.inbios.uliege.be> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1620999798.916904-12818219648644/ > /dev/null 2>&1 && sleep 0' fatal: [galaxy.inbios.uliege.be]: FAILED! => { "changed": false, "invocation": { "module_args": { "allow_unauthenticated": false, "autoclean": false, "autoremove": false, "cache_valid_time": 0, "deb": "/usr/src/rabbitmq-server-3.8.9-1.deb", "default_release": null, "dpkg_options": "force-confdef,force-confold", "force": false, "force_apt_get": false, "install_recommends": null, "only_upgrade": false, "package": null, "policy_rc_d": null, "purge": false, "state": "present", "update_cache": null, "upgrade": null } }, "msg": "Dependency is not satisfiable: erlang-crypto (< 1:24.0)|esl-erlang (< 1:24.0)\n" } Does anyone have any idea how to solve this ? Thanks, Luc ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano

2 4

cyverse-ansible.singularity
by Luc Cornet 10 May '21

10 May '21

Hello all, I have a problem with singularity while installing galaxy with ansible. ``` ASK [cyverse-ansible.singularity : Ubuntu package installation for v3.0 or newer] ************************************************************************* ok: [galaxy.inbios.uliege.be] TASK [cyverse-ansible.singularity : CentOS package installation for v3.0 or newer] ************************************************************************* skipping: [galaxy.inbios.uliege.be] TASK [cyverse-ansible.singularity : check if singularity binary exists] ************************************************************************************ ok: [galaxy.inbios.uliege.be] TASK [cyverse-ansible.singularity : Obtain current binary version] ***************************************************************************************** skipping: [galaxy.inbios.uliege.be] TASK [cyverse-ansible.singularity : download the singularity release] ************************************************************************************** fatal: [galaxy.inbios.uliege.be]: FAILED! => {"changed": false, "dest": "/tmp/singularity-3.7.0.tar.gz", "elapsed": 5, "msg": "Request failed", "response": "HTTP Error 404: Not Found", "status_code": 404, "url": "https://github.com/sylabs/singularity/releases/download/v3.7.0/singularity-…"} ``` The URL for singularity has changed, Do you know how can i fix that ? Maybe a more recent version of : ``` - src: cyverse-ansible.singularity version: 048c4f178077d05c1e67ae8d9893809aac9ab3b7 ``` Thanks for the help, Luc ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano

2 2

May 2021 Galaxy Newsletter: GCC2021 Deadlines Extended
by Dave Clements 05 May '21

05 May '21

Hello all, *The May 2021 edition of Galaxy News <https://galaxyproject.org/news/2021-05-galaxy-update/> is out! * The big news is that *several deadlines for the 2021 Galaxy Community Conference <https://galaxyproject.org/news/2021-05-galaxy-update/#community-news> have been extended, and the deadline for submitting James Taylor videos <https://galaxyproject.org/news/2021-05-galaxy-update/#featured-news> has been extended to this Friday.* And there are other upcoming deadlines too: - *ISMB/ECCB and BOSC 2021 abstracts <https://galaxyproject.org/news/2021-05-galaxy-update/#event-news> are due this Thursday.* - *Apply for the Galaxy Australia Variant Calling Workshops <https://galaxyproject.org/news/2021-05-galaxy-update/#event-news> by May 12.* We also have - *open positions <https://galaxyproject.org/news/2021-05-galaxy-update/#community-news>* - *webinars, CoFests, workshops, community meetups, and other events <https://galaxyproject.org/news/2021-05-galaxy-update/#event-news>*, - news about *public Galaxy platforms <https://galaxyproject.org/news/2021-05-galaxy-update/#galaxy-platforms-news>* , - recent *blog posts <https://galaxyproject.org/news/2021-05-galaxy-update/#galactic-blog-activity>* , - *training and doc updates <https://galaxyproject.org/news/2021-05-galaxy-update/#doc-hub-and-training-…>* , - recent *open-access Galaxy-related publications <https://galaxyproject.org/news/2021-05-galaxy-update/#publications>*, and - *new releases <https://galaxyproject.org/news/2021-05-galaxy-update/#releases>.* As always, thanks for using Galaxy, Dave Clements and Beatriz Serrano-Solano, Editors -- https://galaxyproject.org/

1 0