galaxy-dev

galaxy-dev@lists.galaxyproject.org

1 participants
10006 discussions

September 2021 News: Papercuts resumes; Misconceptions; Hub updated; microGalaxy Community, ...
by Dave Clements 15 Sep '21

15 Sep '21

Hi all, The September newsletter <https://galaxyproject.org/news/2021-09-galaxy-update/> brings lots of news, including - 12 recent use cases for UseGalaxy.eu, covering a broad spectrum of research. - Want to help address misconceptions about Galaxy? - The Galaxy Community Hub is now using Gridsome. - ELIXIR community updates - JXTX Foundation 2021 Awards - 7 open positions in Belgium, Italy, and the US - The Galaxy YouTube channel is growing up - Upcoming events - Galaxy Platforms News - Recent Blog Posts - Doc, Hub, and Training Updates - Recent Publications - and other news too. Sit back, get comfortable, and explore the Galaxy... Beatriz Serrano-Solano and Dave Clements, Editors -- https://galaxyproject.org/

1 0

PaperCuts is Resuming this Thursday, Septermber 16
by Dave Clements 15 Sep '21

15 Sep '21

Hello all, After taking the last 3 months off, PaperCuts is back <https://galaxyproject.org/events/2021-09-papercuts/>. Galaxy PaperCuts is a monthly online gathering where we address irritating but relatively easy to fix bugs in anything related to Galaxy. This can be code, documentation, training materials, Ansible recipes, and do on. We will be on Gitter <https://gitter.im/galaxyproject/Lobby> for chat all day long, and on 3 calls spread across the day. Please take advantage of both to communicate with your collaborators around the world. - *Call 1: Oceania, Australia, Asia* - 14:00 Australia Eastern time. See your time <https://www.timeanddate.com/worldclock/fixedtime.html?msg=14%3A00+Melbourne…> . - Zoom Link <https://zoom.us/j/98189403566?pwd=SWhueXkzNHlKSG1QMU0xUG1ESFFsUT09> - *Call 2: Middle East, Europe, Africa* - 10:00 Central European Summer time. See your time <https://www.timeanddate.com/worldclock/fixedtime.html?msg=10%3A00+Freiburg+…> . - Zoom Link <https://us02web.zoom.us/j/89108992104?pwd=eHhWdFFUQThnVVFwZ3RRall4WjBXQT09> - *Call 3: Americas* - 12:00 US Eastern time. See your time <https://www.timeanddate.com/worldclock/fixedtime.html?msg=12%3A00+Penn+Stat…> . - Zoom Link <https://zoom.us/j/98189403566?pwd=SWhueXkzNHlKSG1QMU0xUG1ESFFsUT09> If you have any ideas for papercut issues, please add them to the appropriate repo. Thanks, and hope to see you soon, Dave C -- https://galaxyproject.org/

1 0

Galaxy Newsletter: August 2021
by Beatriz Serrano-Solano 05 Aug '21

05 Aug '21

Hello all, The August newsletter brings news of GCC2021 (just finished) and GCC2022 (in Minneapolis); the launch of the Galaxy India community, and open positions in 7 countries <https://galaxyproject.org/news/2021-08-galaxy-update/#community-news>; event news <https://galaxyproject.org/news/2021-08-galaxy-update/#event-news>; Galaxy platform news <https://galaxyproject.org/news/2021-08-galaxy-update/#galaxy-platforms-news>; Galactic blog posts <https://galaxyproject.org/news/2021-08-galaxy-update/#galactic-blog-activity>; training and doc updates <https://galaxyproject.org/news/2021-08-galaxy-update/#doc-hub-and-training-…>; recent open-access Galaxy-related publications <https://galaxyproject.org/news/2021-08-galaxy-update/#publications>; and other news too <https://galaxyproject.org/news/2021-08-galaxy-update/#other-news>.** Dave Clements and Beatriz Serrano-Solano, Editors PS: Subscribe to the Galaxy Announce mailing list <https://lists.galaxyproject.org/?count=100&all-lists=> and receive an email whenever this newsletter is published. -- Dr Beatriz Serrano-Solano (she/her) <https://www.hrc.org/resources/talking-about-pronouns-in-the-workplace> Bioinformatics Group Dept. Computer Science Albert-Ludwigs-Universität Freiburg Georges-Köhler-Allee, 106 79110 Freiburg Germany

1 0

Galaxy install problems
by Luc Cornet 09 Jul '21

09 Jul '21

Dear Galaxy admins, I am contacting you due to multiple problems during the installation of our own server of galaxy. We have been struggling to install galaxy on our HPC system for 1.5 year. We have faced so many problems while installing galaxy that we have lost count. Yet, our main problem was and remains the use of pulsar... Even after following two training sessions, switched to a brand new CentOS 8 cluster (while CentOS 8 will die at the end of the year), we are still not able to use our galaxy instance. We followed the ansible documentation and we are now able to reproduce the training sessions (i.e, execute an analysis with pulsar on our HPC system). Nevertheless, we are not able to connect pulsar with slurm (which makes it unusable). We do not want to use DRMAA because it is not well maintained and compatible with slurm. Instead, we choose to use CLI (as mentioned in the galaxy docs) but we are stuck (see below the pulsarservers.yaml file). Is it possible to use something else than DRMAA to connect pulsar to the scheduler? Thanks, Best regards Luc Cornet - - pulsarservers.yaml - - # Put your Galaxy server's fully qualified domain name (FQDN) (or the FQDN of the RabbitMQ server) above. pulsar_root: /opt/pulsar pulsar_pip_install: true pulsar_pycurl_ssl_library: openssl pulsar_systemd: true pulsar_systemd_runner: webless pulsar_create_user: false pulsar_user: {name: pulsar, shell: /bin/bash} pulsar_optional_dependencies: - pyOpenSSL # For remote transfers initiated on the Pulsar end rather than the Galaxy end - pycurl # drmaa required if connecting to an external DRM using it. - drmaa # kombu needed if using a message queue - kombu # amqp 5.0.3 changes behaviour in an unexpected way, pin for now. - 'amqp==5.0.2' # psutil and pylockfile are optional dependencies but can make Pulsar # more robust in small ways. - psutil pulsar_yaml_config: conda_auto_init: True conda_auto_install: True staging_directory: "{{ pulsar_staging_dir }}" persistence_directory: "{{ pulsar_persistence_dir }}" tool_dependency_dir: "{{ pulsar_dependencies_dir }}" # The following are the settings for the pulsar server to contact the message queue with related timeouts etc. message_queue_url: "pyamqp://galaxy_au:{{ rabbitmq_password_galaxy_au }}@{{ galaxy_server_url }}:5671//pulsar/galaxy_au?ssl=1" managers: _default_: type: queued_cli job_plugin: slurm native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" min_polling_interval: 0.5 amqp_publish_retry: True amqp_publish_retry_max_retries: 5 amqp_publish_retry_interval_start: 10 amqp_publish_retry_interval_step: 10 amqp_publish_retry_interval_max: 60 # We also need to create the dependency resolver file so pulsar knows how to # find and install dependencies for the tools we ask it to run. The simplest # method which covers 99% of the use cases is to use conda auto installs similar # to how Galaxy works. pulsar_dependency_resolvers: - name: conda args: - name: auto_init value: true ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano

4 16

Galaxy Newsletter: July 2021
by Dave Clements 01 Jul '21

01 Jul '21

Hello all, The big news for July is that the 2021 Galaxy Community Conference <https://gcc2021.sched.com/> is under way. With GCC going on, this is one of our thinner newsletters <https://galaxyproject.org/news/2021-07-galaxy-update/>, but there are still *open positions <https://galaxyproject.org/news/2021-07-galaxy-update/#community-news>*, *event news <https://galaxyproject.org/news/2021-07-galaxy-update/#event-news>*, *Galaxy platform news <https://galaxyproject.org/news/2021-07-galaxy-update/#galaxy-platforms-news>*, *Galactic blog posts <https://galaxyproject.org/news/2021-07-galaxy-update/#galactic-blog-activity>*, *training and doc updates <https://galaxyproject.org/news/2021-07-galaxy-update/#doc-hub-and-training-…>*, recent *open-access Galaxy-related publications <https://galaxyproject.org/news/2021-07-galaxy-update/#publications>*, *new releases <https://galaxyproject.org/news/2021-07-galaxy-update/#releases>, and other news too <https://galaxyproject.org/news/2021-07-galaxy-update/#other-news>.* We hope to see you at GCC events this week and next. Dave Clements and Beatriz Serrano-Solano, Editors PS: It is too late to register for the 3 day conference next week. It is also officially too late to sign up for the CoFest on 9-10 July, but if you follow this link <https://docs.google.com/forms/d/e/1FAIpQLScdJHTzw7LWVxFaqGI30COZlcxGPksjGL_…>, you just mind find yourself invited anyway. -- https://gaaxyproject.org/ <https://galaxyproject.org/>

1 0

GCC202 Registration Ends Tomorrow
by Beatriz Serrano-Solano 24 Jun '21

24 Jun '21

Hello All, The final registration for the 2021 Galaxy Community Conference is closing tomorrow, Friday 25th of June, at midnight CEST, so you have still some time to register! https://www.vibconferences.be/events/gcc2021-virtual-edition We will meet in Remo (https://remo.co/) and the schedule is already available at https://gcc2021.sched.com/ We are looking forward to seeing you online! Bea on behalf of the GCC2021 Organizing Committee -- Dr Beatriz Serrano-Solano (she/her) Bioinformatics Group Dept. Computer Science Albert-Ludwigs-Universität Freiburg Georges-Köhler-Allee, 106 79110 Freiburg Germany

1 0

Re: Running slurm job with pulsar
by Luc Cornet 18 Jun '21

18 Jun '21

Dear Gianmauro, Thanks for your answer. I think that my question was not clear enough. Below you will find some answer to your suggestions and maybe it will be more clear for you to guide me. * check the Pulsar' log for error messages ->I look into staging directory but as my test analysis is executed with success, I have no error log. When a lunch a test from the galaxy GUI, it is executed with success on the cluster. My problem is that is not executed in a job, not using srun or sbatch command (so not using the scheduler). I would like pulsar to be able to submit a job on the cluster, just like others users, and not execute analysis directly "in the terminal". * verify if your Pulsar server can reach the cluster trough ssh ->The pulsar server is on the HPC cluster. The connection between galaxy (rabbitmq) and the cluster (pulsar) is fine. * in the staging directory of your job should be a command.sh file. You can try to run it manually (sbatch command.sh or something similar) and Yes indeed I have this file in staging directory (see below). We i execute the command.sh file with sbatch command.sh, it fails immediately which is normal since command.sh is not a slurm job. total 48 -rw-r--r-- 1 pulsar pulsar 4 Jun 17 14:15 use_metadata_directory -rw-r--r-- 1 pulsar pulsar 10 Jun 17 14:15 tool_version -rw-r--r-- 1 pulsar pulsar 59 Jun 17 14:15 tool_id drwxr-xr-x 2 pulsar pulsar 6 Jun 17 14:15 tool_files drwxr-xr-x 2 pulsar pulsar 6 Jun 17 14:15 metadata drwxr-xr-x 2 pulsar pulsar 6 Jun 17 14:15 configs drwxr-xr-x 5 pulsar pulsar 30 Jun 17 14:15 .. -rw-r--r-- 1 pulsar pulsar 2551 Jun 17 14:15 launch_config drwxr-xr-x 2 pulsar pulsar 46 Jun 17 14:15 inputs -rw-r--r-- 1 pulsar pulsar 4 Jun 17 14:15 preprocessed -rwx------ 1 pulsar pulsar 5441 Jun 17 14:15 command.sh -rw-r--r-- 1 pulsar pulsar 0 Jun 17 14:15 stdout drwxr-xr-x 2 pulsar pulsar 6 Jun 17 14:15 home drwxr-xr-x 2 pulsar pulsar 58 Jun 17 14:15 working -rw-r--r-- 1 pulsar pulsar 4 Jun 17 14:15 running -rw-r--r-- 1 pulsar pulsar 546 Jun 17 14:15 stderr drwxr-xr-x 2 pulsar pulsar 26 Jun 17 14:15 outputs -rw-r--r-- 1 pulsar pulsar 1 Jun 17 14:15 return_code -rw-r--r-- 1 pulsar pulsar 10 Jun 17 14:15 final_status -rw-r--r-- 1 pulsar pulsar 0 Jun 17 14:15 postprocessed drwxr-xr-x 9 pulsar pulsar 4096 Jun 17 14:15 . ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano ----- Mail original ----- De: "Gianmauro Cuccuru" <gmauro(a)informatik.uni-freiburg.de> À: "Luc Cornet" <luc.cornet(a)uliege.be>, "HelpGalaxy" <galaxy-dev(a)lists.galaxyproject.org> Cc: "Colignon David" <David.Colignon(a)uliege.be>, "Baurain Denis" <Denis.Baurain(a)uliege.be>, "Pierre Becker" <Pierre.Becker(a)sciensano.be> Envoyé: Vendredi 18 Juin 2021 11:48:30 Objet: Re: [galaxy-dev] Running slurm job with pulsar Hi Luc, I am not a Slurm expert but I can suggest you several things: * check the Pulsar' log for error messages * verify if your Pulsar server can reach the cluster trough ssh * in the staging directory of your job should be a command.sh file. You can try to run it manually (sbatch command.sh or something similar) and see if it works Cheers, Gianmauro On 17.06.21 20:18, Luc Cornet wrote: > Dear all, > > I am trying to launch a slurm job with pulsar using CLI (instead of drmaa). > The pulsar playbook below pass without problem but the analyses is still run out of slurm. > The analyses is excited with success but not in a slrum job. > > Can you help me to launch slurm with pulsar ? > What did I miss? > > Thanks, > Luc > > > ``` > # Put your Galaxy server's fully qualified domain name (FQDN) (or the FQDN of the RabbitMQ server) above. > > pulsar_root: /opt/pulsar > > pulsar_pip_install: true > pulsar_pycurl_ssl_library: openssl > pulsar_systemd: true > pulsar_systemd_runner: webless > > pulsar_create_user: false > pulsar_user: {name: pulsar, shell: /bin/bash} > > pulsar_optional_dependencies: > - pyOpenSSL > # For remote transfers initiated on the Pulsar end rather than the Galaxy end > - pycurl > # drmaa required if connecting to an external DRM using it. > - drmaa > # kombu needed if using a message queue > - kombu > # amqp 5.0.3 changes behaviour in an unexpected way, pin for now. > - 'amqp==5.0.2' > # psutil and pylockfile are optional dependencies but can make Pulsar > # more robust in small ways. > - psutil > > pulsar_yaml_config: > conda_auto_init: True > conda_auto_install: True > staging_directory: "{{ pulsar_staging_dir }}" > persistence_directory: "{{ pulsar_persistence_dir }}" > tool_dependency_dir: "{{ pulsar_dependencies_dir }}" > # The following are the settings for the pulsar server to contact the message queue with related timeouts etc. > message_queue_url: "pyamqp://galaxy_au:{{ rabbitmq_password_galaxy_au }}@{{ galaxy_server_url }}:5671//pulsar/galaxy_au?ssl=1" > managers: > _default_: > type: queued_cli > job_plugin: slurm > native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" > min_polling_interval: 0.5 > amqp_publish_retry: True > amqp_publish_retry_max_retries: 5 > amqp_publish_retry_interval_start: 10 > amqp_publish_retry_interval_step: 10 > amqp_publish_retry_interval_max: 60 > > # We also need to create the dependency resolver file so pulsar knows how to > # find and install dependencies for the tools we ask it to run. The simplest > # method which covers 99% of the use cases is to use conda auto installs similar > # to how Galaxy works. > pulsar_dependency_resolvers: > - name: conda > args: > - name: auto_init > value: true > ``` > > ------------ > Luc Cornet, PhD > Bio-informatician > Mycology and Aerobiology > Sciensano > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > %(web_page_url)s > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/ -- Gianmauro Cuccuru UseGalaxy.eu Bioinformatics Group Department of Computer Science Albert-Ludwigs-University Freiburg Georges-Köhler-Allee 106 79110 Freiburg, Germany

2 2

Running slurm job with pulsar
by Luc Cornet 17 Jun '21

17 Jun '21

Dear all, I am trying to launch a slurm job with pulsar using CLI (instead of drmaa). The pulsar playbook below pass without problem but the analyses is still run out of slurm. The analyses is excited with success but not in a slrum job. Can you help me to launch slurm with pulsar ? What did I miss? Thanks, Luc ``` # Put your Galaxy server's fully qualified domain name (FQDN) (or the FQDN of the RabbitMQ server) above. pulsar_root: /opt/pulsar pulsar_pip_install: true pulsar_pycurl_ssl_library: openssl pulsar_systemd: true pulsar_systemd_runner: webless pulsar_create_user: false pulsar_user: {name: pulsar, shell: /bin/bash} pulsar_optional_dependencies: - pyOpenSSL # For remote transfers initiated on the Pulsar end rather than the Galaxy end - pycurl # drmaa required if connecting to an external DRM using it. - drmaa # kombu needed if using a message queue - kombu # amqp 5.0.3 changes behaviour in an unexpected way, pin for now. - 'amqp==5.0.2' # psutil and pylockfile are optional dependencies but can make Pulsar # more robust in small ways. - psutil pulsar_yaml_config: conda_auto_init: True conda_auto_install: True staging_directory: "{{ pulsar_staging_dir }}" persistence_directory: "{{ pulsar_persistence_dir }}" tool_dependency_dir: "{{ pulsar_dependencies_dir }}" # The following are the settings for the pulsar server to contact the message queue with related timeouts etc. message_queue_url: "pyamqp://galaxy_au:{{ rabbitmq_password_galaxy_au }}@{{ galaxy_server_url }}:5671//pulsar/galaxy_au?ssl=1" managers: _default_: type: queued_cli job_plugin: slurm native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" min_polling_interval: 0.5 amqp_publish_retry: True amqp_publish_retry_max_retries: 5 amqp_publish_retry_interval_start: 10 amqp_publish_retry_interval_step: 10 amqp_publish_retry_interval_max: 60 # We also need to create the dependency resolver file so pulsar knows how to # find and install dependencies for the tools we ask it to run. The simplest # method which covers 99% of the use cases is to use conda auto installs similar # to how Galaxy works. pulsar_dependency_resolvers: - name: conda args: - name: auto_init value: true ``` ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano

1 0

Release of Galaxy 21.05
by Marius van den Beek 10 Jun '21

10 Jun '21

Dear Community, The Galaxy Committers team is pleased to announce the release of Galaxy 21.05. The release announcement for developers and admins can be found at https://docs.galaxyproject.org/en/master/releases/21.05_announce.html and user facing release notes are at https://docs.galaxyproject.org/en/master/releases/21.05_announce_user.html. A few release highlights are: ¡Galaxy, ahora en español! ------------------------------------ Thanks to Wendi Bacon, the Spanish language translation of Galaxy has been finalised and merged, so if you prefer to use Galaxy in Spanish, now you can! This update will be part of an ongoing project from Spanish speakers within the Galaxy community to keep the Galaxy interface localisation up to date, and to produce some Spanish language training materials in the GTN. Bugfixes and Stability ----------------------------- This release of Galaxy features fewer user-facing changes, as a huge amount of developer time went into making this a maintenance release with better testing, better stability, and more bugfixes. But watch out, this is all in preparation for the next release of Galaxy, 21.09, which will have some of the biggest changes in years! New development stack -------------------------------- Galaxy release 21.09 will ship with a new web framework (fastAPI <https://fastapi.tiangolo.com/>), Celery <https://docs.celeryproject.org/en/stable/index.html> task queue and process management using Circus <https://circus.readthedocs.io/en/latest/>. You can preview this new stack now by running *APP_WEBSERVER=dev ./run.sh*. Celery for background tasks -------------------------------------- Galaxy can now run certain tasks in the background. The Celery workers are currently not required, but if activated can perform certain long-running tasks, such as creating history export archives. Celery tasks will bridge the gap between rapid requests that can be handled during a web request and jobs that require extensive and relatively slow setup. More robust selection of job handlers ------------------------------------------------- Job throughput can be increased by starting Galaxy with multiple external job handler processes. Jobs were traditionally assigned to a job handler process by the web handler or workflow handler process that created the job. Since Release 19.01 Galaxy has supported additional mechanisms that use database serialization techniques to let job handlers assign processes to themselves. This mechanism is more robust and doesn't require that all job handler processes be alive and known by the web handler process. Galaxy now determines the best method for assigning jobs based on the database in use, if the assignment method is not set explicitly. Older job assignment methods will be removed in Galaxy release 21.09. For more details see the Job Handler Assignment Methods section <https://docs.galaxyproject.org/en/release_21.05/admin/scaling.html#job-hand…> of the Galaxy documentation. Check out the full release notes <https://docs.galaxyproject.org/en/master/releases/21.05_announce.html> for a lot more details, there are many more enhancements, new visualizations and bugfixes as well as instructions for upgrading your Galaxy installation. Thanks for using Galaxy!

1 0

User Data for Research Purpose
by Shamse Tasnim Cynthia 09 Jun '21

09 Jun '21

Hello, I am Shamse Tasnim Cynthia, currently pursuing my M.Sc in Computer Science at the University of Saskatchewan, Canada, under the supervision of Dr. Banani Roy. My current thesis work involves working on adaptive user interfaces for Software Workflow Management Systems. It has been found that existing SWFMSs do not provide special interaction techniques or visual elements to assist with composing complex workflows and do not consider scientists' preferences based on domain expertise, gender, or other individual preferences. To overcome this, we want to investigate how user preferences will affect modelling in different environments. In this regard, I need user data to analyze user preferences and investigate usage logs to refine the user model. Galaxy, being the most popular SWfMS which is installed on over 168 research servers worldwide, certainly holds a huge amount of user data that can be used in my research. Therefore, it will be very helpful for me if I can get the data to complete my research work and it will be a great opportunity for me to contribute to the Software Workflow developing community. Thank you. Regards Shamse Tasnim Cynthia iSE Lab Department of Computer Science, University of Saskatchewan, Saskatoon, Saskatchewan, Canada. Contact: uji657(a)usask.ca tasnim.cynthia99(a)gmail.com

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

galaxy-dev