- galaxy-dev - lists.galaxyproject.org

Release of Galaxy 21.09
by Marius van den Beek 02 Nov '21

02 Nov '21

Dear Community, The Galaxy Committers team is pleased to announce the release of Galaxy 21.09. - Developer and admin release announcement <https://docs.galaxyproject.org/en/latest/releases/21.09_announce.html> - User release announcement <https://docs.galaxyproject.org/en/latest/releases/21.09_announce_user.html> . A few release highlights are: Tool Panel Views We now have "Tool Panel Views"! These are different views into the same toolbox and might help make it easier to find the tools you want. In the future, there are plans to create user-customisable toolboxes, but until then go explore the new EDAM Ontology-based toolboxes which organise tools around scientific areas or processes. For example, a category like "Filtering" might have tools like "select lines" or "filter bam by quality", both doing the same process of filtering, despite their different file types and formats. Unfortunately, not all tools are fully annotated yet, and while we also pull terms from bio.tools, this still doesn't get us full coverage and many tools will still appear under a large section "Uncategorized". Hopefully, this will improve as users and tool developers annotate these tools with the appropriate terms that help everyone find them. Collections (Beta History Updates) Did you ever create *a collection with the wrong dbkey or datatype*? Well, in the Beta History you can now change collections' datatypes and dbkeys! This should save a lot of time for everyone working with large datasets (Pull Request 11799 <https://github.com/galaxyproject/galaxy/pull/11799/>). Additionally, did a *collection ever fail* for you? And you wondered why? Now you can find out with the view details button for collections (Pull Request 12261 <https://github.com/galaxyproject/galaxy/pull/12261>)! Importing Data Selecting datasets from the Remote Files view has gotten easier! Now, you can select folders and files and import at once all of the datasets recursively in those folders. Previously, you could only select files within a folder, so this is a huge improvement in the usability of such a key new feature of Galaxy (Pull Request 12310 <https://github.com/galaxyproject/galaxy/pull/12310>). But with great power comes great responsibility, be careful not to import the entirety of NCBI! Reports Report components used to automatically arrange themselves in a smart way, but due to some issues with the floating of components, these will now be full width until we can figure out a good way to provide a similar feature (Pull Request 12085 <https://github.com/galaxyproject/galaxy/pull/12085>). Enhancements for Working with Remote Data and Distributed Computing Resources Many fixes and enhancements were made to improve how Galaxy can import and write to remote data using the configured file source plugins. In addition to the History Export functionality, which can write to remote locations, we have added an Export datasets Tool for exporting individual Datasets and Dataset collections to configured remote locations. The Tool will automatically maintain the name and structure of Datasets and Dataset Collections. We have added the possibility to import entire folders of remote data and made the data selection dialog more intuitive. We also added numerous improvements and fixes to the extended metadata collection strategy and Pulsar <https://github.com/galaxyproject/pulsar>, so that many more tools are able to run in Pulsar <https://github.com/galaxyproject/pulsar> and write data back to storage without passing through Galaxy first. Migration to FastAPI and Extended Documentation of API Routes We have modernized and migrated many more API routes to FastAPI and have extended the documentation and validation of these routes. Migration to SQLAlchemy 1.4 and Declarative Mapping Galaxy is now using SQLAlchemy <https://www.sqlalchemy.org/> version 1.4, which is a prerequisite for asynchronously interacting with the database. We also modernized the way our database models are defined to the more commonly used Declarative Mapping approach, which is more concise and better documented. Modernization of Tool Form Interface The tool form interface has been almost completely migrated from Backbone <https://backbonejs.org/> to Vue <https://vuejs.org/>. This improves the reactivity of parameter validation and enables the migration of tool parameters to Vue, which will also allow us to add many more types of tool parameters. We further migrated the tool form variant for the Workflow Editor, the Workflow run form and the Show Dataset Parameter page to Vue. New User Welcome Page We have added a page that new users will be directed to after creating a new Galaxy account. Release Notes Please see the full release notes <https://docs.galaxyproject.org/en/latest/releases/21.09_announce.html> for a lot more details and instructions for upgrading your Galaxy installation. Thanks for using Galaxy!

1 0

November 2021 Galaxy Newsletter
by Dave Clements 02 Nov '21

02 Nov '21

Happy November from Galaxy! The November newsletter is thin, *but powerful*: - Galaxy India Launches this Month <https://galaxyproject.org/news/2021-11-galaxy-update/#community-news> - 7 new openings (13 total) <https://galaxyproject.org/news/2021-11-galaxy-update/#community-news> - 9 upcoming Events <https://galaxyproject.org/news/2021-11-galaxy-update/#event-news> - Blog posts on SARS-CoV-2 training in Africa; Sunsetting CloudLaundh; and TIaaS <https://galaxyproject.org/news/2021-11-galaxy-update/#galactic-blog-posts> - UseGalaxy.* Updates <https://galaxyproject.org/news/2021-11-galaxy-update/#galaxy-platforms-news> - Other News too: BY-COVID Launches <https://galaxyproject.org/news/2021-11-galaxy-update/#other-news> Have something for the next newsletter? Please let us know and we will spread the word. Beatriz Serrano-Solano and Dave Clements, Editors -- https://galaxyproject.org/

1 0

October Galaxy News
by Dave Clements 11 Oct '21

11 Oct '21

Hello all, The October Newsletter is out <https://galaxyproject.org/news/2021-10-galaxy-update/>. and it covers Galaxy involvement in Hacktoberfest, and Outreachy, plus a nice batch of trainings, talks and a Galaxy Papercuts CoFest day too. It also brings news of new job openings on two continents, two new platforms, blog posts (where My Little Pony makes an appearance, really), GTN updates (few in number but big in their coolness), publications, and releases. In other words, another active month in the Galaxy. Beatriz Serrano-Solano and Dave Clements, Editors -- https://galaxyproject.org/

1 0

September 2021 News: Papercuts resumes; Misconceptions; Hub updated; microGalaxy Community, ...
by Dave Clements 16 Sep '21

16 Sep '21

Hi all, The September newsletter <https://galaxyproject.org/news/2021-09-galaxy-update/> brings lots of news, including - 12 recent use cases for UseGalaxy.eu, covering a broad spectrum of research. - Want to help address misconceptions about Galaxy? - The Galaxy Community Hub is now using Gridsome. - ELIXIR community updates - JXTX Foundation 2021 Awards - 7 open positions in Belgium, Italy, and the US - The Galaxy YouTube channel is growing up - Upcoming events - Galaxy Platforms News - Recent Blog Posts - Doc, Hub, and Training Updates - Recent Publications - and other news too. Sit back, get comfortable, and explore the Galaxy... Beatriz Serrano-Solano and Dave Clements, Editors -- https://galaxyproject.org/

1 0

PaperCuts is Resuming this Thursday, Septermber 16
by Dave Clements 16 Sep '21

16 Sep '21

Hello all, After taking the last 3 months off, PaperCuts is back <https://galaxyproject.org/events/2021-09-papercuts/>. Galaxy PaperCuts is a monthly online gathering where we address irritating but relatively easy to fix bugs in anything related to Galaxy. This can be code, documentation, training materials, Ansible recipes, and do on. We will be on Gitter <https://gitter.im/galaxyproject/Lobby> for chat all day long, and on 3 calls spread across the day. Please take advantage of both to communicate with your collaborators around the world. - *Call 1: Oceania, Australia, Asia* - 14:00 Australia Eastern time. See your time <https://www.timeanddate.com/worldclock/fixedtime.html?msg=14%3A00+Melbourne…> . - Zoom Link <https://zoom.us/j/98189403566?pwd=SWhueXkzNHlKSG1QMU0xUG1ESFFsUT09> - *Call 2: Middle East, Europe, Africa* - 10:00 Central European Summer time. See your time <https://www.timeanddate.com/worldclock/fixedtime.html?msg=10%3A00+Freiburg+…> . - Zoom Link <https://us02web.zoom.us/j/89108992104?pwd=eHhWdFFUQThnVVFwZ3RRall4WjBXQT09> - *Call 3: Americas* - 12:00 US Eastern time. See your time <https://www.timeanddate.com/worldclock/fixedtime.html?msg=12%3A00+Penn+Stat…> . - Zoom Link <https://zoom.us/j/98189403566?pwd=SWhueXkzNHlKSG1QMU0xUG1ESFFsUT09> If you have any ideas for papercut issues, please add them to the appropriate repo. Thanks, and hope to see you soon, Dave C -- https://galaxyproject.org/

1 0

Galaxy Newsletter: August 2021
by Beatriz Serrano-Solano 06 Aug '21

06 Aug '21

Hello all, The August newsletter brings news of GCC2021 (just finished) and GCC2022 (in Minneapolis); the launch of the Galaxy India community, and open positions in 7 countries <https://galaxyproject.org/news/2021-08-galaxy-update/#community-news>; event news <https://galaxyproject.org/news/2021-08-galaxy-update/#event-news>; Galaxy platform news <https://galaxyproject.org/news/2021-08-galaxy-update/#galaxy-platforms-news>; Galactic blog posts <https://galaxyproject.org/news/2021-08-galaxy-update/#galactic-blog-activity>; training and doc updates <https://galaxyproject.org/news/2021-08-galaxy-update/#doc-hub-and-training-…>; recent open-access Galaxy-related publications <https://galaxyproject.org/news/2021-08-galaxy-update/#publications>; and other news too <https://galaxyproject.org/news/2021-08-galaxy-update/#other-news>.** Dave Clements and Beatriz Serrano-Solano, Editors PS: Subscribe to the Galaxy Announce mailing list <https://lists.galaxyproject.org/?count=100&all-lists=> and receive an email whenever this newsletter is published. -- Dr Beatriz Serrano-Solano (she/her) <https://www.hrc.org/resources/talking-about-pronouns-in-the-workplace> Bioinformatics Group Dept. Computer Science Albert-Ludwigs-Universität Freiburg Georges-Köhler-Allee, 106 79110 Freiburg Germany

1 0

Galaxy install problems
by Luc Cornet 09 Jul '21

09 Jul '21

Dear Galaxy admins, I am contacting you due to multiple problems during the installation of our own server of galaxy. We have been struggling to install galaxy on our HPC system for 1.5 year. We have faced so many problems while installing galaxy that we have lost count. Yet, our main problem was and remains the use of pulsar... Even after following two training sessions, switched to a brand new CentOS 8 cluster (while CentOS 8 will die at the end of the year), we are still not able to use our galaxy instance. We followed the ansible documentation and we are now able to reproduce the training sessions (i.e, execute an analysis with pulsar on our HPC system). Nevertheless, we are not able to connect pulsar with slurm (which makes it unusable). We do not want to use DRMAA because it is not well maintained and compatible with slurm. Instead, we choose to use CLI (as mentioned in the galaxy docs) but we are stuck (see below the pulsarservers.yaml file). Is it possible to use something else than DRMAA to connect pulsar to the scheduler? Thanks, Best regards Luc Cornet - - pulsarservers.yaml - - # Put your Galaxy server's fully qualified domain name (FQDN) (or the FQDN of the RabbitMQ server) above. pulsar_root: /opt/pulsar pulsar_pip_install: true pulsar_pycurl_ssl_library: openssl pulsar_systemd: true pulsar_systemd_runner: webless pulsar_create_user: false pulsar_user: {name: pulsar, shell: /bin/bash} pulsar_optional_dependencies: - pyOpenSSL # For remote transfers initiated on the Pulsar end rather than the Galaxy end - pycurl # drmaa required if connecting to an external DRM using it. - drmaa # kombu needed if using a message queue - kombu # amqp 5.0.3 changes behaviour in an unexpected way, pin for now. - 'amqp==5.0.2' # psutil and pylockfile are optional dependencies but can make Pulsar # more robust in small ways. - psutil pulsar_yaml_config: conda_auto_init: True conda_auto_install: True staging_directory: "{{ pulsar_staging_dir }}" persistence_directory: "{{ pulsar_persistence_dir }}" tool_dependency_dir: "{{ pulsar_dependencies_dir }}" # The following are the settings for the pulsar server to contact the message queue with related timeouts etc. message_queue_url: "pyamqp://galaxy_au:{{ rabbitmq_password_galaxy_au }}@{{ galaxy_server_url }}:5671//pulsar/galaxy_au?ssl=1" managers: _default_: type: queued_cli job_plugin: slurm native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" min_polling_interval: 0.5 amqp_publish_retry: True amqp_publish_retry_max_retries: 5 amqp_publish_retry_interval_start: 10 amqp_publish_retry_interval_step: 10 amqp_publish_retry_interval_max: 60 # We also need to create the dependency resolver file so pulsar knows how to # find and install dependencies for the tools we ask it to run. The simplest # method which covers 99% of the use cases is to use conda auto installs similar # to how Galaxy works. pulsar_dependency_resolvers: - name: conda args: - name: auto_init value: true ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano

4 16

Galaxy Newsletter: July 2021
by Dave Clements 02 Jul '21

02 Jul '21

Hello all, The big news for July is that the 2021 Galaxy Community Conference <https://gcc2021.sched.com/> is under way. With GCC going on, this is one of our thinner newsletters <https://galaxyproject.org/news/2021-07-galaxy-update/>, but there are still *open positions <https://galaxyproject.org/news/2021-07-galaxy-update/#community-news>*, *event news <https://galaxyproject.org/news/2021-07-galaxy-update/#event-news>*, *Galaxy platform news <https://galaxyproject.org/news/2021-07-galaxy-update/#galaxy-platforms-news>*, *Galactic blog posts <https://galaxyproject.org/news/2021-07-galaxy-update/#galactic-blog-activity>*, *training and doc updates <https://galaxyproject.org/news/2021-07-galaxy-update/#doc-hub-and-training-…>*, recent *open-access Galaxy-related publications <https://galaxyproject.org/news/2021-07-galaxy-update/#publications>*, *new releases <https://galaxyproject.org/news/2021-07-galaxy-update/#releases>, and other news too <https://galaxyproject.org/news/2021-07-galaxy-update/#other-news>.* We hope to see you at GCC events this week and next. Dave Clements and Beatriz Serrano-Solano, Editors PS: It is too late to register for the 3 day conference next week. It is also officially too late to sign up for the CoFest on 9-10 July, but if you follow this link <https://docs.google.com/forms/d/e/1FAIpQLScdJHTzw7LWVxFaqGI30COZlcxGPksjGL_…>, you just mind find yourself invited anyway. -- https://gaaxyproject.org/ <https://galaxyproject.org/>

1 0

GCC202 Registration Ends Tomorrow
by Beatriz Serrano-Solano 24 Jun '21

24 Jun '21

Hello All, The final registration for the 2021 Galaxy Community Conference is closing tomorrow, Friday 25th of June, at midnight CEST, so you have still some time to register! https://www.vibconferences.be/events/gcc2021-virtual-edition We will meet in Remo (https://remo.co/) and the schedule is already available at https://gcc2021.sched.com/ We are looking forward to seeing you online! Bea on behalf of the GCC2021 Organizing Committee -- Dr Beatriz Serrano-Solano (she/her) Bioinformatics Group Dept. Computer Science Albert-Ludwigs-Universität Freiburg Georges-Köhler-Allee, 106 79110 Freiburg Germany

1 0

Re: Running slurm job with pulsar
by Luc Cornet 19 Jun '21

19 Jun '21

Dear Gianmauro, Thanks for your answer. I think that my question was not clear enough. Below you will find some answer to your suggestions and maybe it will be more clear for you to guide me. * check the Pulsar' log for error messages ->I look into staging directory but as my test analysis is executed with success, I have no error log. When a lunch a test from the galaxy GUI, it is executed with success on the cluster. My problem is that is not executed in a job, not using srun or sbatch command (so not using the scheduler). I would like pulsar to be able to submit a job on the cluster, just like others users, and not execute analysis directly "in the terminal". * verify if your Pulsar server can reach the cluster trough ssh ->The pulsar server is on the HPC cluster. The connection between galaxy (rabbitmq) and the cluster (pulsar) is fine. * in the staging directory of your job should be a command.sh file. You can try to run it manually (sbatch command.sh or something similar) and Yes indeed I have this file in staging directory (see below). We i execute the command.sh file with sbatch command.sh, it fails immediately which is normal since command.sh is not a slurm job. total 48 -rw-r--r-- 1 pulsar pulsar 4 Jun 17 14:15 use_metadata_directory -rw-r--r-- 1 pulsar pulsar 10 Jun 17 14:15 tool_version -rw-r--r-- 1 pulsar pulsar 59 Jun 17 14:15 tool_id drwxr-xr-x 2 pulsar pulsar 6 Jun 17 14:15 tool_files drwxr-xr-x 2 pulsar pulsar 6 Jun 17 14:15 metadata drwxr-xr-x 2 pulsar pulsar 6 Jun 17 14:15 configs drwxr-xr-x 5 pulsar pulsar 30 Jun 17 14:15 .. -rw-r--r-- 1 pulsar pulsar 2551 Jun 17 14:15 launch_config drwxr-xr-x 2 pulsar pulsar 46 Jun 17 14:15 inputs -rw-r--r-- 1 pulsar pulsar 4 Jun 17 14:15 preprocessed -rwx------ 1 pulsar pulsar 5441 Jun 17 14:15 command.sh -rw-r--r-- 1 pulsar pulsar 0 Jun 17 14:15 stdout drwxr-xr-x 2 pulsar pulsar 6 Jun 17 14:15 home drwxr-xr-x 2 pulsar pulsar 58 Jun 17 14:15 working -rw-r--r-- 1 pulsar pulsar 4 Jun 17 14:15 running -rw-r--r-- 1 pulsar pulsar 546 Jun 17 14:15 stderr drwxr-xr-x 2 pulsar pulsar 26 Jun 17 14:15 outputs -rw-r--r-- 1 pulsar pulsar 1 Jun 17 14:15 return_code -rw-r--r-- 1 pulsar pulsar 10 Jun 17 14:15 final_status -rw-r--r-- 1 pulsar pulsar 0 Jun 17 14:15 postprocessed drwxr-xr-x 9 pulsar pulsar 4096 Jun 17 14:15 . ------------ Luc Cornet, PhD Bio-informatician Mycology and Aerobiology Sciensano ----- Mail original ----- De: "Gianmauro Cuccuru" <gmauro(a)informatik.uni-freiburg.de> À: "Luc Cornet" <luc.cornet(a)uliege.be>, "HelpGalaxy" <galaxy-dev(a)lists.galaxyproject.org> Cc: "Colignon David" <David.Colignon(a)uliege.be>, "Baurain Denis" <Denis.Baurain(a)uliege.be>, "Pierre Becker" <Pierre.Becker(a)sciensano.be> Envoyé: Vendredi 18 Juin 2021 11:48:30 Objet: Re: [galaxy-dev] Running slurm job with pulsar Hi Luc, I am not a Slurm expert but I can suggest you several things: * check the Pulsar' log for error messages * verify if your Pulsar server can reach the cluster trough ssh * in the staging directory of your job should be a command.sh file. You can try to run it manually (sbatch command.sh or something similar) and see if it works Cheers, Gianmauro On 17.06.21 20:18, Luc Cornet wrote: > Dear all, > > I am trying to launch a slurm job with pulsar using CLI (instead of drmaa). > The pulsar playbook below pass without problem but the analyses is still run out of slurm. > The analyses is excited with success but not in a slrum job. > > Can you help me to launch slurm with pulsar ? > What did I miss? > > Thanks, > Luc > > > ``` > # Put your Galaxy server's fully qualified domain name (FQDN) (or the FQDN of the RabbitMQ server) above. > > pulsar_root: /opt/pulsar > > pulsar_pip_install: true > pulsar_pycurl_ssl_library: openssl > pulsar_systemd: true > pulsar_systemd_runner: webless > > pulsar_create_user: false > pulsar_user: {name: pulsar, shell: /bin/bash} > > pulsar_optional_dependencies: > - pyOpenSSL > # For remote transfers initiated on the Pulsar end rather than the Galaxy end > - pycurl > # drmaa required if connecting to an external DRM using it. > - drmaa > # kombu needed if using a message queue > - kombu > # amqp 5.0.3 changes behaviour in an unexpected way, pin for now. > - 'amqp==5.0.2' > # psutil and pylockfile are optional dependencies but can make Pulsar > # more robust in small ways. > - psutil > > pulsar_yaml_config: > conda_auto_init: True > conda_auto_install: True > staging_directory: "{{ pulsar_staging_dir }}" > persistence_directory: "{{ pulsar_persistence_dir }}" > tool_dependency_dir: "{{ pulsar_dependencies_dir }}" > # The following are the settings for the pulsar server to contact the message queue with related timeouts etc. > message_queue_url: "pyamqp://galaxy_au:{{ rabbitmq_password_galaxy_au }}@{{ galaxy_server_url }}:5671//pulsar/galaxy_au?ssl=1" > managers: > _default_: > type: queued_cli > job_plugin: slurm > native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000 -t 10:00" > min_polling_interval: 0.5 > amqp_publish_retry: True > amqp_publish_retry_max_retries: 5 > amqp_publish_retry_interval_start: 10 > amqp_publish_retry_interval_step: 10 > amqp_publish_retry_interval_max: 60 > > # We also need to create the dependency resolver file so pulsar knows how to > # find and install dependencies for the tools we ask it to run. The simplest > # method which covers 99% of the use cases is to use conda auto installs similar > # to how Galaxy works. > pulsar_dependency_resolvers: > - name: conda > args: > - name: auto_init > value: true > ``` > > ------------ > Luc Cornet, PhD > Bio-informatician > Mycology and Aerobiology > Sciensano > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > %(web_page_url)s > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/ -- Gianmauro Cuccuru UseGalaxy.eu Bioinformatics Group Department of Computer Science Albert-Ludwigs-University Freiburg Georges-Köhler-Allee 106 79110 Freiburg, Germany

2 2