Expose Command Line in Galaxy
Dear Galaxy Community, is there any possibility to expose the command line call, generated within galaxy? We searched the net and found this discussions: https://github.com/galaxyproject/galaxy/issues/3252 https://github.com/galaxyproject/galaxy/issues/2954 But both are on the exposure to non-admin users. Within our instance we see the command line call nor for admin, neither for normal users! The problem, behind the question is the error message: "An invalid option was selected for database, 'myCoolDatabaseName', please verify." In Blast (toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastn_wrapper/0.2.01) Local (directly from command line, without Galaxy in between) the blast works fine. We are using the 18.05 Docker Galaxy Version. Thanks in advance! Mit freundlichen Grüßen Matthias Enders ----------------------------------------------------------------------- GERMAN SEED ALLIANCE GmbH c/o Norddeutsche Pflanzenzucht Hans-Georg Lembke KG Hohenlieth, 24363 Holtsee Tel.: +49 (0)4351/ 736-189 Fax: + 49 (0)4351/ 736-271 Mobil: +49 (0)151/ 14247360 Email: m.enders@german-seed-alliance.de Firmensitz Köln Amtsgericht Köln, HRB 73844 -----Ursprüngliche Nachricht----- Von: galaxy-dev <galaxy-dev-bounces@lists.galaxyproject.org> Im Auftrag von galaxy-dev-request@lists.galaxyproject.org Gesendet: Montag, 9. Juli 2018 18:00 An: galaxy-dev@lists.galaxyproject.org Betreff: galaxy-dev Digest, Vol 145, Issue 3 Send galaxy-dev mailing list submissions to galaxy-dev@lists.galaxyproject.org To subscribe or unsubscribe via the World Wide Web, visit https://lists.galaxyproject.org/listinfo/galaxy-dev or, via email, send a message with subject or body 'help' to galaxy-dev-request@lists.galaxyproject.org You can reach the person managing the list at galaxy-dev-owner@lists.galaxyproject.org When replying, please edit your Subject line so it is more specific than "Re: Contents of galaxy-dev digest..." HEY! This is important! If you reply to a thread in a digest, please 1. Change the subject of your response from "Galaxy-dev Digest Vol ..." to the original subject for the thread. 2. Strip out everything else in the digest that is not part of the thread you are responding to. Why? 1. This will keep the subject meaningful. People will have some idea from the subject line if they should read it or not. 2. Not doing this greatly increases the number of emails that match search queries, but that aren't actually informative. Today's Topics: 1. From Paste to uWSGI (Jochen Bick) 2. Re: From Paste to uWSGI (Peter Briggs) 3. Re: From Paste to uWSGI (Martin Čech) ---------------------------------------------------------------------- Message: 1 Date: Mon, 9 Jul 2018 13:49:23 +0200 From: Jochen Bick <jochen.bick@usys.ethz.ch> To: "galaxy-dev@lists.galaxyproject.org" <galaxy-dev@lists.galaxyproject.org> Subject: [galaxy-dev] From Paste to uWSGI Message-ID: <1a0f33ad-938c-64d5-8db8-28914a5ca07e@usys.ethz.ch> Content-Type: text/plain; charset="utf-8"; format=flowed Hi all, we would like to update to 18.05 and thereby also update our Paste Server. https://docs.galaxyproject.org/en/latest/admin/scaling.html Is there any tutorial on how to update from Paste to uWSGI? Cheers Jochen -- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 / LFW B 58.1 8092 Zurich, Switzerland Office: Eschikon 27 8315 Lindau, Switzerland Phone +41 52 354 92 06 jochen.bick@usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch ------------------------------ Message: 2 Date: Mon, 9 Jul 2018 15:31:16 +0000 From: Peter Briggs <peter.briggs@manchester.ac.uk> To: Jochen Bick <jochen.bick@usys.ethz.ch>, "galaxy-dev@lists.galaxyproject.org" <galaxy-dev@lists.galaxyproject.org> Subject: Re: [galaxy-dev] From Paste to uWSGI Message-ID: <E20AB71CD352854B9E24B5830826BC43C8586657@MBXP15.ds.man.ac.uk> Content-Type: text/plain; charset="iso-8859-1" Hello Jochen I'm not aware of a tutorial but here are some observations from my (very) recent experiences upgrading a local paste-based 17.09 instance to a uwsgi-based 18.05 configuration. It's not comprehensive because the configuration will depend on decisions you need to make based on your local setup, but it might be helpful as a starting point. Before beginning: -- Commit to migrating from galaxy.ini to galaxy.yml (if not done previously) Even though its possible to stick with the old galaxy.ini file, it seems best to move to the new format - my feeling is, probably the migration will have to happen at some point anyway, and also the documentation is more focussed on the newer galaxy.yml version. -- Decide on how you're going to handle job submission (essentially, how you're going to replace the old job handler processes). There are a number of possibilities outlined in the documentation here: https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#deploymen... It seems important to figure this out at the beginning because it impacts a lot of the configuration specifics that follow. For our setup case "uwsgi + mules" looked like the most appropriate: https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#uwsgi-for... Then once you're ready to start (nb backing up the database etc before starting are implicitly assumed!): -- Update the Galaxy source code The standard recipe for this seems to work okay i.e. git fetch origin && git checkout release_18.05 && git pull --ff-only origin release_18.05 (You'll probably also need to run scripts/common_startup.sh to update the Python packages, and scripts/manage_db.py to update the database schema.) -- Create galaxy.yml and configure the "galaxy" section: You can make a galaxy.yml from the sample version and go through the "galaxy" section replicating the settings from galaxy.ini. (The settings in the "uwsgi" section can wait at this stage.) Don't forget to move galaxy.ini out of the way once you're done. -- Configure the "uwsgi" section of galaxy.yml: The options for various scenarios are given in the "Deployment/uwsgi" section: https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#uwsgi but the required settings seem to be spread out between different sections so (for me) it felt like it took some piecing together to get it to work when I first tried this. (Also I was thrown a little by the "socket" parameter not having a placeholder in sample galaxy.yml.) The core uwsgi options are here: https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#configura... then you need to set other options dependent on: 1. how job submission is handled (e.g. options for the "uwsgi + mules" setup in my case) 2. how you're proxying Galaxy (for Apache or nginx it looks like it's just setting the uwsgi "socket" option, unless you're serving Galaxy from a URL prefix in which case there are a few more options needed - these are covered in the documentation.) -- Fix your job_conf.xml file If you have an existing job_conf.xml file in your config directory then you might need to update that too (specifically the "handlers" section) depending on the job submission handling you've decided to use. -- Use supervisor to manage your uwsgi processes The docs for this are here: https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#starting-... Essentially supervisor replaces "run.sh", so this becomes how you start and stop Galaxy (e.g. "supervisorctl start galaxy:" replaces "GALAXY_RUN_ALL=1 sh run.sh --daemon".) I think this is quite straightforward from the docs but again you need to tailor it for how you're handling job submission (e.g the "uwsgi + mules" doesn't need any handlers defining, but it looks like other strategies needs extra stuff defining here). -- Set up the proxying Assuming you're proxing via either Apache or nginx (we're using nginx), these are covered in the relevant sections: https://docs.galaxyproject.org/en/release_18.05/admin/apache.html https://docs.galaxyproject.org/en/release_18.05/admin/nginx.html The main difference I noticed for nginx is that you no longer need to use the "upstream"/"proxy_pass" directives, as nginx can talk directly to uwsgi using a "uwsgi_pass" directive that connects to the socket defined in the "uwsgi" section of galaxy.yml previously. (There's also a "gotcha" if you're nginx configuration also tries to serve "static/scripts/packed", which needs to be removed - this is noted in the "breaking changes" for the 18.05 docs but is easy to miss.) Hopefully this is some help. There's lots of documentation, which is a good thing but which can sometimes make it harder to see where to start. Good luck with the update, Best wishes Peter -- Peter Briggs peter.briggs@manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482 ________________________________________ From: galaxy-dev [galaxy-dev-bounces@lists.galaxyproject.org] on behalf of Jochen Bick [jochen.bick@usys.ethz.ch] Sent: Monday, July 09, 2018 12:49 PM To: galaxy-dev@lists.galaxyproject.org Subject: [galaxy-dev] From Paste to uWSGI Hi all, we would like to update to 18.05 and thereby also update our Paste Server. https://docs.galaxyproject.org/en/latest/admin/scaling.html Is there any tutorial on how to update from Paste to uWSGI? Cheers Jochen -- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 / LFW B 58.1 8092 Zurich, Switzerland Office: Eschikon 27 8315 Lindau, Switzerland Phone +41 52 354 92 06 jochen.bick@usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/ ------------------------------ Message: 3 Date: Mon, 9 Jul 2018 11:36:27 -0400 From: Martin Čech <marten@bx.psu.edu> To: Peter Briggs <peter.briggs@manchester.ac.uk> Cc: "galaxy-dev@lists.galaxyproject.org" <galaxy-dev@lists.galaxyproject.org> Subject: Re: [galaxy-dev] From Paste to uWSGI Message-ID: <CAHKBquGXMTog-MUgLbPWD2wgmXvpOxiQGRa4+QCWk5EAHGK3=g@mail.gmail.com> Content-Type: text/plain; charset="utf-8" Another recently updated resource for uwsgi is the training slides: https://galaxyproject.github.io/dagobah-training/2018-gccbosc/10-uwsgi/uwsgi... M. On Mon, Jul 9, 2018 at 11:31 AM Peter Briggs <peter.briggs@manchester.ac.uk> wrote:
Hello Jochen
I'm not aware of a tutorial but here are some observations from my (very) recent experiences upgrading a local paste-based 17.09 instance to a uwsgi-based 18.05 configuration. It's not comprehensive because the configuration will depend on decisions you need to make based on your local setup, but it might be helpful as a starting point.
Before beginning:
-- Commit to migrating from galaxy.ini to galaxy.yml (if not done previously)
Even though its possible to stick with the old galaxy.ini file, it seems best to move to the new format - my feeling is, probably the migration will have to happen at some point anyway, and also the documentation is more focussed on the newer galaxy.yml version.
-- Decide on how you're going to handle job submission (essentially, how you're going to replace the old job handler processes).
There are a number of possibilities outlined in the documentation here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#dep loyment-options
It seems important to figure this out at the beginning because it impacts a lot of the configuration specifics that follow. For our setup case "uwsgi + mules" looked like the most appropriate:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#uws gi-for-web-serving-with-mules-as-job-handlers
Then once you're ready to start (nb backing up the database etc before starting are implicitly assumed!):
-- Update the Galaxy source code
The standard recipe for this seems to work okay i.e.
git fetch origin && git checkout release_18.05 && git pull --ff-only origin release_18.05
(You'll probably also need to run scripts/common_startup.sh to update the Python packages, and scripts/manage_db.py to update the database schema.)
-- Create galaxy.yml and configure the "galaxy" section:
You can make a galaxy.yml from the sample version and go through the "galaxy" section replicating the settings from galaxy.ini. (The settings in the "uwsgi" section can wait at this stage.) Don't forget to move galaxy.ini out of the way once you're done.
-- Configure the "uwsgi" section of galaxy.yml:
The options for various scenarios are given in the "Deployment/uwsgi" section:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#uws gi
but the required settings seem to be spread out between different sections so (for me) it felt like it took some piecing together to get it to work when I first tried this.
(Also I was thrown a little by the "socket" parameter not having a placeholder in sample galaxy.yml.)
The core uwsgi options are here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#con figuration-common-to-all-uwsgi-deployment-styles
then you need to set other options dependent on:
1. how job submission is handled (e.g. options for the "uwsgi + mules" setup in my case) 2. how you're proxying Galaxy (for Apache or nginx it looks like it's just setting the uwsgi "socket" option, unless you're serving Galaxy from a URL prefix in which case there are a few more options needed - these are covered in the documentation.)
-- Fix your job_conf.xml file
If you have an existing job_conf.xml file in your config directory then you might need to update that too (specifically the "handlers" section) depending on the job submission handling you've decided to use.
-- Use supervisor to manage your uwsgi processes
The docs for this are here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#sta rting-and-stopping
Essentially supervisor replaces "run.sh", so this becomes how you start and stop Galaxy (e.g. "supervisorctl start galaxy:" replaces "GALAXY_RUN_ALL=1 sh run.sh --daemon".)
I think this is quite straightforward from the docs but again you need to tailor it for how you're handling job submission (e.g the "uwsgi + mules" doesn't need any handlers defining, but it looks like other strategies needs extra stuff defining here).
-- Set up the proxying
Assuming you're proxing via either Apache or nginx (we're using nginx), these are covered in the relevant sections:
https://docs.galaxyproject.org/en/release_18.05/admin/apache.html https://docs.galaxyproject.org/en/release_18.05/admin/nginx.html
The main difference I noticed for nginx is that you no longer need to use the "upstream"/"proxy_pass" directives, as nginx can talk directly to uwsgi using a "uwsgi_pass" directive that connects to the socket defined in the "uwsgi" section of galaxy.yml previously.
(There's also a "gotcha" if you're nginx configuration also tries to serve "static/scripts/packed", which needs to be removed - this is noted in the "breaking changes" for the 18.05 docs but is easy to miss.)
Hopefully this is some help. There's lots of documentation, which is a good thing but which can sometimes make it harder to see where to start.
Good luck with the update,
Best wishes
Peter
-- Peter Briggs peter.briggs@manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482
________________________________________ From: galaxy-dev [galaxy-dev-bounces@lists.galaxyproject.org] on behalf of Jochen Bick [jochen.bick@usys.ethz.ch] Sent: Monday, July 09, 2018 12:49 PM To: galaxy-dev@lists.galaxyproject.org Subject: [galaxy-dev] From Paste to uWSGI
Hi all,
we would like to update to 18.05 and thereby also update our Paste Server.
https://docs.galaxyproject.org/en/latest/admin/scaling.html
Is there any tutorial on how to update from Paste to uWSGI?
Cheers Jochen
-- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 <https://maps.google.com/?q=Universit%C3%A4tstrasse+2&entry=gmail&sour ce=g> / LFW B 58.1 8092 Zurich, Switzerland Office: Eschikon 27 8315 Lindau, Switzerland
Phone +41 52 354 92 06 <+41%2052%20354%2092%2006> jochen.bick@usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/ ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://lists.galaxyproject.org/pipermail/galaxy-dev/attachments/20180709/a7d44c29/attachment-0001.html>
------------------------------ Subject: Digest Footer _______________________________________________ galaxy-dev mailing list galaxy-dev@lists.galaxyproject.org https://lists.galaxyproject.org/listinfo/galaxy-dev To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ------------------------------ End of galaxy-dev Digest, Vol 145, Issue 3 ******************************************
Hi Matthias, please have a look at the attached screenshot. Given that you are an admin and click on the info-symbol of the dataset you should see the commandline. This is with a 18.05 Docker. Cheers, Bjoern Am 10.07.2018 um 08:58 schrieb Matthias Enders:
Dear Galaxy Community,
is there any possibility to expose the command line call, generated within galaxy?
We searched the net and found this discussions: https://github.com/galaxyproject/galaxy/issues/3252 https://github.com/galaxyproject/galaxy/issues/2954
But both are on the exposure to non-admin users. Within our instance we see the command line call nor for admin, neither for normal users!
The problem, behind the question is the error message: "An invalid option was selected for database, 'myCoolDatabaseName', please verify." In Blast (toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastn_wrapper/0.2.01)
Local (directly from command line, without Galaxy in between) the blast works fine.
We are using the 18.05 Docker Galaxy Version.
Thanks in advance!
Mit freundlichen Grüßen
Matthias Enders -----------------------------------------------------------------------
GERMAN SEED ALLIANCE GmbH c/o Norddeutsche Pflanzenzucht Hans-Georg Lembke KG Hohenlieth, 24363 Holtsee Tel.: +49 (0)4351/ 736-189 Fax: + 49 (0)4351/ 736-271 Mobil: +49 (0)151/ 14247360
Email: m.enders@german-seed-alliance.de
Firmensitz Köln Amtsgericht Köln, HRB 73844
-----Ursprüngliche Nachricht----- Von: galaxy-dev <galaxy-dev-bounces@lists.galaxyproject.org> Im Auftrag von galaxy-dev-request@lists.galaxyproject.org Gesendet: Montag, 9. Juli 2018 18:00 An: galaxy-dev@lists.galaxyproject.org Betreff: galaxy-dev Digest, Vol 145, Issue 3
Send galaxy-dev mailing list submissions to galaxy-dev@lists.galaxyproject.org
To subscribe or unsubscribe via the World Wide Web, visit https://lists.galaxyproject.org/listinfo/galaxy-dev or, via email, send a message with subject or body 'help' to galaxy-dev-request@lists.galaxyproject.org
You can reach the person managing the list at galaxy-dev-owner@lists.galaxyproject.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of galaxy-dev digest..."
HEY! This is important! If you reply to a thread in a digest, please 1. Change the subject of your response from "Galaxy-dev Digest Vol ..." to the original subject for the thread. 2. Strip out everything else in the digest that is not part of the thread you are responding to.
Why? 1. This will keep the subject meaningful. People will have some idea from the subject line if they should read it or not. 2. Not doing this greatly increases the number of emails that match search queries, but that aren't actually informative.
Today's Topics:
1. From Paste to uWSGI (Jochen Bick) 2. Re: From Paste to uWSGI (Peter Briggs) 3. Re: From Paste to uWSGI (Martin Čech)
----------------------------------------------------------------------
Message: 1 Date: Mon, 9 Jul 2018 13:49:23 +0200 From: Jochen Bick <jochen.bick@usys.ethz.ch> To: "galaxy-dev@lists.galaxyproject.org" <galaxy-dev@lists.galaxyproject.org> Subject: [galaxy-dev] From Paste to uWSGI Message-ID: <1a0f33ad-938c-64d5-8db8-28914a5ca07e@usys.ethz.ch> Content-Type: text/plain; charset="utf-8"; format=flowed
Hi all,
we would like to update to 18.05 and thereby also update our Paste Server.
https://docs.galaxyproject.org/en/latest/admin/scaling.html
Is there any tutorial on how to update from Paste to uWSGI?
Cheers Jochen
-- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 / LFW B 58.1 8092 Zurich, Switzerland Office: Eschikon 27 8315 Lindau, Switzerland
Phone +41 52 354 92 06 jochen.bick@usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch
------------------------------
Message: 2 Date: Mon, 9 Jul 2018 15:31:16 +0000 From: Peter Briggs <peter.briggs@manchester.ac.uk> To: Jochen Bick <jochen.bick@usys.ethz.ch>, "galaxy-dev@lists.galaxyproject.org" <galaxy-dev@lists.galaxyproject.org> Subject: Re: [galaxy-dev] From Paste to uWSGI Message-ID: <E20AB71CD352854B9E24B5830826BC43C8586657@MBXP15.ds.man.ac.uk> Content-Type: text/plain; charset="iso-8859-1"
Hello Jochen
I'm not aware of a tutorial but here are some observations from my (very) recent experiences upgrading a local paste-based 17.09 instance to a uwsgi-based 18.05 configuration. It's not comprehensive because the configuration will depend on decisions you need to make based on your local setup, but it might be helpful as a starting point.
Before beginning:
-- Commit to migrating from galaxy.ini to galaxy.yml (if not done previously)
Even though its possible to stick with the old galaxy.ini file, it seems best to move to the new format - my feeling is, probably the migration will have to happen at some point anyway, and also the documentation is more focussed on the newer galaxy.yml version.
-- Decide on how you're going to handle job submission (essentially, how you're going to replace the old job handler processes).
There are a number of possibilities outlined in the documentation here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#deploymen...
It seems important to figure this out at the beginning because it impacts a lot of the configuration specifics that follow. For our setup case "uwsgi + mules" looked like the most appropriate:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#uwsgi-for...
Then once you're ready to start (nb backing up the database etc before starting are implicitly assumed!):
-- Update the Galaxy source code
The standard recipe for this seems to work okay i.e.
git fetch origin && git checkout release_18.05 && git pull --ff-only origin release_18.05
(You'll probably also need to run scripts/common_startup.sh to update the Python packages, and scripts/manage_db.py to update the database schema.)
-- Create galaxy.yml and configure the "galaxy" section:
You can make a galaxy.yml from the sample version and go through the "galaxy" section replicating the settings from galaxy.ini. (The settings in the "uwsgi" section can wait at this stage.) Don't forget to move galaxy.ini out of the way once you're done.
-- Configure the "uwsgi" section of galaxy.yml:
The options for various scenarios are given in the "Deployment/uwsgi" section:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#uwsgi
but the required settings seem to be spread out between different sections so (for me) it felt like it took some piecing together to get it to work when I first tried this.
(Also I was thrown a little by the "socket" parameter not having a placeholder in sample galaxy.yml.)
The core uwsgi options are here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#configura...
then you need to set other options dependent on:
1. how job submission is handled (e.g. options for the "uwsgi + mules" setup in my case) 2. how you're proxying Galaxy (for Apache or nginx it looks like it's just setting the uwsgi "socket" option, unless you're serving Galaxy from a URL prefix in which case there are a few more options needed - these are covered in the documentation.)
-- Fix your job_conf.xml file
If you have an existing job_conf.xml file in your config directory then you might need to update that too (specifically the "handlers" section) depending on the job submission handling you've decided to use.
-- Use supervisor to manage your uwsgi processes
The docs for this are here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#starting-...
Essentially supervisor replaces "run.sh", so this becomes how you start and stop Galaxy (e.g. "supervisorctl start galaxy:" replaces "GALAXY_RUN_ALL=1 sh run.sh --daemon".)
I think this is quite straightforward from the docs but again you need to tailor it for how you're handling job submission (e.g the "uwsgi + mules" doesn't need any handlers defining, but it looks like other strategies needs extra stuff defining here).
-- Set up the proxying
Assuming you're proxing via either Apache or nginx (we're using nginx), these are covered in the relevant sections:
https://docs.galaxyproject.org/en/release_18.05/admin/apache.html https://docs.galaxyproject.org/en/release_18.05/admin/nginx.html
The main difference I noticed for nginx is that you no longer need to use the "upstream"/"proxy_pass" directives, as nginx can talk directly to uwsgi using a "uwsgi_pass" directive that connects to the socket defined in the "uwsgi" section of galaxy.yml previously.
(There's also a "gotcha" if you're nginx configuration also tries to serve "static/scripts/packed", which needs to be removed - this is noted in the "breaking changes" for the 18.05 docs but is easy to miss.)
Hopefully this is some help. There's lots of documentation, which is a good thing but which can sometimes make it harder to see where to start.
Good luck with the update,
Best wishes
Peter
-- Peter Briggs peter.briggs@manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482
________________________________________ From: galaxy-dev [galaxy-dev-bounces@lists.galaxyproject.org] on behalf of Jochen Bick [jochen.bick@usys.ethz.ch] Sent: Monday, July 09, 2018 12:49 PM To: galaxy-dev@lists.galaxyproject.org Subject: [galaxy-dev] From Paste to uWSGI
Hi all,
we would like to update to 18.05 and thereby also update our Paste Server.
https://docs.galaxyproject.org/en/latest/admin/scaling.html
Is there any tutorial on how to update from Paste to uWSGI?
Cheers Jochen
-- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 / LFW B 58.1 8092 Zurich, Switzerland Office: Eschikon 27 8315 Lindau, Switzerland
Phone +41 52 354 92 06 jochen.bick@usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
------------------------------
Message: 3 Date: Mon, 9 Jul 2018 11:36:27 -0400 From: Martin Čech <marten@bx.psu.edu> To: Peter Briggs <peter.briggs@manchester.ac.uk> Cc: "galaxy-dev@lists.galaxyproject.org" <galaxy-dev@lists.galaxyproject.org> Subject: Re: [galaxy-dev] From Paste to uWSGI Message-ID: <CAHKBquGXMTog-MUgLbPWD2wgmXvpOxiQGRa4+QCWk5EAHGK3=g@mail.gmail.com> Content-Type: text/plain; charset="utf-8"
Another recently updated resource for uwsgi is the training slides: https://galaxyproject.github.io/dagobah-training/2018-gccbosc/10-uwsgi/uwsgi...
M.
On Mon, Jul 9, 2018 at 11:31 AM Peter Briggs <peter.briggs@manchester.ac.uk> wrote:
Hello Jochen
I'm not aware of a tutorial but here are some observations from my (very) recent experiences upgrading a local paste-based 17.09 instance to a uwsgi-based 18.05 configuration. It's not comprehensive because the configuration will depend on decisions you need to make based on your local setup, but it might be helpful as a starting point.
Before beginning:
-- Commit to migrating from galaxy.ini to galaxy.yml (if not done previously)
Even though its possible to stick with the old galaxy.ini file, it seems best to move to the new format - my feeling is, probably the migration will have to happen at some point anyway, and also the documentation is more focussed on the newer galaxy.yml version.
-- Decide on how you're going to handle job submission (essentially, how you're going to replace the old job handler processes).
There are a number of possibilities outlined in the documentation here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#dep
loyment-options
It seems important to figure this out at the beginning because it impacts a lot of the configuration specifics that follow. For our setup case "uwsgi + mules" looked like the most appropriate:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#uws
gi-for-web-serving-with-mules-as-job-handlers
Then once you're ready to start (nb backing up the database etc before starting are implicitly assumed!):
-- Update the Galaxy source code
The standard recipe for this seems to work okay i.e.
git fetch origin && git checkout release_18.05 && git pull --ff-only origin release_18.05
(You'll probably also need to run scripts/common_startup.sh to update the Python packages, and scripts/manage_db.py to update the database schema.)
-- Create galaxy.yml and configure the "galaxy" section:
You can make a galaxy.yml from the sample version and go through the "galaxy" section replicating the settings from galaxy.ini. (The settings in the "uwsgi" section can wait at this stage.) Don't forget to move galaxy.ini out of the way once you're done.
-- Configure the "uwsgi" section of galaxy.yml:
The options for various scenarios are given in the "Deployment/uwsgi" section:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#uws
gi
but the required settings seem to be spread out between different sections so (for me) it felt like it took some piecing together to get it to work when I first tried this.
(Also I was thrown a little by the "socket" parameter not having a placeholder in sample galaxy.yml.)
The core uwsgi options are here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#con
figuration-common-to-all-uwsgi-deployment-styles
then you need to set other options dependent on:
1. how job submission is handled (e.g. options for the "uwsgi + mules" setup in my case) 2. how you're proxying Galaxy (for Apache or nginx it looks like it's just setting the uwsgi "socket" option, unless you're serving Galaxy from a URL prefix in which case there are a few more options needed - these are covered in the documentation.)
-- Fix your job_conf.xml file
If you have an existing job_conf.xml file in your config directory then you might need to update that too (specifically the "handlers" section) depending on the job submission handling you've decided to use.
-- Use supervisor to manage your uwsgi processes
The docs for this are here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#sta
rting-and-stopping
Essentially supervisor replaces "run.sh", so this becomes how you start and stop Galaxy (e.g. "supervisorctl start galaxy:" replaces "GALAXY_RUN_ALL=1 sh run.sh --daemon".)
I think this is quite straightforward from the docs but again you need to tailor it for how you're handling job submission (e.g the "uwsgi + mules" doesn't need any handlers defining, but it looks like other strategies needs extra stuff defining here).
-- Set up the proxying
Assuming you're proxing via either Apache or nginx (we're using nginx), these are covered in the relevant sections:
https://docs.galaxyproject.org/en/release_18.05/admin/apache.html https://docs.galaxyproject.org/en/release_18.05/admin/nginx.html
The main difference I noticed for nginx is that you no longer need to use the "upstream"/"proxy_pass" directives, as nginx can talk directly to uwsgi using a "uwsgi_pass" directive that connects to the socket defined in the "uwsgi" section of galaxy.yml previously.
(There's also a "gotcha" if you're nginx configuration also tries to serve "static/scripts/packed", which needs to be removed - this is noted in the "breaking changes" for the 18.05 docs but is easy to miss.)
Hopefully this is some help. There's lots of documentation, which is a good thing but which can sometimes make it harder to see where to start.
Good luck with the update,
Best wishes
Peter
-- Peter Briggs peter.briggs@manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482
________________________________________ From: galaxy-dev [galaxy-dev-bounces@lists.galaxyproject.org] on behalf of Jochen Bick [jochen.bick@usys.ethz.ch] Sent: Monday, July 09, 2018 12:49 PM To: galaxy-dev@lists.galaxyproject.org Subject: [galaxy-dev] From Paste to uWSGI
Hi all,
we would like to update to 18.05 and thereby also update our Paste Server.
https://docs.galaxyproject.org/en/latest/admin/scaling.html
Is there any tutorial on how to update from Paste to uWSGI?
Cheers Jochen
-- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 <https://maps.google.com/?q=Universit%C3%A4tstrasse+2&entry=gmail&sour
ce=g>
/ LFW B 58.1 8092 Zurich, Switzerland Office: Eschikon 27 8315 Lindau, Switzerland
Phone +41 52 354 92 06 <+41%2052%20354%2092%2006> jochen.bick@usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/ ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://lists.galaxyproject.org/pipermail/galaxy-dev/attachments/20180709/a7d44c29/attachment-0001.html>
------------------------------
Subject: Digest Footer
_______________________________________________ galaxy-dev mailing list galaxy-dev@lists.galaxyproject.org https://lists.galaxyproject.org/listinfo/galaxy-dev
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
------------------------------
End of galaxy-dev Digest, Vol 145, Issue 3 ****************************************** ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Dear Björn, thanks for the very fast reply. Unfortunately, this part is not visible in our instance. (see screenshot). As you can see in the topmost bar, this is done, while being loged in as admin. Mit freundlichen Grüßen Matthias Enders ----------------------------------------------------------------------- GERMAN SEED ALLIANCE GmbH c/o Norddeutsche Pflanzenzucht Hans-Georg Lembke KG Hohenlieth, 24363 Holtsee Tel.: +49 (0)4351/ 736-189 Fax: + 49 (0)4351/ 736-271 Mobil: +49 (0)151/ 14247360 Email: m.enders@german-seed-alliance.de Firmensitz Köln Amtsgericht Köln, HRB 73844 -----Ursprüngliche Nachricht----- Von: Björn Grüning <bjoern.gruening@gmail.com> Gesendet: Dienstag, 10. Juli 2018 09:15 An: Matthias Enders <m.enders@german-seed-alliance.de>; galaxy-dev@lists.galaxyproject.org Betreff: Re: [galaxy-dev] Expose Command Line in Galaxy Priorität: Hoch Hi Matthias, please have a look at the attached screenshot. Given that you are an admin and click on the info-symbol of the dataset you should see the commandline. This is with a 18.05 Docker. Cheers, Bjoern Am 10.07.2018 um 08:58 schrieb Matthias Enders:
Dear Galaxy Community,
is there any possibility to expose the command line call, generated within galaxy?
We searched the net and found this discussions: https://github.com/galaxyproject/galaxy/issues/3252 https://github.com/galaxyproject/galaxy/issues/2954
But both are on the exposure to non-admin users. Within our instance we see the command line call nor for admin, neither for normal users!
The problem, behind the question is the error message: "An invalid option was selected for database, 'myCoolDatabaseName', please verify." In Blast (toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastn_wrap per/0.2.01)
Local (directly from command line, without Galaxy in between) the blast works fine.
We are using the 18.05 Docker Galaxy Version.
Thanks in advance!
Mit freundlichen Grüßen
Matthias Enders ---------------------------------------------------------------------- -
GERMAN SEED ALLIANCE GmbH c/o Norddeutsche Pflanzenzucht Hans-Georg Lembke KG Hohenlieth, 24363 Holtsee Tel.: +49 (0)4351/ 736-189 Fax: + 49 (0)4351/ 736-271 Mobil: +49 (0)151/ 14247360
Email: m.enders@german-seed-alliance.de
Firmensitz Köln Amtsgericht Köln, HRB 73844
-----Ursprüngliche Nachricht----- Von: galaxy-dev <galaxy-dev-bounces@lists.galaxyproject.org> Im Auftrag von galaxy-dev-request@lists.galaxyproject.org Gesendet: Montag, 9. Juli 2018 18:00 An: galaxy-dev@lists.galaxyproject.org Betreff: galaxy-dev Digest, Vol 145, Issue 3
Send galaxy-dev mailing list submissions to galaxy-dev@lists.galaxyproject.org
To subscribe or unsubscribe via the World Wide Web, visit https://lists.galaxyproject.org/listinfo/galaxy-dev or, via email, send a message with subject or body 'help' to galaxy-dev-request@lists.galaxyproject.org
You can reach the person managing the list at galaxy-dev-owner@lists.galaxyproject.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of galaxy-dev digest..."
HEY! This is important! If you reply to a thread in a digest, please 1. Change the subject of your response from "Galaxy-dev Digest Vol ..." to the original subject for the thread. 2. Strip out everything else in the digest that is not part of the thread you are responding to.
Why? 1. This will keep the subject meaningful. People will have some idea from the subject line if they should read it or not. 2. Not doing this greatly increases the number of emails that match search queries, but that aren't actually informative.
Today's Topics:
1. From Paste to uWSGI (Jochen Bick) 2. Re: From Paste to uWSGI (Peter Briggs) 3. Re: From Paste to uWSGI (Martin Čech)
----------------------------------------------------------------------
Message: 1 Date: Mon, 9 Jul 2018 13:49:23 +0200 From: Jochen Bick <jochen.bick@usys.ethz.ch> To: "galaxy-dev@lists.galaxyproject.org" <galaxy-dev@lists.galaxyproject.org> Subject: [galaxy-dev] From Paste to uWSGI Message-ID: <1a0f33ad-938c-64d5-8db8-28914a5ca07e@usys.ethz.ch> Content-Type: text/plain; charset="utf-8"; format=flowed
Hi all,
we would like to update to 18.05 and thereby also update our Paste Server.
https://docs.galaxyproject.org/en/latest/admin/scaling.html
Is there any tutorial on how to update from Paste to uWSGI?
Cheers Jochen
-- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 / LFW B 58.1 8092 Zurich, Switzerland Office: Eschikon 27 8315 Lindau, Switzerland
Phone +41 52 354 92 06 jochen.bick@usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch
------------------------------
Message: 2 Date: Mon, 9 Jul 2018 15:31:16 +0000 From: Peter Briggs <peter.briggs@manchester.ac.uk> To: Jochen Bick <jochen.bick@usys.ethz.ch>, "galaxy-dev@lists.galaxyproject.org" <galaxy-dev@lists.galaxyproject.org> Subject: Re: [galaxy-dev] From Paste to uWSGI Message-ID: <E20AB71CD352854B9E24B5830826BC43C8586657@MBXP15.ds.man.ac.uk> Content-Type: text/plain; charset="iso-8859-1"
Hello Jochen
I'm not aware of a tutorial but here are some observations from my (very) recent experiences upgrading a local paste-based 17.09 instance to a uwsgi-based 18.05 configuration. It's not comprehensive because the configuration will depend on decisions you need to make based on your local setup, but it might be helpful as a starting point.
Before beginning:
-- Commit to migrating from galaxy.ini to galaxy.yml (if not done previously)
Even though its possible to stick with the old galaxy.ini file, it seems best to move to the new format - my feeling is, probably the migration will have to happen at some point anyway, and also the documentation is more focussed on the newer galaxy.yml version.
-- Decide on how you're going to handle job submission (essentially, how you're going to replace the old job handler processes).
There are a number of possibilities outlined in the documentation here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#dep loyment-options
It seems important to figure this out at the beginning because it impacts a lot of the configuration specifics that follow. For our setup case "uwsgi + mules" looked like the most appropriate:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#uws gi-for-web-serving-with-mules-as-job-handlers
Then once you're ready to start (nb backing up the database etc before starting are implicitly assumed!):
-- Update the Galaxy source code
The standard recipe for this seems to work okay i.e.
git fetch origin && git checkout release_18.05 && git pull --ff-only origin release_18.05
(You'll probably also need to run scripts/common_startup.sh to update the Python packages, and scripts/manage_db.py to update the database schema.)
-- Create galaxy.yml and configure the "galaxy" section:
You can make a galaxy.yml from the sample version and go through the "galaxy" section replicating the settings from galaxy.ini. (The settings in the "uwsgi" section can wait at this stage.) Don't forget to move galaxy.ini out of the way once you're done.
-- Configure the "uwsgi" section of galaxy.yml:
The options for various scenarios are given in the "Deployment/uwsgi" section:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#uws gi
but the required settings seem to be spread out between different sections so (for me) it felt like it took some piecing together to get it to work when I first tried this.
(Also I was thrown a little by the "socket" parameter not having a placeholder in sample galaxy.yml.)
The core uwsgi options are here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#con figuration-common-to-all-uwsgi-deployment-styles
then you need to set other options dependent on:
1. how job submission is handled (e.g. options for the "uwsgi + mules" setup in my case) 2. how you're proxying Galaxy (for Apache or nginx it looks like it's just setting the uwsgi "socket" option, unless you're serving Galaxy from a URL prefix in which case there are a few more options needed - these are covered in the documentation.)
-- Fix your job_conf.xml file
If you have an existing job_conf.xml file in your config directory then you might need to update that too (specifically the "handlers" section) depending on the job submission handling you've decided to use.
-- Use supervisor to manage your uwsgi processes
The docs for this are here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#sta rting-and-stopping
Essentially supervisor replaces "run.sh", so this becomes how you start and stop Galaxy (e.g. "supervisorctl start galaxy:" replaces "GALAXY_RUN_ALL=1 sh run.sh --daemon".)
I think this is quite straightforward from the docs but again you need to tailor it for how you're handling job submission (e.g the "uwsgi + mules" doesn't need any handlers defining, but it looks like other strategies needs extra stuff defining here).
-- Set up the proxying
Assuming you're proxing via either Apache or nginx (we're using nginx), these are covered in the relevant sections:
https://docs.galaxyproject.org/en/release_18.05/admin/apache.html https://docs.galaxyproject.org/en/release_18.05/admin/nginx.html
The main difference I noticed for nginx is that you no longer need to use the "upstream"/"proxy_pass" directives, as nginx can talk directly to uwsgi using a "uwsgi_pass" directive that connects to the socket defined in the "uwsgi" section of galaxy.yml previously.
(There's also a "gotcha" if you're nginx configuration also tries to serve "static/scripts/packed", which needs to be removed - this is noted in the "breaking changes" for the 18.05 docs but is easy to miss.)
Hopefully this is some help. There's lots of documentation, which is a good thing but which can sometimes make it harder to see where to start.
Good luck with the update,
Best wishes
Peter
-- Peter Briggs peter.briggs@manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482
________________________________________ From: galaxy-dev [galaxy-dev-bounces@lists.galaxyproject.org] on behalf of Jochen Bick [jochen.bick@usys.ethz.ch] Sent: Monday, July 09, 2018 12:49 PM To: galaxy-dev@lists.galaxyproject.org Subject: [galaxy-dev] From Paste to uWSGI
Hi all,
we would like to update to 18.05 and thereby also update our Paste Server.
https://docs.galaxyproject.org/en/latest/admin/scaling.html
Is there any tutorial on how to update from Paste to uWSGI?
Cheers Jochen
-- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 / LFW B 58.1 8092 Zurich, Switzerland Office: Eschikon 27 8315 Lindau, Switzerland
Phone +41 52 354 92 06 jochen.bick@usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
------------------------------
Message: 3 Date: Mon, 9 Jul 2018 11:36:27 -0400 From: Martin Čech <marten@bx.psu.edu> To: Peter Briggs <peter.briggs@manchester.ac.uk> Cc: "galaxy-dev@lists.galaxyproject.org" <galaxy-dev@lists.galaxyproject.org> Subject: Re: [galaxy-dev] From Paste to uWSGI Message-ID: <CAHKBquGXMTog-MUgLbPWD2wgmXvpOxiQGRa4+QCWk5EAHGK3=g@mail.gmail.com> Content-Type: text/plain; charset="utf-8"
Another recently updated resource for uwsgi is the training slides: https://galaxyproject.github.io/dagobah-training/2018-gccbosc/10-uwsgi /uwsgi.html#1
M.
On Mon, Jul 9, 2018 at 11:31 AM Peter Briggs <peter.briggs@manchester.ac.uk> wrote:
Hello Jochen
I'm not aware of a tutorial but here are some observations from my (very) recent experiences upgrading a local paste-based 17.09 instance to a uwsgi-based 18.05 configuration. It's not comprehensive because the configuration will depend on decisions you need to make based on your local setup, but it might be helpful as a starting point.
Before beginning:
-- Commit to migrating from galaxy.ini to galaxy.yml (if not done previously)
Even though its possible to stick with the old galaxy.ini file, it seems best to move to the new format - my feeling is, probably the migration will have to happen at some point anyway, and also the documentation is more focussed on the newer galaxy.yml version.
-- Decide on how you're going to handle job submission (essentially, how you're going to replace the old job handler processes).
There are a number of possibilities outlined in the documentation here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#de p
loyment-options
It seems important to figure this out at the beginning because it impacts a lot of the configuration specifics that follow. For our setup case "uwsgi + mules" looked like the most appropriate:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#uw s
gi-for-web-serving-with-mules-as-job-handlers
Then once you're ready to start (nb backing up the database etc before starting are implicitly assumed!):
-- Update the Galaxy source code
The standard recipe for this seems to work okay i.e.
git fetch origin && git checkout release_18.05 && git pull --ff-only origin release_18.05
(You'll probably also need to run scripts/common_startup.sh to update the Python packages, and scripts/manage_db.py to update the database schema.)
-- Create galaxy.yml and configure the "galaxy" section:
You can make a galaxy.yml from the sample version and go through the "galaxy" section replicating the settings from galaxy.ini. (The settings in the "uwsgi" section can wait at this stage.) Don't forget to move galaxy.ini out of the way once you're done.
-- Configure the "uwsgi" section of galaxy.yml:
The options for various scenarios are given in the "Deployment/uwsgi" section:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#uw s
gi
but the required settings seem to be spread out between different sections so (for me) it felt like it took some piecing together to get it to work when I first tried this.
(Also I was thrown a little by the "socket" parameter not having a placeholder in sample galaxy.yml.)
The core uwsgi options are here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#co n
figuration-common-to-all-uwsgi-deployment-styles
then you need to set other options dependent on:
1. how job submission is handled (e.g. options for the "uwsgi + mules" setup in my case) 2. how you're proxying Galaxy (for Apache or nginx it looks like it's just setting the uwsgi "socket" option, unless you're serving Galaxy from a URL prefix in which case there are a few more options needed - these are covered in the documentation.)
-- Fix your job_conf.xml file
If you have an existing job_conf.xml file in your config directory then you might need to update that too (specifically the "handlers" section) depending on the job submission handling you've decided to use.
-- Use supervisor to manage your uwsgi processes
The docs for this are here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#st a
rting-and-stopping
Essentially supervisor replaces "run.sh", so this becomes how you start and stop Galaxy (e.g. "supervisorctl start galaxy:" replaces "GALAXY_RUN_ALL=1 sh run.sh --daemon".)
I think this is quite straightforward from the docs but again you need to tailor it for how you're handling job submission (e.g the "uwsgi + mules" doesn't need any handlers defining, but it looks like other strategies needs extra stuff defining here).
-- Set up the proxying
Assuming you're proxing via either Apache or nginx (we're using nginx), these are covered in the relevant sections:
https://docs.galaxyproject.org/en/release_18.05/admin/apache.html https://docs.galaxyproject.org/en/release_18.05/admin/nginx.html
The main difference I noticed for nginx is that you no longer need to use the "upstream"/"proxy_pass" directives, as nginx can talk directly to uwsgi using a "uwsgi_pass" directive that connects to the socket defined in the "uwsgi" section of galaxy.yml previously.
(There's also a "gotcha" if you're nginx configuration also tries to serve "static/scripts/packed", which needs to be removed - this is noted in the "breaking changes" for the 18.05 docs but is easy to miss.)
Hopefully this is some help. There's lots of documentation, which is a good thing but which can sometimes make it harder to see where to start.
Good luck with the update,
Best wishes
Peter
-- Peter Briggs peter.briggs@manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482
________________________________________ From: galaxy-dev [galaxy-dev-bounces@lists.galaxyproject.org] on behalf of Jochen Bick [jochen.bick@usys.ethz.ch] Sent: Monday, July 09, 2018 12:49 PM To: galaxy-dev@lists.galaxyproject.org Subject: [galaxy-dev] From Paste to uWSGI
Hi all,
we would like to update to 18.05 and thereby also update our Paste Server.
https://docs.galaxyproject.org/en/latest/admin/scaling.html
Is there any tutorial on how to update from Paste to uWSGI?
Cheers Jochen
-- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 <https://maps.google.com/?q=Universit%C3%A4tstrasse+2&entry=gmail&sou r
ce=g>
/ LFW B 58.1 8092 Zurich, Switzerland Office: Eschikon 27 8315 Lindau, Switzerland
Phone +41 52 354 92 06 <+41%2052%20354%2092%2006> jochen.bick@usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/ ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://lists.galaxyproject.org/pipermail/galaxy-dev/attachments/2018 0709/a7d44c29/attachment-0001.html>
------------------------------
Subject: Digest Footer
_______________________________________________ galaxy-dev mailing list galaxy-dev@lists.galaxyproject.org https://lists.galaxyproject.org/listinfo/galaxy-dev
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
------------------------------
End of galaxy-dev Digest, Vol 145, Issue 3 ****************************************** ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
There is a setting to enable showing the command line string and full paths on disk via the "i" icon from a history entry, see https://docs.galaxyproject.org/en/master/admin/options.html#expose-dataset-p... # This option allows users to see the full path of datasets via the # "View Details" option in the history. This option also exposes the # command line to non-administrative users. Administrators can always # see dataset paths. #expose_dataset_path: false https://github.com/galaxyproject/galaxy/blob/release_18.05/config/galaxy.yml... It existed in galaxy.ini too, which has since be replaced by galaxy.yml Peter On Tue, Jul 10, 2018 at 8:19 AM, Matthias Enders <m.enders@german-seed-alliance.de> wrote:
Dear Björn,
thanks for the very fast reply. Unfortunately, this part is not visible in our instance. (see screenshot). As you can see in the topmost bar, this is done, while being loged in as admin.
Mit freundlichen Grüßen
Matthias Enders -----------------------------------------------------------------------
GERMAN SEED ALLIANCE GmbH c/o Norddeutsche Pflanzenzucht Hans-Georg Lembke KG Hohenlieth, 24363 Holtsee Tel.: +49 (0)4351/ 736-189 Fax: + 49 (0)4351/ 736-271 Mobil: +49 (0)151/ 14247360
Email: m.enders@german-seed-alliance.de
Firmensitz Köln Amtsgericht Köln, HRB 73844
-----Ursprüngliche Nachricht----- Von: Björn Grüning <bjoern.gruening@gmail.com> Gesendet: Dienstag, 10. Juli 2018 09:15 An: Matthias Enders <m.enders@german-seed-alliance.de>; galaxy-dev@lists.galaxyproject.org Betreff: Re: [galaxy-dev] Expose Command Line in Galaxy Priorität: Hoch
Hi Matthias,
please have a look at the attached screenshot. Given that you are an admin and click on the info-symbol of the dataset you should see the commandline.
This is with a 18.05 Docker. Cheers, Bjoern
Am 10.07.2018 um 08:58 schrieb Matthias Enders:
Dear Galaxy Community,
is there any possibility to expose the command line call, generated within galaxy?
We searched the net and found this discussions: https://github.com/galaxyproject/galaxy/issues/3252 https://github.com/galaxyproject/galaxy/issues/2954
But both are on the exposure to non-admin users. Within our instance we see the command line call nor for admin, neither for normal users!
...
Dear all, finally a restart of the container solved the issue with the database/blast. Regarding the problem with the command line: For other tools, the command line showed up, but not for blastn, as the wrapper encountered the error before creating the command line (this is my interpretation). So the error: "An invalid option was selected for database, 'myCoolDatabaseName', please verify." Was produced by the wrapper, which did not find the new database before restarting the galaxy container. So the real issue here is perhaps more on "How could blast+ be made aware of new integrated databases, without restarting"? Mit freundlichen Grüßen Matthias Enders ----------------------------------------------------------------------- GERMAN SEED ALLIANCE GmbH c/o Norddeutsche Pflanzenzucht Hans-Georg Lembke KG Hohenlieth, 24363 Holtsee Tel.: +49 (0)4351/ 736-189 Fax: + 49 (0)4351/ 736-271 Mobil: +49 (0)151/ 14247360 Email: m.enders@german-seed-alliance.de Firmensitz Köln Amtsgericht Köln, HRB 73844 -----Ursprüngliche Nachricht----- Von: Peter Cock <p.j.a.cock@googlemail.com> Gesendet: Dienstag, 10. Juli 2018 12:11 An: Matthias Enders <m.enders@german-seed-alliance.de> Cc: Björn Grüning <bjoern.gruening@gmail.com>; galaxy-dev@lists.galaxyproject.org Betreff: Re: [galaxy-dev] Expose Command Line in Galaxy There is a setting to enable showing the command line string and full paths on disk via the "i" icon from a history entry, see https://docs.galaxyproject.org/en/master/admin/options.html#expose-dataset-p... # This option allows users to see the full path of datasets via the # "View Details" option in the history. This option also exposes the # command line to non-administrative users. Administrators can always # see dataset paths. #expose_dataset_path: false https://github.com/galaxyproject/galaxy/blob/release_18.05/config/galaxy.yml... It existed in galaxy.ini too, which has since be replaced by galaxy.yml Peter On Tue, Jul 10, 2018 at 8:19 AM, Matthias Enders <m.enders@german-seed-alliance.de> wrote:
Dear Björn,
thanks for the very fast reply. Unfortunately, this part is not visible in our instance. (see screenshot). As you can see in the topmost bar, this is done, while being loged in as admin.
Mit freundlichen Grüßen
Matthias Enders ---------------------------------------------------------------------- -
GERMAN SEED ALLIANCE GmbH c/o Norddeutsche Pflanzenzucht Hans-Georg Lembke KG Hohenlieth, 24363 Holtsee Tel.: +49 (0)4351/ 736-189 Fax: + 49 (0)4351/ 736-271 Mobil: +49 (0)151/ 14247360
Email: m.enders@german-seed-alliance.de
Firmensitz Köln Amtsgericht Köln, HRB 73844
-----Ursprüngliche Nachricht----- Von: Björn Grüning <bjoern.gruening@gmail.com> Gesendet: Dienstag, 10. Juli 2018 09:15 An: Matthias Enders <m.enders@german-seed-alliance.de>; galaxy-dev@lists.galaxyproject.org Betreff: Re: [galaxy-dev] Expose Command Line in Galaxy Priorität: Hoch
Hi Matthias,
please have a look at the attached screenshot. Given that you are an admin and click on the info-symbol of the dataset you should see the commandline.
This is with a 18.05 Docker. Cheers, Bjoern
Am 10.07.2018 um 08:58 schrieb Matthias Enders:
Dear Galaxy Community,
is there any possibility to expose the command line call, generated within galaxy?
We searched the net and found this discussions: https://github.com/galaxyproject/galaxy/issues/3252 https://github.com/galaxyproject/galaxy/issues/2954
But both are on the exposure to non-admin users. Within our instance we see the command line call nor for admin, neither for normal users!
...
Thanks for getting back to us - this was actually my guess from the original email - that it was not a blastn error message, but a Galaxy message before ever calling BLAST. My guess is you are seeing this issue, or something like it, with having to restart Galaxy to reload the *.loc files? https://github.com/galaxyproject/galaxy/issues/3171 Peter On Tue, Jul 10, 2018 at 12:22 PM, Matthias Enders <m.enders@german-seed-alliance.de> wrote:
Dear all,
finally a restart of the container solved the issue with the database/blast.
Regarding the problem with the command line: For other tools, the command line showed up, but not for blastn, as the wrapper encountered the error before creating the command line (this is my interpretation).
So the error: "An invalid option was selected for database, 'myCoolDatabaseName', please verify." Was produced by the wrapper, which did not find the new database before restarting the galaxy container.
So the real issue here is perhaps more on "How could blast+ be made aware of new integrated databases, without restarting"?
Mit freundlichen Grüßen
Matthias Enders
Dear Peter, up to our galaxy admin, we make use of the Data Manager, so this seems not to be the issue here?: [cid:image001.jpg@01D41868.AAB17910] Mit freundlichen Grüßen Matthias Enders ----------------------------------------------------------------------- GERMAN SEED ALLIANCE GmbH c/o Norddeutsche Pflanzenzucht Hans-Georg Lembke KG Hohenlieth, 24363 Holtsee Tel.: +49 (0)4351/ 736-189 Fax: + 49 (0)4351/ 736-271 Mobil: +49 (0)151/ 14247360 Email: m.enders@german-seed-alliance.de Firmensitz Köln Amtsgericht Köln, HRB 73844 -----Ursprüngliche Nachricht----- Von: Peter Cock <p.j.a.cock@googlemail.com> Gesendet: Dienstag, 10. Juli 2018 13:27 An: Matthias Enders <m.enders@german-seed-alliance.de> Cc: Björn Grüning <bjoern.gruening@gmail.com>; galaxy-dev@lists.galaxyproject.org Betreff: Re: [galaxy-dev] Expose Command Line in Galaxy Priorität: Hoch Thanks for getting back to us - this was actually my guess from the original email - that it was not a blastn error message, but a Galaxy message before ever calling BLAST. My guess is you are seeing this issue, or something like it, with having to restart Galaxy to reload the *.loc files? https://github.com/galaxyproject/galaxy/issues/3171 Peter On Tue, Jul 10, 2018 at 12:22 PM, Matthias Enders <m.enders@german-seed-alliance.de<mailto:m.enders@german-seed-alliance.de>> wrote:
Dear all,
finally a restart of the container solved the issue with the database/blast.
Regarding the problem with the command line:
For other tools, the command line showed up, but not for blastn, as the wrapper encountered the error before creating the command line (this is my interpretation).
So the error: "An invalid option was selected for database, 'myCoolDatabaseName', please verify." Was produced by the wrapper, which did not find the new database before restarting the galaxy container.
So the real issue here is perhaps more on "How could blast+ be made aware of new integrated databases, without restarting"?
Mit freundlichen Grüßen
Matthias Enders
Is there anything in the Admin's interface to let you reload the BLAST *.loc files? If yes, did you try it but it didn't work? What version of Galaxy do you have? Peter On Tue, Jul 10, 2018 at 3:11 PM, Matthias Enders < m.enders@german-seed-alliance.de> wrote:
Dear Peter,
up to our galaxy admin, we make use of the Data Manager, so this seems not to be the issue here?:
Mit freundlichen Grüßen
Matthias Enders
Dear Peter, yes we did reload the Blast *.loc files using the admin interface. Our Version is 18.05 (based on the Galaxy – Docker image). Mit freundlichen Grüßen Matthias Enders ----------------------------------------------------------------------- [cid:image001.png@01D03960.8254BEE0] GERMAN SEED ALLIANCE GmbH c/o Norddeutsche Pflanzenzucht Hans-Georg Lembke KG Hohenlieth, 24363 Holtsee Tel.: +49 (0)4351/ 736-189 Fax: + 49 (0)4351/ 736-271 Mobil: +49 (0)151/ 14247360 Email: m.enders@german-seed-alliance.de<mailto:m.enders@german-seed-alliance.de> Firmensitz Köln Amtsgericht Köln, HRB 73844 Von: Peter Cock <p.j.a.cock@googlemail.com> Gesendet: Mittwoch, 11. Juli 2018 16:01 An: Matthias Enders <m.enders@german-seed-alliance.de> Cc: Björn Grüning <bjoern.gruening@gmail.com>; galaxy-dev@lists.galaxyproject.org Betreff: Re: [galaxy-dev] Expose Command Line in Galaxy Is there anything in the Admin's interface to let you reload the BLAST *.loc files? If yes, did you try it but it didn't work? What version of Galaxy do you have? Peter On Tue, Jul 10, 2018 at 3:11 PM, Matthias Enders <m.enders@german-seed-alliance.de<mailto:m.enders@german-seed-alliance.de>> wrote: Dear Peter, up to our galaxy admin, we make use of the Data Manager, so this seems not to be the issue here?: [cid:image003.jpg@01D41931.60604200] Mit freundlichen Grüßen Matthias Enders
Good. So is everything resolved now? Peter On Wed, Jul 11, 2018 at 3:08 PM, Matthias Enders < m.enders@german-seed-alliance.de> wrote:
Dear Peter,
yes we did reload the Blast *.loc files using the admin interface. Our Version is 18.05 (based on the Galaxy – Docker image).
Mit freundlichen Grüßen
Matthias Enders
Dear Peter, the reloading of the blast *.loc files did not solve the Problem. Actually only a complete restart of the container solved the problem for us. Mit freundlichen Grüßen Matthias Enders ----------------------------------------------------------------------- [cid:image001.png@01D03960.8254BEE0] GERMAN SEED ALLIANCE GmbH c/o Norddeutsche Pflanzenzucht Hans-Georg Lembke KG Hohenlieth, 24363 Holtsee Tel.: +49 (0)4351/ 736-189 Fax: + 49 (0)4351/ 736-271 Mobil: +49 (0)151/ 14247360 Email: m.enders@german-seed-alliance.de<mailto:m.enders@german-seed-alliance.de> Firmensitz Köln Amtsgericht Köln, HRB 73844 Von: Peter Cock <p.j.a.cock@googlemail.com> Gesendet: Mittwoch, 11. Juli 2018 16:12 An: Matthias Enders <m.enders@german-seed-alliance.de> Cc: Björn Grüning <bjoern.gruening@gmail.com>; galaxy-dev@lists.galaxyproject.org; f.kaiser@npz.de Betreff: Re: [galaxy-dev] Expose Command Line in Galaxy Good. So is everything resolved now? Peter On Wed, Jul 11, 2018 at 3:08 PM, Matthias Enders <m.enders@german-seed-alliance.de<mailto:m.enders@german-seed-alliance.de>> wrote: Dear Peter, yes we did reload the Blast *.loc files using the admin interface. Our Version is 18.05 (based on the Galaxy – Docker image). Mit freundlichen Grüßen Matthias Enders
You may have found a new Galaxy bug then :( Could you log an issue here: https://github.com/galaxyproject/galaxy/issues (From what you've said, I don't think that this is a problem in the BLAST+ wrappers themselves) Peter On Wed, Jul 11, 2018 at 3:20 PM, Matthias Enders < m.enders@german-seed-alliance.de> wrote:
Dear Peter,
the reloading of the blast *.loc files did not solve the Problem. Actually only a complete restart of the container solved the problem for us.
Mit freundlichen Grüßen
Matthias Enders
Dear Peter, we will re-create this problem with a new database in our instance, (to be sure on the reproducibility) and then raise an issue. Mit freundlichen Grüßen Matthias Enders ----------------------------------------------------------------------- [cid:image001.png@01D03960.8254BEE0] GERMAN SEED ALLIANCE GmbH c/o Norddeutsche Pflanzenzucht Hans-Georg Lembke KG Hohenlieth, 24363 Holtsee Tel.: +49 (0)4351/ 736-189 Fax: + 49 (0)4351/ 736-271 Mobil: +49 (0)151/ 14247360 Email: m.enders@german-seed-alliance.de<mailto:m.enders@german-seed-alliance.de> Firmensitz Köln Amtsgericht Köln, HRB 73844 Von: Peter Cock <p.j.a.cock@googlemail.com> Gesendet: Mittwoch, 11. Juli 2018 16:42 An: Matthias Enders <m.enders@german-seed-alliance.de> Cc: Björn Grüning <bjoern.gruening@gmail.com>; galaxy-dev@lists.galaxyproject.org; f.kaiser@npz.de Betreff: Re: [galaxy-dev] Expose Command Line in Galaxy You may have found a new Galaxy bug then :( Could you log an issue here: https://github.com/galaxyproject/galaxy/issues (From what you've said, I don't think that this is a problem in the BLAST+ wrappers themselves) Peter On Wed, Jul 11, 2018 at 3:20 PM, Matthias Enders <m.enders@german-seed-alliance.de<mailto:m.enders@german-seed-alliance.de>> wrote: Dear Peter, the reloading of the blast *.loc files did not solve the Problem. Actually only a complete restart of the container solved the problem for us. Mit freundlichen Grüßen Matthias Enders
Hi Matthias If for whatever reasons the "i" (view details) icon doesn't work for you, you can use the Galaxy Report Webapp (for docker installations see: https://github.com/bgruening/docker-galaxy-stable#Galaxy-Report-Webapp ) as an alternative. You can find your (failed) job via "Jobs in error per day this month". Once you clicked on the 'job id' you will get the executed command line/ Hope this helps. Regards, Hans-Rudolf On 07/10/2018 08:58 AM, Matthias Enders wrote:
Dear Galaxy Community,
is there any possibility to expose the command line call, generated within galaxy?
We searched the net and found this discussions: https://github.com/galaxyproject/galaxy/issues/3252 https://github.com/galaxyproject/galaxy/issues/2954
But both are on the exposure to non-admin users. Within our instance we see the command line call nor for admin, neither for normal users!
The problem, behind the question is the error message: "An invalid option was selected for database, 'myCoolDatabaseName', please verify." In Blast (toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastn_wrapper/0.2.01)
Local (directly from command line, without Galaxy in between) the blast works fine.
We are using the 18.05 Docker Galaxy Version.
Thanks in advance!
Mit freundlichen Grüßen
Matthias Enders -----------------------------------------------------------------------
GERMAN SEED ALLIANCE GmbH c/o Norddeutsche Pflanzenzucht Hans-Georg Lembke KG Hohenlieth, 24363 Holtsee Tel.: +49 (0)4351/ 736-189 Fax: + 49 (0)4351/ 736-271 Mobil: +49 (0)151/ 14247360
Email: m.enders@german-seed-alliance.de
Firmensitz Köln Amtsgericht Köln, HRB 73844
-----Ursprüngliche Nachricht----- Von: galaxy-dev <galaxy-dev-bounces@lists.galaxyproject.org> Im Auftrag von galaxy-dev-request@lists.galaxyproject.org Gesendet: Montag, 9. Juli 2018 18:00 An: galaxy-dev@lists.galaxyproject.org Betreff: galaxy-dev Digest, Vol 145, Issue 3
Send galaxy-dev mailing list submissions to galaxy-dev@lists.galaxyproject.org
To subscribe or unsubscribe via the World Wide Web, visit https://lists.galaxyproject.org/listinfo/galaxy-dev or, via email, send a message with subject or body 'help' to galaxy-dev-request@lists.galaxyproject.org
You can reach the person managing the list at galaxy-dev-owner@lists.galaxyproject.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of galaxy-dev digest..."
HEY! This is important! If you reply to a thread in a digest, please 1. Change the subject of your response from "Galaxy-dev Digest Vol ..." to the original subject for the thread. 2. Strip out everything else in the digest that is not part of the thread you are responding to.
Why? 1. This will keep the subject meaningful. People will have some idea from the subject line if they should read it or not. 2. Not doing this greatly increases the number of emails that match search queries, but that aren't actually informative.
Today's Topics:
1. From Paste to uWSGI (Jochen Bick) 2. Re: From Paste to uWSGI (Peter Briggs) 3. Re: From Paste to uWSGI (Martin Čech)
----------------------------------------------------------------------
Message: 1 Date: Mon, 9 Jul 2018 13:49:23 +0200 From: Jochen Bick <jochen.bick@usys.ethz.ch> To: "galaxy-dev@lists.galaxyproject.org" <galaxy-dev@lists.galaxyproject.org> Subject: [galaxy-dev] From Paste to uWSGI Message-ID: <1a0f33ad-938c-64d5-8db8-28914a5ca07e@usys.ethz.ch> Content-Type: text/plain; charset="utf-8"; format=flowed
Hi all,
we would like to update to 18.05 and thereby also update our Paste Server.
https://docs.galaxyproject.org/en/latest/admin/scaling.html
Is there any tutorial on how to update from Paste to uWSGI?
Cheers Jochen
-- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 / LFW B 58.1 8092 Zurich, Switzerland Office: Eschikon 27 8315 Lindau, Switzerland
Phone +41 52 354 92 06 jochen.bick@usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch
------------------------------
Message: 2 Date: Mon, 9 Jul 2018 15:31:16 +0000 From: Peter Briggs <peter.briggs@manchester.ac.uk> To: Jochen Bick <jochen.bick@usys.ethz.ch>, "galaxy-dev@lists.galaxyproject.org" <galaxy-dev@lists.galaxyproject.org> Subject: Re: [galaxy-dev] From Paste to uWSGI Message-ID: <E20AB71CD352854B9E24B5830826BC43C8586657@MBXP15.ds.man.ac.uk> Content-Type: text/plain; charset="iso-8859-1"
Hello Jochen
I'm not aware of a tutorial but here are some observations from my (very) recent experiences upgrading a local paste-based 17.09 instance to a uwsgi-based 18.05 configuration. It's not comprehensive because the configuration will depend on decisions you need to make based on your local setup, but it might be helpful as a starting point.
Before beginning:
-- Commit to migrating from galaxy.ini to galaxy.yml (if not done previously)
Even though its possible to stick with the old galaxy.ini file, it seems best to move to the new format - my feeling is, probably the migration will have to happen at some point anyway, and also the documentation is more focussed on the newer galaxy.yml version.
-- Decide on how you're going to handle job submission (essentially, how you're going to replace the old job handler processes).
There are a number of possibilities outlined in the documentation here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#deploymen...
It seems important to figure this out at the beginning because it impacts a lot of the configuration specifics that follow. For our setup case "uwsgi + mules" looked like the most appropriate:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#uwsgi-for...
Then once you're ready to start (nb backing up the database etc before starting are implicitly assumed!):
-- Update the Galaxy source code
The standard recipe for this seems to work okay i.e.
git fetch origin && git checkout release_18.05 && git pull --ff-only origin release_18.05
(You'll probably also need to run scripts/common_startup.sh to update the Python packages, and scripts/manage_db.py to update the database schema.)
-- Create galaxy.yml and configure the "galaxy" section:
You can make a galaxy.yml from the sample version and go through the "galaxy" section replicating the settings from galaxy.ini. (The settings in the "uwsgi" section can wait at this stage.) Don't forget to move galaxy.ini out of the way once you're done.
-- Configure the "uwsgi" section of galaxy.yml:
The options for various scenarios are given in the "Deployment/uwsgi" section:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#uwsgi
but the required settings seem to be spread out between different sections so (for me) it felt like it took some piecing together to get it to work when I first tried this.
(Also I was thrown a little by the "socket" parameter not having a placeholder in sample galaxy.yml.)
The core uwsgi options are here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#configura...
then you need to set other options dependent on:
1. how job submission is handled (e.g. options for the "uwsgi + mules" setup in my case) 2. how you're proxying Galaxy (for Apache or nginx it looks like it's just setting the uwsgi "socket" option, unless you're serving Galaxy from a URL prefix in which case there are a few more options needed - these are covered in the documentation.)
-- Fix your job_conf.xml file
If you have an existing job_conf.xml file in your config directory then you might need to update that too (specifically the "handlers" section) depending on the job submission handling you've decided to use.
-- Use supervisor to manage your uwsgi processes
The docs for this are here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#starting-...
Essentially supervisor replaces "run.sh", so this becomes how you start and stop Galaxy (e.g. "supervisorctl start galaxy:" replaces "GALAXY_RUN_ALL=1 sh run.sh --daemon".)
I think this is quite straightforward from the docs but again you need to tailor it for how you're handling job submission (e.g the "uwsgi + mules" doesn't need any handlers defining, but it looks like other strategies needs extra stuff defining here).
-- Set up the proxying
Assuming you're proxing via either Apache or nginx (we're using nginx), these are covered in the relevant sections:
https://docs.galaxyproject.org/en/release_18.05/admin/apache.html https://docs.galaxyproject.org/en/release_18.05/admin/nginx.html
The main difference I noticed for nginx is that you no longer need to use the "upstream"/"proxy_pass" directives, as nginx can talk directly to uwsgi using a "uwsgi_pass" directive that connects to the socket defined in the "uwsgi" section of galaxy.yml previously.
(There's also a "gotcha" if you're nginx configuration also tries to serve "static/scripts/packed", which needs to be removed - this is noted in the "breaking changes" for the 18.05 docs but is easy to miss.)
Hopefully this is some help. There's lots of documentation, which is a good thing but which can sometimes make it harder to see where to start.
Good luck with the update,
Best wishes
Peter
-- Peter Briggs peter.briggs@manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482
________________________________________ From: galaxy-dev [galaxy-dev-bounces@lists.galaxyproject.org] on behalf of Jochen Bick [jochen.bick@usys.ethz.ch] Sent: Monday, July 09, 2018 12:49 PM To: galaxy-dev@lists.galaxyproject.org Subject: [galaxy-dev] From Paste to uWSGI
Hi all,
we would like to update to 18.05 and thereby also update our Paste Server.
https://docs.galaxyproject.org/en/latest/admin/scaling.html
Is there any tutorial on how to update from Paste to uWSGI?
Cheers Jochen
-- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 / LFW B 58.1 8092 Zurich, Switzerland Office: Eschikon 27 8315 Lindau, Switzerland
Phone +41 52 354 92 06 jochen.bick@usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
------------------------------
Message: 3 Date: Mon, 9 Jul 2018 11:36:27 -0400 From: Martin Čech <marten@bx.psu.edu> To: Peter Briggs <peter.briggs@manchester.ac.uk> Cc: "galaxy-dev@lists.galaxyproject.org" <galaxy-dev@lists.galaxyproject.org> Subject: Re: [galaxy-dev] From Paste to uWSGI Message-ID: <CAHKBquGXMTog-MUgLbPWD2wgmXvpOxiQGRa4+QCWk5EAHGK3=g@mail.gmail.com> Content-Type: text/plain; charset="utf-8"
Another recently updated resource for uwsgi is the training slides: https://galaxyproject.github.io/dagobah-training/2018-gccbosc/10-uwsgi/uwsgi...
M.
On Mon, Jul 9, 2018 at 11:31 AM Peter Briggs <peter.briggs@manchester.ac.uk> wrote:
Hello Jochen
I'm not aware of a tutorial but here are some observations from my (very) recent experiences upgrading a local paste-based 17.09 instance to a uwsgi-based 18.05 configuration. It's not comprehensive because the configuration will depend on decisions you need to make based on your local setup, but it might be helpful as a starting point.
Before beginning:
-- Commit to migrating from galaxy.ini to galaxy.yml (if not done previously)
Even though its possible to stick with the old galaxy.ini file, it seems best to move to the new format - my feeling is, probably the migration will have to happen at some point anyway, and also the documentation is more focussed on the newer galaxy.yml version.
-- Decide on how you're going to handle job submission (essentially, how you're going to replace the old job handler processes).
There are a number of possibilities outlined in the documentation here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#dep loyment-options
It seems important to figure this out at the beginning because it impacts a lot of the configuration specifics that follow. For our setup case "uwsgi + mules" looked like the most appropriate:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#uws gi-for-web-serving-with-mules-as-job-handlers
Then once you're ready to start (nb backing up the database etc before starting are implicitly assumed!):
-- Update the Galaxy source code
The standard recipe for this seems to work okay i.e.
git fetch origin && git checkout release_18.05 && git pull --ff-only origin release_18.05
(You'll probably also need to run scripts/common_startup.sh to update the Python packages, and scripts/manage_db.py to update the database schema.)
-- Create galaxy.yml and configure the "galaxy" section:
You can make a galaxy.yml from the sample version and go through the "galaxy" section replicating the settings from galaxy.ini. (The settings in the "uwsgi" section can wait at this stage.) Don't forget to move galaxy.ini out of the way once you're done.
-- Configure the "uwsgi" section of galaxy.yml:
The options for various scenarios are given in the "Deployment/uwsgi" section:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#uws gi
but the required settings seem to be spread out between different sections so (for me) it felt like it took some piecing together to get it to work when I first tried this.
(Also I was thrown a little by the "socket" parameter not having a placeholder in sample galaxy.yml.)
The core uwsgi options are here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#con figuration-common-to-all-uwsgi-deployment-styles
then you need to set other options dependent on:
1. how job submission is handled (e.g. options for the "uwsgi + mules" setup in my case) 2. how you're proxying Galaxy (for Apache or nginx it looks like it's just setting the uwsgi "socket" option, unless you're serving Galaxy from a URL prefix in which case there are a few more options needed - these are covered in the documentation.)
-- Fix your job_conf.xml file
If you have an existing job_conf.xml file in your config directory then you might need to update that too (specifically the "handlers" section) depending on the job submission handling you've decided to use.
-- Use supervisor to manage your uwsgi processes
The docs for this are here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#sta rting-and-stopping
Essentially supervisor replaces "run.sh", so this becomes how you start and stop Galaxy (e.g. "supervisorctl start galaxy:" replaces "GALAXY_RUN_ALL=1 sh run.sh --daemon".)
I think this is quite straightforward from the docs but again you need to tailor it for how you're handling job submission (e.g the "uwsgi + mules" doesn't need any handlers defining, but it looks like other strategies needs extra stuff defining here).
-- Set up the proxying
Assuming you're proxing via either Apache or nginx (we're using nginx), these are covered in the relevant sections:
https://docs.galaxyproject.org/en/release_18.05/admin/apache.html https://docs.galaxyproject.org/en/release_18.05/admin/nginx.html
The main difference I noticed for nginx is that you no longer need to use the "upstream"/"proxy_pass" directives, as nginx can talk directly to uwsgi using a "uwsgi_pass" directive that connects to the socket defined in the "uwsgi" section of galaxy.yml previously.
(There's also a "gotcha" if you're nginx configuration also tries to serve "static/scripts/packed", which needs to be removed - this is noted in the "breaking changes" for the 18.05 docs but is easy to miss.)
Hopefully this is some help. There's lots of documentation, which is a good thing but which can sometimes make it harder to see where to start.
Good luck with the update,
Best wishes
Peter
-- Peter Briggs peter.briggs@manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482
________________________________________ From: galaxy-dev [galaxy-dev-bounces@lists.galaxyproject.org] on behalf of Jochen Bick [jochen.bick@usys.ethz.ch] Sent: Monday, July 09, 2018 12:49 PM To: galaxy-dev@lists.galaxyproject.org Subject: [galaxy-dev] From Paste to uWSGI
Hi all,
we would like to update to 18.05 and thereby also update our Paste Server.
https://docs.galaxyproject.org/en/latest/admin/scaling.html
Is there any tutorial on how to update from Paste to uWSGI?
Cheers Jochen
-- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 <https://maps.google.com/?q=Universit%C3%A4tstrasse+2&entry=gmail&sour ce=g> / LFW B 58.1 8092 Zurich, Switzerland Office: Eschikon 27 8315 Lindau, Switzerland
Phone +41 52 354 92 06 <+41%2052%20354%2092%2006> jochen.bick@usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/ ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
participants (4)
-
Björn Grüning -
Hans-Rudolf Hotz -
Matthias Enders -
Peter Cock