Hi all, we would like to update to 18.05 and thereby also update our Paste Server. https://docs.galaxyproject.org/en/latest/admin/scaling.html Is there any tutorial on how to update from Paste to uWSGI? Cheers Jochen -- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 / LFW B 58.1 8092 Zurich, Switzerland Office: Eschikon 27 8315 Lindau, Switzerland Phone +41 52 354 92 06 jochen.bick@usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch
Hello Jochen I'm not aware of a tutorial but here are some observations from my (very) recent experiences upgrading a local paste-based 17.09 instance to a uwsgi-based 18.05 configuration. It's not comprehensive because the configuration will depend on decisions you need to make based on your local setup, but it might be helpful as a starting point. Before beginning: -- Commit to migrating from galaxy.ini to galaxy.yml (if not done previously) Even though its possible to stick with the old galaxy.ini file, it seems best to move to the new format - my feeling is, probably the migration will have to happen at some point anyway, and also the documentation is more focussed on the newer galaxy.yml version. -- Decide on how you're going to handle job submission (essentially, how you're going to replace the old job handler processes). There are a number of possibilities outlined in the documentation here: https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#deploymen... It seems important to figure this out at the beginning because it impacts a lot of the configuration specifics that follow. For our setup case "uwsgi + mules" looked like the most appropriate: https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#uwsgi-for... Then once you're ready to start (nb backing up the database etc before starting are implicitly assumed!): -- Update the Galaxy source code The standard recipe for this seems to work okay i.e. git fetch origin && git checkout release_18.05 && git pull --ff-only origin release_18.05 (You'll probably also need to run scripts/common_startup.sh to update the Python packages, and scripts/manage_db.py to update the database schema.) -- Create galaxy.yml and configure the "galaxy" section: You can make a galaxy.yml from the sample version and go through the "galaxy" section replicating the settings from galaxy.ini. (The settings in the "uwsgi" section can wait at this stage.) Don't forget to move galaxy.ini out of the way once you're done. -- Configure the "uwsgi" section of galaxy.yml: The options for various scenarios are given in the "Deployment/uwsgi" section: https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#uwsgi but the required settings seem to be spread out between different sections so (for me) it felt like it took some piecing together to get it to work when I first tried this. (Also I was thrown a little by the "socket" parameter not having a placeholder in sample galaxy.yml.) The core uwsgi options are here: https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#configura... then you need to set other options dependent on: 1. how job submission is handled (e.g. options for the "uwsgi + mules" setup in my case) 2. how you're proxying Galaxy (for Apache or nginx it looks like it's just setting the uwsgi "socket" option, unless you're serving Galaxy from a URL prefix in which case there are a few more options needed - these are covered in the documentation.) -- Fix your job_conf.xml file If you have an existing job_conf.xml file in your config directory then you might need to update that too (specifically the "handlers" section) depending on the job submission handling you've decided to use. -- Use supervisor to manage your uwsgi processes The docs for this are here: https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#starting-... Essentially supervisor replaces "run.sh", so this becomes how you start and stop Galaxy (e.g. "supervisorctl start galaxy:" replaces "GALAXY_RUN_ALL=1 sh run.sh --daemon".) I think this is quite straightforward from the docs but again you need to tailor it for how you're handling job submission (e.g the "uwsgi + mules" doesn't need any handlers defining, but it looks like other strategies needs extra stuff defining here). -- Set up the proxying Assuming you're proxing via either Apache or nginx (we're using nginx), these are covered in the relevant sections: https://docs.galaxyproject.org/en/release_18.05/admin/apache.html https://docs.galaxyproject.org/en/release_18.05/admin/nginx.html The main difference I noticed for nginx is that you no longer need to use the "upstream"/"proxy_pass" directives, as nginx can talk directly to uwsgi using a "uwsgi_pass" directive that connects to the socket defined in the "uwsgi" section of galaxy.yml previously. (There's also a "gotcha" if you're nginx configuration also tries to serve "static/scripts/packed", which needs to be removed - this is noted in the "breaking changes" for the 18.05 docs but is easy to miss.) Hopefully this is some help. There's lots of documentation, which is a good thing but which can sometimes make it harder to see where to start. Good luck with the update, Best wishes Peter -- Peter Briggs peter.briggs@manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482 ________________________________________ From: galaxy-dev [galaxy-dev-bounces@lists.galaxyproject.org] on behalf of Jochen Bick [jochen.bick@usys.ethz.ch] Sent: Monday, July 09, 2018 12:49 PM To: galaxy-dev@lists.galaxyproject.org Subject: [galaxy-dev] From Paste to uWSGI Hi all, we would like to update to 18.05 and thereby also update our Paste Server. https://docs.galaxyproject.org/en/latest/admin/scaling.html Is there any tutorial on how to update from Paste to uWSGI? Cheers Jochen -- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 / LFW B 58.1 8092 Zurich, Switzerland Office: Eschikon 27 8315 Lindau, Switzerland Phone +41 52 354 92 06 jochen.bick@usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Another recently updated resource for uwsgi is the training slides: https://galaxyproject.github.io/dagobah-training/2018-gccbosc/10-uwsgi/uwsgi... M. On Mon, Jul 9, 2018 at 11:31 AM Peter Briggs <peter.briggs@manchester.ac.uk> wrote:
Hello Jochen
I'm not aware of a tutorial but here are some observations from my (very) recent experiences upgrading a local paste-based 17.09 instance to a uwsgi-based 18.05 configuration. It's not comprehensive because the configuration will depend on decisions you need to make based on your local setup, but it might be helpful as a starting point.
Before beginning:
-- Commit to migrating from galaxy.ini to galaxy.yml (if not done previously)
Even though its possible to stick with the old galaxy.ini file, it seems best to move to the new format - my feeling is, probably the migration will have to happen at some point anyway, and also the documentation is more focussed on the newer galaxy.yml version.
-- Decide on how you're going to handle job submission (essentially, how you're going to replace the old job handler processes).
There are a number of possibilities outlined in the documentation here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#deploymen...
It seems important to figure this out at the beginning because it impacts a lot of the configuration specifics that follow. For our setup case "uwsgi + mules" looked like the most appropriate:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#uwsgi-for...
Then once you're ready to start (nb backing up the database etc before starting are implicitly assumed!):
-- Update the Galaxy source code
The standard recipe for this seems to work okay i.e.
git fetch origin && git checkout release_18.05 && git pull --ff-only origin release_18.05
(You'll probably also need to run scripts/common_startup.sh to update the Python packages, and scripts/manage_db.py to update the database schema.)
-- Create galaxy.yml and configure the "galaxy" section:
You can make a galaxy.yml from the sample version and go through the "galaxy" section replicating the settings from galaxy.ini. (The settings in the "uwsgi" section can wait at this stage.) Don't forget to move galaxy.ini out of the way once you're done.
-- Configure the "uwsgi" section of galaxy.yml:
The options for various scenarios are given in the "Deployment/uwsgi" section:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#uwsgi
but the required settings seem to be spread out between different sections so (for me) it felt like it took some piecing together to get it to work when I first tried this.
(Also I was thrown a little by the "socket" parameter not having a placeholder in sample galaxy.yml.)
The core uwsgi options are here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#configura...
then you need to set other options dependent on:
1. how job submission is handled (e.g. options for the "uwsgi + mules" setup in my case) 2. how you're proxying Galaxy (for Apache or nginx it looks like it's just setting the uwsgi "socket" option, unless you're serving Galaxy from a URL prefix in which case there are a few more options needed - these are covered in the documentation.)
-- Fix your job_conf.xml file
If you have an existing job_conf.xml file in your config directory then you might need to update that too (specifically the "handlers" section) depending on the job submission handling you've decided to use.
-- Use supervisor to manage your uwsgi processes
The docs for this are here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#starting-...
Essentially supervisor replaces "run.sh", so this becomes how you start and stop Galaxy (e.g. "supervisorctl start galaxy:" replaces "GALAXY_RUN_ALL=1 sh run.sh --daemon".)
I think this is quite straightforward from the docs but again you need to tailor it for how you're handling job submission (e.g the "uwsgi + mules" doesn't need any handlers defining, but it looks like other strategies needs extra stuff defining here).
-- Set up the proxying
Assuming you're proxing via either Apache or nginx (we're using nginx), these are covered in the relevant sections:
https://docs.galaxyproject.org/en/release_18.05/admin/apache.html https://docs.galaxyproject.org/en/release_18.05/admin/nginx.html
The main difference I noticed for nginx is that you no longer need to use the "upstream"/"proxy_pass" directives, as nginx can talk directly to uwsgi using a "uwsgi_pass" directive that connects to the socket defined in the "uwsgi" section of galaxy.yml previously.
(There's also a "gotcha" if you're nginx configuration also tries to serve "static/scripts/packed", which needs to be removed - this is noted in the "breaking changes" for the 18.05 docs but is easy to miss.)
Hopefully this is some help. There's lots of documentation, which is a good thing but which can sometimes make it harder to see where to start.
Good luck with the update,
Best wishes
Peter
-- Peter Briggs peter.briggs@manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482
________________________________________ From: galaxy-dev [galaxy-dev-bounces@lists.galaxyproject.org] on behalf of Jochen Bick [jochen.bick@usys.ethz.ch] Sent: Monday, July 09, 2018 12:49 PM To: galaxy-dev@lists.galaxyproject.org Subject: [galaxy-dev] From Paste to uWSGI
Hi all,
we would like to update to 18.05 and thereby also update our Paste Server.
https://docs.galaxyproject.org/en/latest/admin/scaling.html
Is there any tutorial on how to update from Paste to uWSGI?
Cheers Jochen
-- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 <https://maps.google.com/?q=Universit%C3%A4tstrasse+2&entry=gmail&source=g> / LFW B 58.1 8092 Zurich, Switzerland Office: Eschikon 27 8315 Lindau, Switzerland
Phone +41 52 354 92 06 <+41%2052%20354%2092%2006> jochen.bick@usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/ ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Thanks I think that helps! Cheers Jochen On 09.07.2018 17:36, Martin Čech wrote:
Another recently updated resource for uwsgi is the training slides: https://galaxyproject.github.io/dagobah-training/2018-gccbosc/10-uwsgi/uwsgi...
M.
On Mon, Jul 9, 2018 at 11:31 AM Peter Briggs <peter.briggs@manchester.ac.uk> wrote:
Hello Jochen
I'm not aware of a tutorial but here are some observations from my (very) recent experiences upgrading a local paste-based 17.09 instance to a uwsgi-based 18.05 configuration. It's not comprehensive because the configuration will depend on decisions you need to make based on your local setup, but it might be helpful as a starting point.
Before beginning:
-- Commit to migrating from galaxy.ini to galaxy.yml (if not done previously)
Even though its possible to stick with the old galaxy.ini file, it seems best to move to the new format - my feeling is, probably the migration will have to happen at some point anyway, and also the documentation is more focussed on the newer galaxy.yml version.
-- Decide on how you're going to handle job submission (essentially, how you're going to replace the old job handler processes).
There are a number of possibilities outlined in the documentation here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#deploymen...
It seems important to figure this out at the beginning because it impacts a lot of the configuration specifics that follow. For our setup case "uwsgi + mules" looked like the most appropriate:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#uwsgi-for...
Then once you're ready to start (nb backing up the database etc before starting are implicitly assumed!):
-- Update the Galaxy source code
The standard recipe for this seems to work okay i.e.
git fetch origin && git checkout release_18.05 && git pull --ff-only origin release_18.05
(You'll probably also need to run scripts/common_startup.sh to update the Python packages, and scripts/manage_db.py to update the database schema.)
-- Create galaxy.yml and configure the "galaxy" section:
You can make a galaxy.yml from the sample version and go through the "galaxy" section replicating the settings from galaxy.ini. (The settings in the "uwsgi" section can wait at this stage.) Don't forget to move galaxy.ini out of the way once you're done.
-- Configure the "uwsgi" section of galaxy.yml:
The options for various scenarios are given in the "Deployment/uwsgi" section:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#uwsgi
but the required settings seem to be spread out between different sections so (for me) it felt like it took some piecing together to get it to work when I first tried this.
(Also I was thrown a little by the "socket" parameter not having a placeholder in sample galaxy.yml.)
The core uwsgi options are here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#configura...
then you need to set other options dependent on:
1. how job submission is handled (e.g. options for the "uwsgi + mules" setup in my case) 2. how you're proxying Galaxy (for Apache or nginx it looks like it's just setting the uwsgi "socket" option, unless you're serving Galaxy from a URL prefix in which case there are a few more options needed - these are covered in the documentation.)
-- Fix your job_conf.xml file
If you have an existing job_conf.xml file in your config directory then you might need to update that too (specifically the "handlers" section) depending on the job submission handling you've decided to use.
-- Use supervisor to manage your uwsgi processes
The docs for this are here:
https://docs.galaxyproject.org/en/release_18.05/admin/scaling.html#starting-...
Essentially supervisor replaces "run.sh", so this becomes how you start and stop Galaxy (e.g. "supervisorctl start galaxy:" replaces "GALAXY_RUN_ALL=1 sh run.sh --daemon".)
I think this is quite straightforward from the docs but again you need to tailor it for how you're handling job submission (e.g the "uwsgi + mules" doesn't need any handlers defining, but it looks like other strategies needs extra stuff defining here).
-- Set up the proxying
Assuming you're proxing via either Apache or nginx (we're using nginx), these are covered in the relevant sections:
https://docs.galaxyproject.org/en/release_18.05/admin/apache.html https://docs.galaxyproject.org/en/release_18.05/admin/nginx.html
The main difference I noticed for nginx is that you no longer need to use the "upstream"/"proxy_pass" directives, as nginx can talk directly to uwsgi using a "uwsgi_pass" directive that connects to the socket defined in the "uwsgi" section of galaxy.yml previously.
(There's also a "gotcha" if you're nginx configuration also tries to serve "static/scripts/packed", which needs to be removed - this is noted in the "breaking changes" for the 18.05 docs but is easy to miss.)
Hopefully this is some help. There's lots of documentation, which is a good thing but which can sometimes make it harder to see where to start.
Good luck with the update,
Best wishes
Peter
-- Peter Briggs peter.briggs@manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482
________________________________________ From: galaxy-dev [galaxy-dev-bounces@lists.galaxyproject.org] on behalf of Jochen Bick [jochen.bick@usys.ethz.ch] Sent: Monday, July 09, 2018 12:49 PM To: galaxy-dev@lists.galaxyproject.org Subject: [galaxy-dev] From Paste to uWSGI
Hi all,
we would like to update to 18.05 and thereby also update our Paste Server.
https://docs.galaxyproject.org/en/latest/admin/scaling.html
Is there any tutorial on how to update from Paste to uWSGI?
Cheers Jochen
-- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 <https://maps.google.com/?q=Universit%C3%A4tstrasse+2&entry=gmail&source=g> / LFW B 58.1 8092 Zurich, Switzerland Office: Eschikon 27 8315 Lindau, Switzerland
Phone +41 52 354 92 06 <+41%2052%20354%2092%2006> jochen.bick@usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/ ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
-- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 / LFW B 58.1 Office: Tannenstrasse 1 / TAN D 6.2 8092 Zurich, Switzerland Phone +41 44 632 28 25 jochen.bick@usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch
participants (3)
-
Jochen Bick -
Martin Čech -
Peter Briggs