Hi Nate,
So I went ahead and set `static_enabled = False` in universe_wsgi.ini, restarted uWSGI, and browsed to localhost. I'm including the results of each step below:
After restarting uWSGI, the last few lines of uwsgi.log looked like this:
WSGI app 0 (mountpoint='') ready in 11 seconds on interpreter 0xa934c0 pid: 3561 (default app) *** uWSGI is running in multiple interpreter mode *** spawned uWSGI master process (pid: 3561) spawned uWSGI worker 1 (pid: 3575, cores: 4) spawned uWSGI worker 2 (pid: 3576, cores: 4) spawned uWSGI worker 3 (pid: 3583, cores: 4) spawned uWSGI worker 4 (pid: 3587, cores: 4) spawned uWSGI worker 5 (pid: 3591, cores: 4) spawned uWSGI worker 6 (pid: 3595, cores: 4) spawned uWSGI worker 7 (pid: 3599, cores: 4) spawned uWSGI worker 8 (pid: 3603, cores: 4) spawned uWSGI worker 9 (pid: 3607, cores: 4) spawned uWSGI worker 10 (pid: 3611, cores: 4) *** Stats server enabled on 127.0.0.1:9191 fd: 30 ***
...so no traceback this time!
Then, browsing to localhost redirects to https://localhost and finally returns:
"Internal Server Error Galaxy was unable to successfully complete your request
An error occurred. This may be an intermittent problem due to load or other unpredictable factors, reloading the page may address the problem.
The error has been logged to our team."
That request was logged to uwsgi.log:
127.0.0.1 - - [03/Feb/2015:14:51:01 -0400] "GET / HTTP/1.1" 500 - "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:35.0) Gecko/20100101 Firefox/35.0" Error - <type 'exceptions.KeyError'>: 'SCRIPT_NAME' URL: https://localhost/ File 'lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File 'lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File 'lib/galaxy/web/framework/base.py', line 159 in handle_request trans = self.transaction_factory( environ ) File 'lib/galaxy/web/framework/webapp.py', line 71 in <lambda> self.set_transaction_factory( lambda e: self.transaction_chooser( e, galaxy_app, session_cookie ) ) File 'lib/galaxy/web/framework/webapp.py', line 102 in transaction_chooser return GalaxyWebTransaction( environ, galaxy_app, self, session_cookie ) File 'lib/galaxy/web/framework/webapp.py', line 207 in __init__ self._ensure_logged_in_user( environ, session_cookie ) File 'lib/galaxy/web/framework/webapp.py', line 441 in _ensure_logged_in_user if self.request.path.startswith( external_display_path ): File 'lib/galaxy/web/framework/base.py', line 258 in __get__ value = self.func( obj ) File 'lib/galaxy/web/framework/base.py', line 341 in path return self.environ['SCRIPT_NAME'] + self.environ['PATH_INFO'] KeyError: 'SCRIPT_NAME'
CGI Variables ------------- DOCUMENT_ROOT: '/usr/share/nginx/html' HTTP_ACCEPT: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' HTTP_ACCEPT_ENCODING: 'gzip, deflate' HTTP_ACCEPT_LANGUAGE: 'en-US,en;q=0.5' HTTP_CONNECTION: 'keep-alive' HTTP_HOST: 'localhost' HTTP_USER_AGENT: 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:35.0) Gecko/20100101 Firefox/35.0' PATH_INFO: '/' REMOTE_ADDR: '127.0.0.1' REMOTE_PORT: '50064' REQUEST_METHOD: 'GET' REQUEST_URI: '/' SERVER_NAME: 'localhost' SERVER_PORT: '443' SERVER_PROTOCOL: 'HTTP/1.1' UWSGI_SCHEME: 'https'
WSGI Variables -------------- application: <paste.recursive.RecursiveMiddleware object at 0x7f2ec2f32290> is_api_request: False paste.cookies: (<SimpleCookie: >, '') paste.expected_exceptions: [<class 'paste.httpexceptions.HTTPException'>] paste.httpexceptions: <paste.httpexceptions.HTTPExceptionHandler object at 0x7f2ec2f32210> paste.recursive.forward: <paste.recursive.Forwarder from /> paste.recursive.include: <paste.recursive.Includer from /> paste.recursive.include_app_iter: <paste.recursive.IncluderAppIter from /> paste.recursive.script_name: '' paste.throw_errors: True request_id: 'fadecee2abdd11e4bfb614feb5e02dd6' uwsgi.core: 0 uwsgi.node: 'i changed this string to hide the server name' uwsgi.version: '2.0.9' wsgi process: 'Multi process AND threads (?)' wsgi.file_wrapper: <built-in function uwsgi_sendfile> ------------------------------------------------------------ [pid: 3607|app: 0|req: 1/1] 127.0.0.1 () {40 vars in 593 bytes} [Tue Feb 3 14:51:01 2015] GET / => generated 777 bytes in 230 msecs (HTTP/1.1 500) 1 headers in 63 bytes (1 switches on core 0)
I thought I would include some pieces of universe_wsgi.ini to help with troubleshooting:
# ---- HTTP Server ----------------------------------------------------------
# Configuration of the internal HTTP server.
[server:main]
# The internal HTTP server to use. Currently only Paste is provided. This # option is required. use = egg:Paste#http
# The port on which to listen. #port = 8080
# The address on which to listen. By default, only listen to localhost (Galaxy # will not be accessible over the network). Use '0.0.0.0' to listen on all # available network interfaces. host = 0.0.0.0
# Use a threadpool for the web server instead of creating a thread for each # request. use_threadpool = True
# Number of threads in the web server thread pool. threadpool_workers = 10
# Set the number of seconds a thread can work before you should kill it (assuming it will never finish) to 3 hours. threadpool_kill_thread_limit = 10800
# ---- uwsgi ----------------------------------------------------------
[uwsgi] processes = 10 stats = 127.0.0.1:9191 socket = 127.0.0.1:4001 pythonpath = lib threads = 4 logto = /home/galaxy/galaxy-dist/uwsgi.log master = True
Also, we are using a self-signed certificate as a temporary solution, so not sure if that matters.
Thanks!
On Mon, Feb 2, 2015 at 11:32 AM, Nate Coraor nate@bx.psu.edu wrote:
Hi Nicholas,
Set `static_enabled = False` in universe_wsgi.ini and restart uWSGI, and see if that makes a difference.
Thanks, --nate
On Mon, Feb 2, 2015 at 11:26 AM, Nicholas Kline nxk60@case.edu wrote:
Hi Nate. Thanks a lot for all the help so far. So, I started uWSGI like so:
- activate virtual env - cd ~/galaxy-dist - PYTHONPATH=eggs/PasteDeploy-1.5.0-py2.7.egg uwsgi --ini-pasteuniverse_wsgi.ini
Then browsed to localhost, which redirected to https://localhost/ as expected since I want all traffic to be over HTTPS, and finally returned:
502 Bad GatewayI checked uwsgi.log and it's quite large, almost 3,000 lines. Should I post it to this list for review? I've included the last few lines below:
*** uWSGI is running in multiple interpreter mode *** spawned uWSGI master process (pid: 4083) spawned uWSGI worker 1 (pid: 4195, cores: 4) spawned uWSGI worker 2 (pid: 4199, cores: 4) spawned uWSGI worker 3 (pid: 4203, cores: 4) spawned uWSGI worker 4 (pid: 4207, cores: 4) spawned uWSGI worker 5 (pid: 4211, cores: 4) spawned uWSGI worker 6 (pid: 4215, cores: 4) spawned uWSGI worker 7 (pid: 4219, cores: 4) spawned uWSGI worker 8 (pid: 4223, cores: 4) spawned uWSGI worker 9 (pid: 4224, cores: 4) spawned uWSGI worker 10 (pid: 4225, cores: 4) *** Stats server enabled on 127.0.0.1:9191 fd: 30 *** Traceback (most recent call last): File "/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/urlmap.py", line 201, in __call__ environ['SCRIPT_NAME'] += app_url KeyError: 'SCRIPT_NAME' [pid: 4219|app: 0|req: 1/1] 127.0.0.1 () {40 vars in 593 bytes} [Fri Jan 30 15:15:10 2015] GET / => generated 0 bytes in 18 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0)
If you need me to post my Nginx config too, just let me know. Thanks!
On Wed, Jan 28, 2015 at 2:53 PM, Nate Coraor nate@bx.psu.edu wrote:
On Wed, Jan 28, 2015 at 11:55 AM, Nicholas Kline nxk60@case.edu wrote:
Hi Nate. Thank you for the reply. So I installed uWSGI inside my Python virtual environment since that was one of the options listed in the documentation, like so:
cd ~/galaxy_env/ source bin/activate pip install uwsgiIt looks like uWSGI was successfully installed. I then exited/deactivated the virtual environment via 'deactivate'.
Next, I tried to start uWSGI:
~/galaxy-dist$ PYTHONPATH=eggs/PasteDeploy-1.5.0-py2.7.egg uwsgi--ini-paste config/galaxy.ini
Calling uwsgi after deactivating the virtualenv means it'd be running from wherever it's found on $PATH, which is probably not the one you just installed into the virtualenv. But if it works, it may not matter.
... but it returned this:
realpath() of config/galaxy.ini failed: No such file or directory[core/utils.c line 3607]
So, I pointed it to universe_wsgi.ini instead, like so:
~/galaxy-dist$ PYTHONPATH=eggs/PasteDeploy-1.5.0-py2.7.egg uwsgi--ini-paste universe_wsgi.ini
universe_wsgi.ini was renamed to config/galaxy.ini late last year. However, if you still have it, it's perfectly fine to leave it there and continue using it.
...and this was returned:
[uWSGI] getting INI configuration from universe_wsgi.ini
At that point, Terminal just kinda sits there and does not display any output, error messages, etc.
If uwsgi is running, this suggests it's working. If you have set up the uwsgi proxy in nginx you should be able to try to access Galaxy via the web at this point. More detail may be available in the uwsgi log file - did you set `logto` in the `[uwsgi]` section of universe_wsgi.ini?
Questions:
- Was I correct in specifying universe_wsgi.ini ?
The location of the config file is up to you. universe_wsgi.ini in the root of the Galaxy directory was the old default.
- Nothing is returned after "[uWSGI] getting INI configuration from
universe_wsgi.ini". Is that normal?
Yes. You probably want to set `logto` as described above and start uwsgi in the background, e.g. with the `-d` command line option.
- Should I activate the Python virtual environment first (via "source
bin/activate") before executing "~/galaxy-dist$ PYTHONPATH=eggs/PasteDeploy-1.5.0-py2.7.egg uwsgi --ini-paste universe_wsgi.ini" ?
Yes, I would do this.
Is the next step to start Galaxy, like so?
cd galaxy-dist GALAXY_RUN_ALL=1 sh run.sh --daemon
uwsgi is running your Galaxy web processes, so I would check to ensure they are working first. Once you've verified that, you can use the command above to start any job handlers you have configured.
--nate
I appreciate your help.
On Mon, Jan 26, 2015 at 5:22 PM, Nate Coraor nate@bx.psu.edu wrote:
Hi Nicholas,
Sorry for the somewhat misleading text in the documentation. The part about "no extra modules or recompiling" refers to nginx itself - uWSGI protocol support is built in to the core functionality of nginx. This does not mean that uWSGI itself is installed. Installing uWSGI is addressed a bit further up:
You will also need to have uWSGI installed. There are a variety of ways to do this. It can be installed system-wide by installing from your system's package manager (on Debian and Ubuntu systems, the uwsgi and uwsgi-plugin-python provide the necessary components), or with theeasy_install or pip commands (which will install it to the system's Python site-packages directory). Alternatively, if you are already running Galaxy from a Python virtualenv, you can use pip install uwsgi with that virtualenv's copy of pip to install to that virtualenv as your unprivileged Galaxy user.
In the case of Ubuntu sudo apt-get install uwsgi uwsgi-plugin-python should provide the necessary components.
--nate
On Mon, Jan 26, 2015 at 2:46 PM, Nicholas Kline nxk60@case.edu wrote:
Hello,
I am trying to setup a local production Galaxy server for our department. Our Galaxy server environment is as follows:
- Ubuntu 14.04 64-bit
- 94 GB Memory
- Intel Xeon
- Python virtual environment
- Nginx
- PostgreSQL
- ProFTPd
Today I tried starting the Galaxy server for the first time. I think the first step in that process is starting uWSGI, correct? To do so, I executed these commands:
cd ~/galaxy-dist/ PYTHONPATH=eggs/PasteDeploy-1.5.0-py2.7.egg uwsgi --ini-paste config/galaxy.ini
However, it appears something failed. This information is returned:
The program 'uwsgi' can be found in the following packages:
- uwsgi-core
- uwsgi-plugin-alarm-curl
- uwsgi-plugin-alarm-xmpp
- uwsgi-plugin-curl-cron
- uwsgi-plugin-emperor-pg
- uwsgi-plugin-erlang
- uwsgi-plugin-geoip
- uwsgi-plugin-graylog2
- uwsgi-plugin-jvm-openjdk-6
- uwsgi-plugin-jvm-openjdk-7
- uwsgi-plugin-ldap
- uwsgi-plugin-lua5.1
- uwsgi-plugin-lua5.2
- uwsgi-plugin-php
- uwsgi-plugin-psgi
- uwsgi-plugin-python
- uwsgi-plugin-python3
- uwsgi-plugin-rack-ruby1.9.1
- uwsgi-plugin-router-access
- uwsgi-plugin-sqlite3
- uwsgi-plugin-v8
- uwsgi-plugin-xslt
Ask your administrator to install one of them
According to the galaxy documentation at https://wiki.galaxyproject.org/Admin/Config/Performance/Scaling , "uWSGI support is built in to nginx, so no extra modules or recompiling should be required".
Also, should '--ini-paste config/galaxy.ini' really be configured like this instead: --ini-paste universe_wsgi.ini ?
Thank you ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/