On Thu, Aug 8, 2013 at 1:31 PM, Nate Coraor <nate@bx.psu.edu> wrote:
On Aug 5, 2013, at 6:23 PM, Rodolfo Aramayo wrote:
Hi,
I have a 80 core/1TB RAM Dell server. In order improve Galaxy performance I modified the universe_wsgi.ini file so as to improve the Scaling and Load Balancing as follows:
# Use a threadpool for the web server instead of creating a thread for each # request. use_threadpool = True
# Number of threads in the web server thread pool. threadpool_workers = 200
################################################################################ [server:web0] use = egg:Paste#http port = 8090 host = 127.0.0.1 use_threadpool = true threadpool_workers = 10
... snip ...
[server:handler18] use = egg:Paste#http port = 8318 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5
# [server:handler19] # use = egg:Paste#http # port = 8319 # host = 127.0.0.1 # use_threadpool = true # threadpool_workers = 5
# [server:handler20] # use = egg:Paste#http # port = 8320 # host = 127.0.0.1 # use_threadpool = true # threadpool_workers = 5
################################################################################ job_manager = manager job_handlers = handler0,handler1,,handler2,handler3,handler4,handler5,handler6,handler7,handler8,handler9,handler10,handler11,handler12,handler13,handler14,handler15,handler16,handler17,handler18
Hi Rodolfo,
The job_manager and job_handlers options go under the [server:main] section, it looks like they are probably underneath the last [server:...] section right now.
Please note that all of these features should now be configured via the job configuration file (job_conf.xml), and I've just updated the documentation you referred to accordingly.
--nate
Dear Nate, Thanks Just to be sure I understand this correctly I now have in my 'universe' file the following: ##### # ---- HTTP Server ---------------------------------------------------------- # Configuration of the internal HTTP server. [server:main] # job_manager = manager <I ASSUME THIS IS CORRECT because I get errors when it is not commented ################################################################################ [server:manager] use = egg:Paste#http port = 8200 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 ################################################################################ [server:handler0] use = egg:Paste#http port = 8300 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler1] use = egg:Paste#http port = 8301 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 ...snip... [server:handler19] use = egg:Paste#http port = 8319 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 # [server:handler20] # use = egg:Paste#http # port = 8320 # host = 127.0.0.1 # use_threadpool = true # threadpool_workers = 5 ################################################################################ # The internal HTTP server to use. Currently only Paste is provided. This # option is required. use = egg:Paste#http # The port on which to listen. #port = 8080 # The address on which to listen. By default, only listen to localhost (Galaxy # will not be accessible over the network). Use '0.0.0.0' to listen on all # available network interfaces. #host = 127.0.0.1 host = 0.0.0.0 # Use a threadpool for the web server instead of creating a thread for each # request. use_threadpool = True # Number of threads in the web server thread pool. threadpool_workers = 200 ################################################################################ [server:web0] use = egg:Paste#http port = 8400 host = 127.0.0.1 use_threadpool = true threadpool_workers = 10 [server:web1] use = egg:Paste#http port = 8401 host = 127.0.0.1 use_threadpool = true threadpool_workers = 10 ...snip... [server:web11] use = egg:Paste#http port = 8411 host = 127.0.0.1 use_threadpool = true threadpool_workers = 10 [server:web12] use = egg:Paste#http port = 8412 host = 127.0.0.1 use_threadpool = true threadpool_workers = 10 ################################################################################ # ---- Filters -------------------------------------------------------------- # Filters sit between Galaxy and the HTTP server. ... ##### And in my job_conf.xml: <handler id="handler0" tags="handlers"/> <handler id="handler1" tags="handlers"/> ...snip... <handler id="handler18" tags="handlers"/> <handler id="handler19" tags="handlers"/> ##### Is this correct? Also, I guess the part I do not understand is what is the magic formula between the number of threadpool_workers as configured here: # Use a threadpool for the web server instead of creating a thread for each # request. use_threadpool = True # Number of threads in the web server thread pool. threadpool_workers = 200 and those configured on the web servers and on the server handlers? Thanks --Rodolfo