maximum value for local_job_queue_workers
Hi, What is the difference between threadpool_workers and local_job_queue_workers*.* what is the maximum size that i can set local_job_queue_workers value? -- Thanks & Regards, Hari Krishna .M
hari krishna wrote:
Hi,
What is the difference between threadpool_workers and local_job_queue_workers*.*
Hi, threadpool_workers is the number of threads which will be available to serve web requests. This means the number of possible concurrent browser connections, but this value can remain fairly low since most web transactions are < 1 second. local_job_queue_workers is the number of tool processes that Galaxy will start and allow to run at the same time. For example, if you set this value to 2 and then submit 3 jobs, the third one will wait for one of the first two to finish before it will execute.
what is the maximum size that i can set local_job_queue_workers value?
There's not really a maximum, but it doesn't make sense to set this much higher than the number of CPU cores in your server. I would suggest setting it to (cores - 1) to ensure the Galaxy process itself always has a core to run on. --nate
-- Thanks & Regards, Hari Krishna .M
_______________________________________________ To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Hi Nate, Thanks for ur reply. I have made a job runner which will submit job to job meta-scheduler, which can handle 200 jobs concurrently. So is it safe to set local_job_queue_workers = 200 ?? This instance of Galaxy will be accessed by some 50 users. So can threadpool_workers be set to 50?? Also, When a job is completed it shows the output file into the history window where there is option to save output file. But when job is not successful instead of generating output file, it generates a job error file. How can i place this error file in the place of output file in the history window. Thanks and Regards Hari On Thu, Feb 24, 2011 at 6:43 PM, Nate Coraor <nate@bx.psu.edu> wrote:
hari krishna wrote:
Hi,
What is the difference between threadpool_workers and local_job_queue_workers*.*
Hi,
threadpool_workers is the number of threads which will be available to serve web requests. This means the number of possible concurrent browser connections, but this value can remain fairly low since most web transactions are < 1 second.
local_job_queue_workers is the number of tool processes that Galaxy will start and allow to run at the same time. For example, if you set this value to 2 and then submit 3 jobs, the third one will wait for one of the first two to finish before it will execute.
what is the maximum size that i can set local_job_queue_workers value?
There's not really a maximum, but it doesn't make sense to set this much higher than the number of CPU cores in your server. I would suggest setting it to (cores - 1) to ensure the Galaxy process itself always has a core to run on.
--nate
-- Thanks & Regards, Hari Krishna .M
_______________________________________________ To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Thanks & Regards, Hari Krishna .M
hari krishna wrote:
Hi Nate, Thanks for ur reply. I have made a job runner which will submit job to job meta-scheduler, which can handle 200 jobs concurrently. So is it safe to set local_job_queue_workers = 200 ??
That depends on whether your runner uses that option. Note that it's used directly by the local runner to define the size of a worker pool which will be used to start local job processes. I don't know how your runner is written so I don't know what, if anything, it would do in that context.
This instance of Galaxy will be accessed by some 50 users. So can threadpool_workers be set to 50??
It can probably be significantly less than that, since the likelihood of all 50 users making concurrent requests is low. Threads incur some overhead, so I'd start with a lower value and increase it if you find that the server is not handling all of the requests. Also, because of limitations in the way threading ing the Python interpreter is written, you may want to run multiple server processes rather than increasing threads. Details on how to set this up are in the documentation at usegalaxy.org/production
Also, When a job is completed it shows the output file into the history window where there is option to save output file. But when job is not successful instead of generating output file, it generates a job error file. How can i place this error file in the place of output file in the history window.
The easiest way for users to set and send in the full error report is to set the 'smtp_server' and 'error_email_to' options in universe_wsgi.ini. This will enable the "bug report" icon you see on our public server. --nate
Thanks and Regards Hari
On Thu, Feb 24, 2011 at 6:43 PM, Nate Coraor <nate@bx.psu.edu> wrote:
hari krishna wrote:
Hi,
What is the difference between threadpool_workers and local_job_queue_workers*.*
Hi,
threadpool_workers is the number of threads which will be available to serve web requests. This means the number of possible concurrent browser connections, but this value can remain fairly low since most web transactions are < 1 second.
local_job_queue_workers is the number of tool processes that Galaxy will start and allow to run at the same time. For example, if you set this value to 2 and then submit 3 jobs, the third one will wait for one of the first two to finish before it will execute.
what is the maximum size that i can set local_job_queue_workers value?
There's not really a maximum, but it doesn't make sense to set this much higher than the number of CPU cores in your server. I would suggest setting it to (cores - 1) to ensure the Galaxy process itself always has a core to run on.
--nate
-- Thanks & Regards, Hari Krishna .M
_______________________________________________ To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Thanks & Regards, Hari Krishna .M
participants (2)
-
hari krishna -
Nate Coraor