Re: [galaxy-dev] Make Galaxy use multiple processors
I am seeing two python processes. One is my runner - maxing at appr. 100%, the other is my web process just ticking along at about 5%. We have a user who has kick off 5 version of SAM to Interval. I would expect to see 5 processes, preferably taking multiple processors - I am seeing 1 On Tue, Jun 8, 2010 at 7:38 AM, Dennis Gascoigne <dennis.gascoigne@gmail.com
wrote:
I am seeing two python processes. One is my runner - maxing at appr. 100%, the other is my web process just ticking along at about 5%. There are 5 instances of SAM to Interval. I would expect to see 5 processes, preferably taking multiple processors.
On Mon, Jun 7, 2010 at 11:35 PM, Nate Coraor <nate@bx.psu.edu> wrote:
James Taylor wrote:
Nate, shouldn't python tools run as different processes scale out to multiple processors without any special configuration though? E.g. running 5 gops jobs in parallel should use multiple processes. Is there something that prevents this?
Oh, yes, I suppose I failed to read this email properly. It should definitely be starting separate processes, which would remove the GIL's one core limitation. Dennis, are you seeing multiple python processes when you run these tools?
--nate
On Jun 7, 2010, at 9:29 AM, Nate Coraor wrote:
Dennis Gascoigne wrote:
I know it must be possible to make Galaxy use more of the resources available to it but I haven't been able to figure out how. If galaxy is calling python scripts, it only seems to use a single processor to do so. We have an 8 core machine and if other binaries are spawned then they get other processors no problems, but any of the python operations though seem to operate on the same processor no matter how many of these 'python based' jobs are running i.e. If i run 5 tools simultaneously, I would expect allocation to make use of all available processing resources. Is there some config to make this happen? I am sure I am missing something basic.
Hi Dennis,
This is something I've referred to on the ProductionServer wiki page, but have not yet documented. I also covered it in my Developer Conference talk (slides: http://usegalaxy.org/dev2010 ).
However, probably the best place to find it right now is the conversation on the list from last month between Davide Cittaro and I:
http://lists.bx.psu.edu/pipermail/galaxy-dev/2010-May/002634.html
I'd suggest copying universe_wsgi.ini in both instances rather than moving in one, as I originally suggested. Otherwise, egg-checking routines will fail because of the missing config file.
--nate
Cheers
------------------------------------------------------------------------ _______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
_______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
-- Dennis Gascoigne 0407 639 995 dennis.gascoigne@gmail.com
-- Dennis Gascoigne 0407 639 995 dennis.gascoigne@gmail.com
Dennis Gascoigne wrote:
I am seeing two python processes. One is my runner - maxing at appr. 100%, the other is my web process just ticking along at about 5%. We have a user who has kick off 5 version of SAM to Interval. I would expect to see 5 processes, preferably taking multiple processors - I am seeing 1
Dennis, I suspect the tool may have finished, but the process of setting metadata is still running. Try setting the following in your job runner's config file: set_metadata_externally = True This will cause metadata generation to happen in its own process. --nate
On Tue, Jun 8, 2010 at 7:38 AM, Dennis Gascoigne <dennis.gascoigne@gmail.com <mailto:dennis.gascoigne@gmail.com>> wrote:
I am seeing two python processes. One is my runner - maxing at appr. 100%, the other is my web process just ticking along at about 5%. There are 5 instances of SAM to Interval. I would expect to see 5 processes, preferably taking multiple processors.
On Mon, Jun 7, 2010 at 11:35 PM, Nate Coraor <nate@bx.psu.edu <mailto:nate@bx.psu.edu>> wrote:
James Taylor wrote:
Nate, shouldn't python tools run as different processes scale out to multiple processors without any special configuration though? E.g. running 5 gops jobs in parallel should use multiple processes. Is there something that prevents this?
Oh, yes, I suppose I failed to read this email properly. It should definitely be starting separate processes, which would remove the GIL's one core limitation. Dennis, are you seeing multiple python processes when you run these tools?
--nate
On Jun 7, 2010, at 9:29 AM, Nate Coraor wrote:
Dennis Gascoigne wrote:
I know it must be possible to make Galaxy use more of the resources available to it but I haven't been able to figure out how. If galaxy is calling python scripts, it only seems to use a single processor to do so. We have an 8 core machine and if other binaries are spawned then they get other processors no problems, but any of the python operations though seem to operate on the same processor no matter how many of these 'python based' jobs are running i.e. If i run 5 tools simultaneously, I would expect allocation to make use of all available processing resources. Is there some config to make this happen? I am sure I am missing something basic.
Hi Dennis,
This is something I've referred to on the ProductionServer wiki page, but have not yet documented. I also covered it in my Developer Conference talk (slides: http://usegalaxy.org/dev2010 ).
However, probably the best place to find it right now is the conversation on the list from last month between Davide Cittaro and I:
http://lists.bx.psu.edu/pipermail/galaxy-dev/2010-May/002634.html
I'd suggest copying universe_wsgi.ini in both instances rather than moving in one, as I originally suggested. Otherwise, egg-checking routines will fail because of the missing config file.
--nate
Cheers ------------------------------------------------------------------------ _______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu <mailto:galaxy-dev@lists.bx.psu.edu> http://lists.bx.psu.edu/listinfo/galaxy-dev
_______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu <mailto:galaxy-dev@lists.bx.psu.edu> http://lists.bx.psu.edu/listinfo/galaxy-dev
-- Dennis Gascoigne 0407 639 995 dennis.gascoigne@gmail.com <mailto:dennis.gascoigne@gmail.com>
-- Dennis Gascoigne 0407 639 995 dennis.gascoigne@gmail.com <mailto:dennis.gascoigne@gmail.com>
participants (2)
-
Dennis Gascoigne -
Nate Coraor