On Tue, Nov 27, 2012 at 8:58 AM, Andreas Kuntzagk <andreas.kuntzagk@mdc-berlin.de> wrote:
Hi,
the four processes I saw where all called "lastz" and ran in parallel and consumed 100% of a core each. My guess is that the lastz_wrapper.py is responsible for this. Looking at it I see a some code regarding queuing and in the very beginning this line:
WORKERS = 4
and further one the class BaseQueue which starts "threads". BTW. there seems to be no way to adjust this number other than editing the source file - bad.
And this get's me wondering if there are other such surprises hidden in galaxy.
regards, Andreas
As the author of several tool wrappers, I've been asking for a Galaxy wide mechanism for Galaxy to tell the tool how many threads it can use, for example via an environment variable. The value could then be set with a general default, per runner default, or even per tool using the existing runner configuration under [galaxy:tool_runners] in universe_wsgi.ini See: http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-March/009037.html and: http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-June/010153.html In your example, and others like the BWA and BLAST+ wrappers where the tool XML is hard coded to 8 threads, you would probably want to use a custom runner in universe_wsgi.ini setting the cluster submission to request that many slots/CPUs. For our local cluster, I modify the BLAST+ wrapper XML to use 4 threads, and have something like this in my universe_wsgi.ini file: [galaxy:tool_runners] ncbi_blastp_wrapper = drmaa://-V -pe smp 4/ ncbi_blastn_wrapper = drmaa://-V -pe smp 4/ ncbi_blastx_wrapper = drmaa://-V -pe smp 4/ ncbi_tblastn_wrapper = drmaa://-V -pe smp 4/ ncbi_tblastx_wrapper = drmaa://-V -pe smp 4/ Peter