Andreas, I am not sure if you can call these "surprises". Some tools (which I highly appreciate) of Peter have been "parallelised" to get the job done more quickly. I earlier mentioned the ncbi blast+ wrappers but there the tool by itself handles the multithreading. Other tools I am aware that use a python script/wrapper to chunk up the initial query and rejoin later are tools like signalp, TMHMM and such. Usually it also involves some parsing of output to data that galaxy can subsequently handle. In the latter examples its done using python scripts, but for some of our custom tools we did it in perl, some using bash parallel, or using R. I wouldn't have a solution to getting to know this without going through the initial wrappers... Alex -----Oorspronkelijk bericht----- Van: galaxy-dev-bounces@lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu] Namens Andreas Kuntzagk Verzonden: dinsdag 27 november 2012 9:58 Aan: Bob Harris CC: galaxy-dev@lists.bx.psu.edu Onderwerp: Re: [galaxy-dev] multithreaded tools Hi, the four processes I saw where all called "lastz" and ran in parallel and consumed 100% of a core each. My guess is that the lastz_wrapper.py is responsible for this. Looking at it I see a some code regarding queuing and in the very beginning this line: WORKERS = 4 and further one the class BaseQueue which starts "threads". BTW. there seems to be no way to adjust this number other than editing the source file - bad. And this get's me wondering if there are other such surprises hidden in galaxy. regards, Andreas On 26.11.2012 15:55, Bob Harris wrote:
Howdy, Andreas,
The four processes started for a galaxy lastz job must involve post-processing the lastz output through some other shell tool. Lastz by itself doesn't support multiple threads or processes.
Bob H
On Nov 26, 2012, at 3:58 AM, Andreas Kuntzagk wrote:
Hi,
I'm wandering how galaxy supports tools that are multithreaded or multi-process. When working with lastz I noticed that it starts 4 parallel processes. Is that always so? Can this be adjusted? What other tools also are multi-process?
regards, Andreas
-- Andreas Kuntzagk
SystemAdministrator
Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany
http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/