Previously, the BLAST wrapper was able to use multiple cores but recently users have started complaining it has got really slow and when I look at the cluster a job is only using a single core. I don’t want jobs split across multiple cluster nodes, but each node has 16 cores so it would be good if they could be used. I’m still using the older 16.01 release and this has been upgraded repeatedly over the last few years so I’m still using the universe_wsg.ini file (sym linked to config/galaxy.ini) and I don’t have a job_conf.xml set up. I’m using drmaa to drive an LSF cluster. BLAST is the main issue here so I was wondering if there’s a way to pass the -num_threads flag without breaking everything until I can build up a new server? Dr. Shane Sturrock NZGL BioIT Admin nzgl@biomatters.com
Hi Shane, We've not touched anything on the BLAST+ wrapper here for a while - the command line is always built using: -num_threads "\${GALAXY_SLOTS:-8}" That means use the environment variable $GALAXY_SLOTS if set, defaulting to 8 threads if not. See: https://github.com/peterjc/galaxy_blast/blob/master/tools/ncbi_blast_plus/nc... Galaxy itself should be setting $GALAXY_SLOTS via your cluster configuration - and from your description this seems to be set to 1 thread/slot only. Can you tell us more about how your Galaxy LSF is setup? Have you got BLAST specific cluster job settings? (I don't use LSF so don't have the details to hand) Peter On Wed, Jul 20, 2016 at 10:03 PM, Shane Sturrock <shane@biomatters.com> wrote:
Previously, the BLAST wrapper was able to use multiple cores but recently users have started complaining it has got really slow and when I look at the cluster a job is only using a single core. I don’t want jobs split across multiple cluster nodes, but each node has 16 cores so it would be good if they could be used. I’m still using the older 16.01 release and this has been upgraded repeatedly over the last few years so I’m still using the universe_wsg.ini file (sym linked to config/galaxy.ini) and I don’t have a job_conf.xml set up. I’m using drmaa to drive an LSF cluster. BLAST is the main issue here so I was wondering if there’s a way to pass the -num_threads flag without breaking everything until I can build up a new server?
Dr. Shane Sturrock NZGL BioIT Admin nzgl@biomatters.com
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Peter,
On 21/07/2016, at 8:50 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
Hi Shane,
We've not touched anything on the BLAST+ wrapper here for a while - the command line is always built using:
-num_threads "\${GALAXY_SLOTS:-8}"
That means use the environment variable $GALAXY_SLOTS if set, defaulting to 8 threads if not. See:
https://github.com/peterjc/galaxy_blast/blob/master/tools/ncbi_blast_plus/nc...
Galaxy itself should be setting $GALAXY_SLOTS via your cluster configuration - and from your description this seems to be set to 1 thread/slot only.
I suspect since I’m running an old configuration that my cluster setup isn’t optimal. I didn’t really want my users going nuts since our cluster is small (6 nodes, 16 cores each) so I wasn’t keen to give them too much power because and was happy that tools would only run on single cores although I’ve set things up as per the docs way back when so it uses multiple handlers and so on and with default_cluster_job_runner = drmaa:/// all in the galaxy.ini (universe_wsgi.ini actually)
Can you tell us more about how your Galaxy LSF is setup? Have you got BLAST specific cluster job settings?
Nothing specific, but it would seem that the GALAXY_SLOTS variable has gone from being unset to set with the install I’ve got now and that would explain why BLAST has slowed down a lot.
(I don't use LSF so don't have the details to hand)
I’m just using the drmaa plugin so I’m guessing I need to either specify a job runner for BLAST which sets GALAXY_SLOTS. The documentation isn’t entirely clear although it seems I need to create a job_conf.xml which sets local_slots to a value for runner drmaa unless there’s a way to set that in the galaxy.ini file instead. Shane
Peter
On Wed, Jul 20, 2016 at 10:03 PM, Shane Sturrock <shane@biomatters.com> wrote:
Previously, the BLAST wrapper was able to use multiple cores but recently users have started complaining it has got really slow and when I look at the cluster a job is only using a single core. I don’t want jobs split across multiple cluster nodes, but each node has 16 cores so it would be good if they could be used. I’m still using the older 16.01 release and this has been upgraded repeatedly over the last few years so I’m still using the universe_wsg.ini file (sym linked to config/galaxy.ini) and I don’t have a job_conf.xml set up. I’m using drmaa to drive an LSF cluster. BLAST is the main issue here so I was wondering if there’s a way to pass the -num_threads flag without breaking everything until I can build up a new server?
Dr. Shane Sturrock NZGL BioIT Admin nzgl@biomatters.com
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Dr. Shane Sturrock NZGL BioIT Admin nzgl@biomatters.com
On Thu, Jul 21, 2016 at 10:35 AM, Shane Sturrock <shane@biomatters.com> wrote:
I’m just using the drmaa plugin so I’m guessing I need to either specify a job runner for BLAST which sets GALAXY_SLOTS. The documentation isn’t entirely clear although it seems I need to create a job_conf.xml which sets local_slots to a value for runner drmaa unless there’s a way to set that in the galaxy.ini file instead.
There is a legacy job setting via galaxy.ini aka universe_wsgi.ini where we used to use this: [galaxy:tool_runners] ncbi_blastp_wrapper = drmaa://-V -pe smp 4/ ncbi_blastn_wrapper = drmaa://-V -pe smp 4/ ncbi_blastx_wrapper = drmaa://-V -pe smp 4/ ncbi_tblastn_wrapper = drmaa://-V -pe smp 4/ ncbi_tblastx_wrapper = drmaa://-V -pe smp 4/ blast_reciprocal_best_hits = drmaa://-V -pe smp 4/ The -V was an SGE switch to copy the environment variables, and -pe smp 4 is for a 4 core job. For job_conf.xml I am trying something similar with less repetition: ... <!-- Default queue on the SGE cluster (four cores) --> <destination id="all4.q" runner="sge" tags="gruffalo"> <param id="nativeSpecification">-pe smp 4</param> <env file="/mnt/shared/galaxy/apps/galaxy-paths.sh"/> </destination> ... <tool id="ncbi_blastp_wrapper" destination="all4.q"/> <tool id="ncbi_blastn_wrapper" destination="all4.q"/> <tool id="ncbi_blastx_wrapper" destination="all4.q"/> <tool id="ncbi_tblastn_wrapper" destination="all4.q"/> <tool id="ncbi_tblastx_wrapper" destination="all4.q"/> <tool id="ncbi_rpsblast_wrapper" destination="all4.q"/> <tool id="ncbi_rpstblastn_wrapper" destination="all4.q"/> <tool id="blast_reciprocal_best_hits" destination="all4.q"/> ... i.e. I've set the computationally heavy BLAST+ tools to use a destination configured to use just four cores. Because this is SGE, we do this via the native syntax of -pe smp 4, i.e. just like at the command line we'd do: $ qsub -pe smp 4 ... This detail will be different for your LSF cluster. Peter
Thanks Peter, that was the hint I was needing. It looks like the solution for my LSF cluster was to use drmaa://-n 8/ in the universe_wsgi.ini and now I can see BLAST using 8 cores. It doesn’t scale perfectly so going higher isn't sensible but turnaround time for my test job has improved markedly. Shane Dr. Shane Sturrock NZGL BioIT Admin nzgl@biomatters.com
participants (2)
-
Peter Cock -
Shane Sturrock