Hi,
This is a question I posted to galaxy user mailing list a while back and was redirected to dev for possible answers: Is it possible in Galaxy to design a tool whose sole purpose is to run other tools. This is motivated by our desire to enhance execution capabilities of existing tools via a generic tool which acts as a wrapper.
Thanks, Ketan
Galaxy has an API that is capable of running tools - certainly this is one path forward on something like this. I am not sure it is the best path forward though. Probably the best way to enhance Galaxy's execution capabilities is to extend the Galaxy core framework itself - this has its own downsides though.
If you can offer more details about how you would like to enhance Galaxy - what it cannot do that you would like it to do - I or others may be able to provide more specific ideas. Otherwise, sorry I have not been or more help.
-John
On Tue, Feb 4, 2014 at 2:51 PM, Ketan Maheshwari ketan@mcs.anl.gov wrote:
Hi,
This is a question I posted to galaxy user mailing list a while back and was redirected to dev for possible answers: Is it possible in Galaxy to design a tool whose sole purpose is to run other tools. This is motivated by our desire to enhance execution capabilities of existing tools via a generic tool which acts as a wrapper.
Thanks, Ketan
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Ketan, Are workflows not the way to go for this? We have similar tools that are usually run in sequence. We made them all separate to allow reusing them elsewhere. For these linked tools one could write a superscript/wrapper or design a workflow which can be public or privately shared. Best wishes, Alex
-----Oorspronkelijk bericht----- Van: galaxy-dev-bounces@lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu] Namens John Chilton Verzonden: woensdag 5 februari 2014 22:54 Aan: Ketan Maheshwari CC: galaxy-dev@lists.bx.psu.edu Onderwerp: Re: [galaxy-dev] running tools within tool
Galaxy has an API that is capable of running tools - certainly this is one path forward on something like this. I am not sure it is the best path forward though. Probably the best way to enhance Galaxy's execution capabilities is to extend the Galaxy core framework itself - this has its own downsides though.
If you can offer more details about how you would like to enhance Galaxy - what it cannot do that you would like it to do - I or others may be able to provide more specific ideas. Otherwise, sorry I have not been or more help.
-John
On Tue, Feb 4, 2014 at 2:51 PM, Ketan Maheshwari ketan@mcs.anl.gov wrote:
Hi,
This is a question I posted to galaxy user mailing list a while back and was redirected to dev for possible answers: Is it possible in Galaxy to design a tool whose sole purpose is to run other tools. This is motivated by our desire to enhance execution capabilities of existing tools via a generic tool which acts as a wrapper.
Thanks, Ketan
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi John, Alex, All,
Elaborating on the motivation behind my question of running tools within tool. First, running a tool in parallel at large-scale. For example, if I need to find a pattern from 1000 files via Galaxy Select tool from Text and Filter tool-group, I am limited by providing one file at a time to the tool which will take a long time to finish. Please correct me if there is a more sophisticated way to approach this problem. Second, related concern is running a tool in parallel on one or more HPC resources.
We want to write a generic wrapper Galaxy tool, powered by Swift parallel framework such that it can run any arbitrary Galaxy tool in parallel on HPC resources. Currently, we have developed this capability but for external executables which is not a most secure way of using Galaxy as I understand from previous conversation.
Having such a wrapper tool in a standard way is desirable so that it preserves the tool contract and binding within Galaxy environment. That is maintaining the history and metadata conventions of Galaxy.
Thanks, Ketan
On Wed, Feb 5, 2014 at 3:53 PM, John Chilton chilton@msi.umn.edu wrote:
Galaxy has an API that is capable of running tools - certainly this is one path forward on something like this. I am not sure it is the best path forward though. Probably the best way to enhance Galaxy's execution capabilities is to extend the Galaxy core framework itself - this has its own downsides though.
If you can offer more details about how you would like to enhance Galaxy - what it cannot do that you would like it to do - I or others may be able to provide more specific ideas. Otherwise, sorry I have not been or more help.
-John
On Tue, Feb 4, 2014 at 2:51 PM, Ketan Maheshwari ketan@mcs.anl.gov wrote:
Hi,
This is a question I posted to galaxy user mailing list a while back and
was
redirected to dev for possible answers: Is it possible in Galaxy to design a tool whose sole purpose is to run
other
tools. This is motivated by our desire to enhance execution capabilities
of
existing tools via a generic tool which acts as a wrapper.
Thanks, Ketan
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Ketan,
Have you taken a look at galaxy's built-in parallelism framework? For a great current example of a tool using this, look at Peter's NCBI BLAST+ wrappers. https://github.com/peterjc/galaxy_blast
-Dannon
On Thu, Feb 6, 2014 at 10:32 AM, Ketan Maheshwari < ketancmaheshwari@gmail.com> wrote:
Hi John, Alex, All,
Elaborating on the motivation behind my question of running tools within tool. First, running a tool in parallel at large-scale. For example, if I need to find a pattern from 1000 files via Galaxy Select tool from Text and Filter tool-group, I am limited by providing one file at a time to the tool which will take a long time to finish. Please correct me if there is a more sophisticated way to approach this problem. Second, related concern is running a tool in parallel on one or more HPC resources.
We want to write a generic wrapper Galaxy tool, powered by Swift parallel framework such that it can run any arbitrary Galaxy tool in parallel on HPC resources. Currently, we have developed this capability but for external executables which is not a most secure way of using Galaxy as I understand from previous conversation.
Having such a wrapper tool in a standard way is desirable so that it preserves the tool contract and binding within Galaxy environment. That is maintaining the history and metadata conventions of Galaxy.
Thanks, Ketan
On Wed, Feb 5, 2014 at 3:53 PM, John Chilton chilton@msi.umn.edu wrote:
Galaxy has an API that is capable of running tools - certainly this is one path forward on something like this. I am not sure it is the best path forward though. Probably the best way to enhance Galaxy's execution capabilities is to extend the Galaxy core framework itself - this has its own downsides though.
If you can offer more details about how you would like to enhance Galaxy - what it cannot do that you would like it to do - I or others may be able to provide more specific ideas. Otherwise, sorry I have not been or more help.
-John
On Tue, Feb 4, 2014 at 2:51 PM, Ketan Maheshwari ketan@mcs.anl.gov wrote:
Hi,
This is a question I posted to galaxy user mailing list a while back
and was
redirected to dev for possible answers: Is it possible in Galaxy to design a tool whose sole purpose is to run
other
tools. This is motivated by our desire to enhance execution
capabilities of
existing tools via a generic tool which acts as a wrapper.
Thanks, Ketan
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
-- Ketan
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Thanks Dannon for the reference. I checked out the tool and installed from toolshed on my local Galaxy instance. I also checked out the related paper which refers that the Blast executables run in parallel by partitioning the input files into fragments and running batches in parallel. That sounds cool. I browsed the code but could not find the exact mechanism. Is the parallelism at workflow level aka branch parallelism or is it at the tool level that is the tool invokes parallel code?
Thanks, Ketan
On Thu, Feb 6, 2014 at 9:42 AM, Dannon Baker dannon.baker@gmail.com wrote:
Ketan,
Have you taken a look at galaxy's built-in parallelism framework? For a great current example of a tool using this, look at Peter's NCBI BLAST+ wrappers. https://github.com/peterjc/galaxy_blast
-Dannon
On Thu, Feb 6, 2014 at 10:32 AM, Ketan Maheshwari < ketancmaheshwari@gmail.com> wrote:
Hi John, Alex, All,
Elaborating on the motivation behind my question of running tools within tool. First, running a tool in parallel at large-scale. For example, if I need to find a pattern from 1000 files via Galaxy Select tool from Text and Filter tool-group, I am limited by providing one file at a time to the tool which will take a long time to finish. Please correct me if there is a more sophisticated way to approach this problem. Second, related concern is running a tool in parallel on one or more HPC resources.
We want to write a generic wrapper Galaxy tool, powered by Swift parallel framework such that it can run any arbitrary Galaxy tool in parallel on HPC resources. Currently, we have developed this capability but for external executables which is not a most secure way of using Galaxy as I understand from previous conversation.
Having such a wrapper tool in a standard way is desirable so that it preserves the tool contract and binding within Galaxy environment. That is maintaining the history and metadata conventions of Galaxy.
Thanks, Ketan
On Wed, Feb 5, 2014 at 3:53 PM, John Chilton chilton@msi.umn.edu wrote:
Galaxy has an API that is capable of running tools - certainly this is one path forward on something like this. I am not sure it is the best path forward though. Probably the best way to enhance Galaxy's execution capabilities is to extend the Galaxy core framework itself - this has its own downsides though.
If you can offer more details about how you would like to enhance Galaxy - what it cannot do that you would like it to do - I or others may be able to provide more specific ideas. Otherwise, sorry I have not been or more help.
-John
On Tue, Feb 4, 2014 at 2:51 PM, Ketan Maheshwari ketan@mcs.anl.gov wrote:
Hi,
This is a question I posted to galaxy user mailing list a while back
and was
redirected to dev for possible answers: Is it possible in Galaxy to design a tool whose sole purpose is to run
other
tools. This is motivated by our desire to enhance execution
capabilities of
existing tools via a generic tool which acts as a wrapper.
Thanks, Ketan
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
-- Ketan
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Thanks Dannon for the reference. I checked out the tool and installed from toolshed on my local Galaxy instance. I also checked out the related paper which refers that the Blast executables run in parallel by partitioning the input files into fragments and running batches in parallel. That sounds cool. I browsed the code but could not find the exact mechanism. Is the parallelism at workflow level aka branch parallelism or is it at the tool level that is the tool invokes parallel code?
Thanks, Ketan
On Sun, Feb 9, 2014 at 7:50 PM, Ketan Maheshwari ketan@mcs.anl.gov wrote:
Thanks Dannon for the reference. I checked out the tool and installed from toolshed on my local Galaxy instance. I also checked out the related paper which refers that the Blast executables run in parallel by partitioning the input files into fragments and running batches in parallel. That sounds cool. I browsed the code but could not find the exact mechanism. Is the parallelism at workflow level aka branch parallelism or is it at the tool level that is the tool invokes parallel code?
Thanks, Ketan
On Thu, Feb 6, 2014 at 9:42 AM, Dannon Baker dannon.baker@gmail.comwrote:
Ketan,
Have you taken a look at galaxy's built-in parallelism framework? For a great current example of a tool using this, look at Peter's NCBI BLAST+ wrappers. https://github.com/peterjc/galaxy_blast
-Dannon
On Thu, Feb 6, 2014 at 10:32 AM, Ketan Maheshwari < ketancmaheshwari@gmail.com> wrote:
Hi John, Alex, All,
Elaborating on the motivation behind my question of running tools within tool. First, running a tool in parallel at large-scale. For example, if I need to find a pattern from 1000 files via Galaxy Select tool from Text and Filter tool-group, I am limited by providing one file at a time to the tool which will take a long time to finish. Please correct me if there is a more sophisticated way to approach this problem. Second, related concern is running a tool in parallel on one or more HPC resources.
We want to write a generic wrapper Galaxy tool, powered by Swift parallel framework such that it can run any arbitrary Galaxy tool in parallel on HPC resources. Currently, we have developed this capability but for external executables which is not a most secure way of using Galaxy as I understand from previous conversation.
Having such a wrapper tool in a standard way is desirable so that it preserves the tool contract and binding within Galaxy environment. That is maintaining the history and metadata conventions of Galaxy.
Thanks, Ketan
On Wed, Feb 5, 2014 at 3:53 PM, John Chilton chilton@msi.umn.eduwrote:
Galaxy has an API that is capable of running tools - certainly this is one path forward on something like this. I am not sure it is the best path forward though. Probably the best way to enhance Galaxy's execution capabilities is to extend the Galaxy core framework itself - this has its own downsides though.
If you can offer more details about how you would like to enhance Galaxy - what it cannot do that you would like it to do - I or others may be able to provide more specific ideas. Otherwise, sorry I have not been or more help.
-John
On Tue, Feb 4, 2014 at 2:51 PM, Ketan Maheshwari ketan@mcs.anl.gov wrote:
Hi,
This is a question I posted to galaxy user mailing list a while back
and was
redirected to dev for possible answers: Is it possible in Galaxy to design a tool whose sole purpose is to
run other
tools. This is motivated by our desire to enhance execution
capabilities of
existing tools via a generic tool which acts as a wrapper.
Thanks, Ketan
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
-- Ketan
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
The BLAST+ binaries support multi-threaded operation, which is handled via the $GALAXY_SLOTS environment variable. This should be set automatically by Galaxy via your job runner settings, which allows you to (for example) allocate four cores to each BLAST job.
In addition, the BLAST+ wrappers also support high level parallelism by task splitting if "use_tasked_jobs = True" is enabled in your "universe_wsgi.ini" configuration file. Essentially, the FASTA input query files are broken up into batches of 1000 sequences, a separate BLAST child job is run for each chunk, and then the BLAST output files are merged (in order). This is transparent for the end user.
Each tool enables this via their XML file, e.g.
<parallelism method="multi" split_inputs="query" split_mode="to_size" split_size="1000" merge_outputs="output1"></parallelism>
This requires splitting support in the FASTA input datatypes, and merging support in the selected output datatype (e.g. BLAST XML, tabular, etc). This is done by methods in the Python datatype classes.
It would be interesting to see if any of John's work on collections of files of the same type might fit nicely with this approach (and thus avoid the disk IO overhead of the merge step?).
Peter
On Mon, Feb 10, 2014 at 1:56 AM, Ketan Maheshwari ketancmaheshwari@gmail.com wrote:
Thanks Dannon for the reference. I checked out the tool and installed from toolshed on my local Galaxy instance. I also checked out the related paper which refers that the Blast executables run in parallel by partitioning the input files into fragments and running batches in parallel. That sounds cool. I browsed the code but could not find the exact mechanism. Is the parallelism at workflow level aka branch parallelism or is it at the tool level that is the tool invokes parallel code?
Thanks, Ketan
On Sun, Feb 9, 2014 at 7:50 PM, Ketan Maheshwari ketan@mcs.anl.gov wrote:
Thanks Dannon for the reference. I checked out the tool and installed from toolshed on my local Galaxy instance. I also checked out the related paper which refers that the Blast executables run in parallel by partitioning the input files into fragments and running batches in parallel. That sounds cool. I browsed the code but could not find the exact mechanism. Is the parallelism at workflow level aka branch parallelism or is it at the tool level that is the tool invokes parallel code?
Thanks, Ketan
On Thu, Feb 6, 2014 at 9:42 AM, Dannon Baker dannon.baker@gmail.com wrote:
Ketan,
Have you taken a look at galaxy's built-in parallelism framework? For a great current example of a tool using this, look at Peter's NCBI BLAST+ wrappers. https://github.com/peterjc/galaxy_blast
-Dannon
galaxy-dev@lists.galaxyproject.org
-
Bossers, Alex -
Dannon Baker -
John Chilton -
Ketan Maheshwari -
Ketan Maheshwari -
Peter Cock