Dynamically Generated Tools
TL;DR I have a tool that generates other tools. Where should those tools go, and how should I add them (without modifying Galaxy code)? I apologize, in advance, for the wall of text. I would like some advice on what the best way to go about setting up a tool that generates other tools... As some of you (especially those who saw my boss's talk at GCC) already know, I'm working on a tool that lets galaxy users add web service operations as tools to Galaxy. We have a general purpose client for invoking web services, and we have a tool that let's a user enter a WSDL/WADL URL, select the operations they want to add, and generates tool config XML files for Galaxy. Right now, I'm in the process of making our tool "tool shed" compatible, and that's going along great (I especially like the tool dependency installation stuff), but I'm trying to determine the best course of action on the following: 1. Where is the best place to put the tool config (XML) files that are generated by my tool? My predecessors just placed them in a subdirectory under our tool's directory. I'm ok with this, but does that mean that all the generated files will get removed if they upgrade our tool via the tool shed (haven't taken the time to see if Galaxy just does an "hg pull" or if it's more complicated than that)? Would it be better to have my tool create a local tool shed repository (just for these generated tools), and if so, can the tool shed API let my tool install this repository? Having not looked at install_tool_shed_repositories.py yet, it's unclear from the wiki documentation whether or not POST /api/tool_shed_repositories/install_repository_revision requires that the tool shed already registered in tool_sheds_conf.xml)? 2. What's the best way to add the generated tools to Galaxy? My predecessors modified the tool_config.xml directly and required users to restart Galaxy. Now, If these tools are in a tool shed repo, I know that I can use the add_to_tool_panel() method in lib/tool_shed/util/tool_util.py if these tools are in a tool shed repo, however, if that's not the recommended course of action, then I'd like some advice on what to do? Thanks! Sincerely, Michael E. Cotterell Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Faculty Liaison, CS Graduate Student Association, University of Georgia mepcotterell@gmail.com (mailto:mepcotterell@gmail.com) mepcott@uga.edu (mailto:mepcott@uga.edu) mec@cs.uga.edu (mailto:mec@cs.uga.edu) http://michaelcotterell.com/
On Wed, Jul 17, 2013 at 4:58 PM, Michael E. Cotterell <mepcotterell@gmail.com> wrote:
TL;DR I have a tool that generates other tools. Where should those tools go, and how should I add them (without modifying Galaxy code)?
I apologize, in advance, for the wall of text. I would like some advice on what the best way to go about setting up a tool that generates other tools...
As some of you (especially those who saw my boss's talk at GCC) already know, I'm working on a tool that lets galaxy users add web service operations as tools to Galaxy. We have a general purpose client for invoking web services, and we have a tool that let's a user enter a WSDL/WADL URL, select the operations they want to add, and generates tool config XML files for Galaxy.
For those not at GCC2013, this was the talk presented by Jessica C. Kissinger, "Extension of Galaxy to Utilize Web Services and A Semantic Suggestion Engine" Abstract: http://wiki.galaxyproject.org/Events/GCC2013/Abstracts#Events.2FGCC2013.2FAb... Slides: http://wiki.galaxyproject.org/Documents/Presentations/GCC2013?action=AttachFile&do=view&target=KissingerWebServices.pdf Regards, Peter
Hi Michael, I think you will enter new ground with your tool. The closest tool that will do something similar is Ross toolfactory, I think: http://toolshed.g2.bx.psu.edu/view/fubar/toolfactory For me one question is, do you really want to offer that service to every user and is it save?
TL;DR I have a tool that generates other tools. Where should those tools go, and how should I add them (without modifying Galaxy code)?
I apologize, in advance, for the wall of text. I would like some advice on what the best way to go about setting up a tool that generates other tools...
As some of you (especially those who saw my boss's talk at GCC) already know, I'm working on a tool that lets galaxy users add web service operations as tools to Galaxy. We have a general purpose client for invoking web services, and we have a tool that let's a user enter a WSDL/WADL URL, select the operations they want to add, and generates tool config XML files for Galaxy.
Right now, I'm in the process of making our tool "tool shed" compatible, and that's going along great (I especially like the tool dependency installation stuff), but I'm trying to determine the best course of action on the following:
1. Where is the best place to put the tool config (XML) files that are generated by my tool? My predecessors just placed them in a subdirectory under our tool's directory. I'm ok with this, but does that mean that all the generated files will get removed if they upgrade our tool via the tool shed (haven't taken the time to see if Galaxy just does an "hg pull" or if it's more complicated than that)?
It is mainly a hg pull. But if you deinstall that repository and reinstall it, all your files are lost.
Would it be better to have my tool create a local tool shed repository (just for these generated tools), and if so, can the tool shed API let my tool install this repository? Having not looked at install_tool_shed_repositories.py yet, it's unclear from the wiki documentation whether or not POST /api/tool_shed_repositories/install_repository_revision requires that the tool shed already registered in tool_sheds_conf.xml)?
I think that is a better way to do it. The API can be used to install your tools. But it need to be configured beforehand from the user, administrator.
2. What's the best way to add the generated tools to Galaxy? My predecessors modified the tool_config.xml directly and required users to restart Galaxy. Now, If these tools are in a tool shed repo, I know that I can use the add_to_tool_panel() method in lib/tool_shed/util/tool_util.py if these tools are in a tool shed repo, however, if that's not the recommended course of action, then I'd like some advice on what to do?
There is a feature request, that you can specify in your toolshed-repository the preferred location in the tool panel. For now I would ignore it and just install the tools somewhere and hope that feature request is quiet high in Gregs ToDo list :) Cheers, Bjoern
Thanks!
Sincerely, Michael E. Cotterell
Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Faculty Liaison, CS Graduate Student Association, University of Georgia mepcotterell@gmail.com (mailto:mepcotterell@gmail.com) mepcott@uga.edu (mailto:mepcott@uga.edu) mec@cs.uga.edu (mailto:mec@cs.uga.edu) http://michaelcotterell.com/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi, Michael, As others have said, in the long term, tool wrappers and their dependencies will be distributed and installed through tool sheds rather than being distributed with Galaxy source, so you might want to plan to generate and upload tool shed compatible archives from the get-go. The test and main tool sheds contain plenty of examples and Björn's tools exercise pretty much all of the functionality - you can browse the repo structure which is identical to the gz archive structure you need to upload to a new repo, or download the repo as a gz, unpack it and get exactly the kind of directory structure and contents you need to emulate for your tools. Once you have a working tool packaging the archive up is straightforward. The format for tool shed repo uploads and syntax for the tags used to define dependencies is very well documented in the tool shed section of the wiki. As Björn points out, the tool factory python wrapper might be a useful source of ideas and perhaps code. Your tool generator will need to do something similar to write the content and generate complete tool shed archives. When generating a new tool, the TF uses an ugly XML wrapper generator (contributed improvements would be very welcome!) and (probably more usefully) the few lines of code you need to package up the functional test data and the XML and wrapper if you need one in toolshed archive format. On Thu, Jul 18, 2013 at 5:30 AM, Björn Grüning < bjoern.gruening@pharmazie.uni-freiburg.de> wrote:
Hi Michael,
I think you will enter new ground with your tool. The closest tool that will do something similar is Ross toolfactory, I think:
http://toolshed.g2.bx.psu.edu/view/fubar/toolfactory
For me one question is, do you really want to offer that service to every user and is it save?
TL;DR I have a tool that generates other tools. Where should those tools go, and how should I add them (without modifying Galaxy code)?
I apologize, in advance, for the wall of text. I would like some advice on what the best way to go about setting up a tool that generates other tools...
As some of you (especially those who saw my boss's talk at GCC) already know, I'm working on a tool that lets galaxy users add web service operations as tools to Galaxy. We have a general purpose client for invoking web services, and we have a tool that let's a user enter a WSDL/WADL URL, select the operations they want to add, and generates tool config XML files for Galaxy.
Right now, I'm in the process of making our tool "tool shed" compatible, and that's going along great (I especially like the tool dependency installation stuff), but I'm trying to determine the best course of action on the following:
1. Where is the best place to put the tool config (XML) files that are generated by my tool? My predecessors just placed them in a subdirectory under our tool's directory. I'm ok with this, but does that mean that all the generated files will get removed if they upgrade our tool via the tool shed (haven't taken the time to see if Galaxy just does an "hg pull" or if it's more complicated than that)?
It is mainly a hg pull. But if you deinstall that repository and reinstall it, all your files are lost.
Would it be better to have my tool create a local tool shed repository (just for these generated tools), and if so, can the tool shed API let my tool install this repository? Having not looked at install_tool_shed_repositories.py yet, it's unclear from the wiki documentation whether or not POST /api/tool_shed_repositories/install_repository_revision requires that the tool shed already registered in tool_sheds_conf.xml)?
I think that is a better way to do it. The API can be used to install your tools. But it need to be configured beforehand from the user, administrator.
2. What's the best way to add the generated tools to Galaxy? My predecessors modified the tool_config.xml directly and required users to restart Galaxy. Now, If these tools are in a tool shed repo, I know that I can use the add_to_tool_panel() method in lib/tool_shed/util/tool_util.py if these tools are in a tool shed repo, however, if that's not the recommended course of action, then I'd like some advice on what to do?
There is a feature request, that you can specify in your toolshed-repository the preferred location in the tool panel. For now I would ignore it and just install the tools somewhere and hope that feature request is quiet high in Gregs ToDo list :)
Cheers, Bjoern
Ross, Thanks for the comments. I've already made my main tool "tool_shed" compatible. What I'm wondering about is what do I do with the tools that my tool creates? Sincerely, Michael E. Cotterell Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Faculty Liaison, CS Graduate Student Association, University of Georgia mepcotterell@gmail.com (mailto:mepcotterell@gmail.com) mepcott@uga.edu (mailto:mepcott@uga.edu) mec@cs.uga.edu (mailto:mec@cs.uga.edu) http://michaelcotterell.com/ On Wednesday, July 17, 2013 at 7:44 PM, Ross wrote:
Hi, Michael,
As others have said, in the long term, tool wrappers and their dependencies will be distributed and installed through tool sheds rather than being distributed with Galaxy source, so you might want to plan to generate and upload tool shed compatible archives from the get-go. The test and main tool sheds contain plenty of examples and Björn's tools exercise pretty much all of the functionality - you can browse the repo structure which is identical to the gz archive structure you need to upload to a new repo, or download the repo as a gz, unpack it and get exactly the kind of directory structure and contents you need to emulate for your tools. Once you have a working tool packaging the archive up is straightforward.
The format for tool shed repo uploads and syntax for the tags used to define dependencies is very well documented in the tool shed section of the wiki. As Björn points out, the tool factory python wrapper might be a useful source of ideas and perhaps code. Your tool generator will need to do something similar to write the content and generate complete tool shed archives. When generating a new tool, the TF uses an ugly XML wrapper generator (contributed improvements would be very welcome!) and (probably more usefully) the few lines of code you need to package up the functional test data and the XML and wrapper if you need one in toolshed archive format.
On Thu, Jul 18, 2013 at 5:30 AM, Björn Grüning <bjoern.gruening@pharmazie.uni-freiburg.de (mailto:bjoern.gruening@pharmazie.uni-freiburg.de)> wrote:
Hi Michael,
I think you will enter new ground with your tool. The closest tool that will do something similar is Ross toolfactory, I think:
http://toolshed.g2.bx.psu.edu/view/fubar/toolfactory
For me one question is, do you really want to offer that service to every user and is it save?
TL;DR I have a tool that generates other tools. Where should those tools go, and how should I add them (without modifying Galaxy code)?
I apologize, in advance, for the wall of text. I would like some advice on what the best way to go about setting up a tool that generates other tools...
As some of you (especially those who saw my boss's talk at GCC) already know, I'm working on a tool that lets galaxy users add web service operations as tools to Galaxy. We have a general purpose client for invoking web services, and we have a tool that let's a user enter a WSDL/WADL URL, select the operations they want to add, and generates tool config XML files for Galaxy.
Right now, I'm in the process of making our tool "tool shed" compatible, and that's going along great (I especially like the tool dependency installation stuff), but I'm trying to determine the best course of action on the following:
1. Where is the best place to put the tool config (XML) files that are generated by my tool? My predecessors just placed them in a subdirectory under our tool's directory. I'm ok with this, but does that mean that all the generated files will get removed if they upgrade our tool via the tool shed (haven't taken the time to see if Galaxy just does an "hg pull" or if it's more complicated than that)?
It is mainly a hg pull. But if you deinstall that repository and reinstall it, all your files are lost.
Would it be better to have my tool create a local tool shed repository (just for these generated tools), and if so, can the tool shed API let my tool install this repository? Having not looked at install_tool_shed_repositories.py yet, it's unclear from the wiki documentation whether or not POST /api/tool_shed_repositories/install_repository_revision requires that the tool shed already registered in tool_sheds_conf.xml)?
I think that is a better way to do it. The API can be used to install your tools. But it need to be configured beforehand from the user, administrator.
2. What's the best way to add the generated tools to Galaxy? My predecessors modified the tool_config.xml directly and required users to restart Galaxy. Now, If these tools are in a tool shed repo, I know that I can use the add_to_tool_panel() method in lib/tool_shed/util/tool_util.py if these tools are in a tool shed repo, however, if that's not the recommended course of action, then I'd like some advice on what to do?
There is a feature request, that you can specify in your toolshed-repository the preferred location in the tool panel. For now I would ignore it and just install the tools somewhere and hope that feature request is quiet high in Gregs ToDo list :)
Cheers, Bjoern
Sorry if my post was rambling and unclear. Here's the executive summary for what it's worth: IMHO, a Galaxy tool that generates tools should emit them as tool shed compatible artefacts, ready for uploading to a TS repository for automated installation to any target instance. On Thu, Jul 18, 2013 at 11:25 AM, Michael E. Cotterell < mepcotterell@gmail.com> wrote:
Ross,
Thanks for the comments.
I've already made my main tool "tool_shed" compatible. What I'm wondering about is what do I do with the tools that my tool creates?
Sincerely, Michael E. Cotterell
Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Faculty Liaison, CS Graduate Student Association, University of Georgia mepcotterell@gmail.com (mailto:mepcotterell@gmail.com) mepcott@uga.edu (mailto:mepcott@uga.edu) mec@cs.uga.edu (mailto:mec@cs.uga.edu) http://michaelcotterell.com/
On Wednesday, July 17, 2013 at 7:44 PM, Ross wrote:
Hi, Michael,
As others have said, in the long term, tool wrappers and their dependencies will be distributed and installed through tool sheds rather than being distributed with Galaxy source, so you might want to plan to generate and upload tool shed compatible archives from the get-go. The test and main tool sheds contain plenty of examples and Björn's tools exercise pretty much all of the functionality - you can browse the repo structure which is identical to the gz archive structure you need to upload to a new repo, or download the repo as a gz, unpack it and get exactly the kind of directory structure and contents you need to emulate for your tools. Once you have a working tool packaging the archive up is straightforward.
The format for tool shed repo uploads and syntax for the tags used to define dependencies is very well documented in the tool shed section of the wiki. As Björn points out, the tool factory python wrapper might be a useful source of ideas and perhaps code. Your tool generator will need to do something similar to write the content and generate complete tool shed archives. When generating a new tool, the TF uses an ugly XML wrapper generator (contributed improvements would be very welcome!) and (probably more usefully) the few lines of code you need to package up the functional test data and the XML and wrapper if you need one in toolshed archive format.
Hi Michael,
I think you will enter new ground with your tool. The closest tool that will do something similar is Ross toolfactory, I think:
http://toolshed.g2.bx.psu.edu/view/fubar/toolfactory
For me one question is, do you really want to offer that service to every user and is it save?
TL;DR I have a tool that generates other tools. Where should those tools go, and how should I add them (without modifying Galaxy code)?
I apologize, in advance, for the wall of text. I would like some advice on what the best way to go about setting up a tool that generates other tools...
As some of you (especially those who saw my boss's talk at GCC) already know, I'm working on a tool that lets galaxy users add web service operations as tools to Galaxy. We have a general purpose client for invoking web services, and we have a tool that let's a user enter a WSDL/WADL URL, select the operations
Right now, I'm in the process of making our tool "tool shed"
compatible, and that's going along great
(I especially like the tool dependency installation stuff), but I'm
1. Where is the best place to put the tool config (XML) files that
are generated by my tool? My predecessors just placed them in a subdirectory under our tool's directory.
I'm ok with this, but does that mean that all the generated files will get removed if they upgrade our tool via the tool shed (haven't taken
Galaxy just does an "hg pull" or if it's more complicated than that)?
It is mainly a hg pull. But if you deinstall that repository and reinstall it, all your files are lost.
Would it be better to have my tool create a local tool shed repository (just for these generated tools), and if so, can the tool shed API let my tool install this repository? Having not looked at install_tool_shed_repositories.py yet, it's unclear from the wiki documentation whether or not POST /api/tool_shed_repositories/install_repository_revision requires that the tool shed already registered in tool_sheds_conf.xml)?
I think that is a better way to do it. The API can be used to install your tools. But it need to be configured beforehand from the user, administrator.
2. What's the best way to add the generated tools to Galaxy? My
Now, If these tools are in a tool shed repo, I know that I can use
On Thu, Jul 18, 2013 at 5:30 AM, Björn Grüning < bjoern.gruening@pharmazie.uni-freiburg.de (mailto: bjoern.gruening@pharmazie.uni-freiburg.de)> wrote: they want to add, and generates tool config XML files for Galaxy. trying to determine the best course of action on the following: the time to see if predecessors modified the tool_config.xml directly and required users to restart Galaxy. the add_to_tool_panel() method in lib/tool_shed/util/tool_util.py if these tools are in a tool shed repo, however,
if that's not the recommended course of action, then I'd like some advice on what to do?
There is a feature request, that you can specify in your toolshed-repository the preferred location in the tool panel. For now I would ignore it and just install the tools somewhere and hope that feature request is quiet high in Gregs ToDo list :)
Cheers, Bjoern
I didn't mean to imply it was unclear. It definitely made sense. I completely agree with you. However, is there for a way (maybe via the toolshed api) for my tool to push these tools to a TS repo? Also, doesn't that add some more complexity for the user? Not everyone runs a local tool shed. Sincerely, Michael E. Cotterell Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Faculty Liaison, CS Graduate Student Association, University of Georgia mepcotterell@gmail.com (mailto:mepcotterell@gmail.com) mepcott@uga.edu (mailto:mepcott@uga.edu) mec@cs.uga.edu (mailto:mec@cs.uga.edu) http://michaelcotterell.com/ On Wednesday, July 17, 2013 at 10:05 PM, Ross wrote:
Sorry if my post was rambling and unclear.
Here's the executive summary for what it's worth: IMHO, a Galaxy tool that generates tools should emit them as tool shed compatible artefacts, ready for uploading to a TS repository for automated installation to any target instance.
On Thu, Jul 18, 2013 at 11:25 AM, Michael E. Cotterell <mepcotterell@gmail.com (mailto:mepcotterell@gmail.com)> wrote:
Ross,
Thanks for the comments.
I've already made my main tool "tool_shed" compatible. What I'm wondering about is what do I do with the tools that my tool creates?
Sincerely, Michael E. Cotterell
Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Faculty Liaison, CS Graduate Student Association, University of Georgia mepcotterell@gmail.com (mailto:mepcotterell@gmail.com) (mailto:mepcotterell@gmail.com) mepcott@uga.edu (mailto:mepcott@uga.edu) (mailto:mepcott@uga.edu) mec@cs.uga.edu (mailto:mec@cs.uga.edu) (mailto:mec@cs.uga.edu) http://michaelcotterell.com/
On Wednesday, July 17, 2013 at 7:44 PM, Ross wrote:
Hi, Michael,
As others have said, in the long term, tool wrappers and their dependencies will be distributed and installed through tool sheds rather than being distributed with Galaxy source, so you might want to plan to generate and upload tool shed compatible archives from the get-go. The test and main tool sheds contain plenty of examples and Björn's tools exercise pretty much all of the functionality - you can browse the repo structure which is identical to the gz archive structure you need to upload to a new repo, or download the repo as a gz, unpack it and get exactly the kind of directory structure and contents you need to emulate for your tools. Once you have a working tool packaging the archive up is straightforward.
The format for tool shed repo uploads and syntax for the tags used to define dependencies is very well documented in the tool shed section of the wiki. As Björn points out, the tool factory python wrapper might be a useful source of ideas and perhaps code. Your tool generator will need to do something similar to write the content and generate complete tool shed archives. When generating a new tool, the TF uses an ugly XML wrapper generator (contributed improvements would be very welcome!) and (probably more usefully) the few lines of code you need to package up the functional test data and the XML and wrapper if you need one in toolshed archive format.
On Thu, Jul 18, 2013 at 5:30 AM, Björn Grüning <bjoern.gruening@pharmazie.uni-freiburg.de (mailto:bjoern.gruening@pharmazie.uni-freiburg.de) (mailto:bjoern.gruening@pharmazie.uni-freiburg.de)> wrote:
Hi Michael,
I think you will enter new ground with your tool. The closest tool that will do something similar is Ross toolfactory, I think:
http://toolshed.g2.bx.psu.edu/view/fubar/toolfactory
For me one question is, do you really want to offer that service to every user and is it save?
TL;DR I have a tool that generates other tools. Where should those tools go, and how should I add them (without modifying Galaxy code)?
I apologize, in advance, for the wall of text. I would like some advice on what the best way to go about setting up a tool that generates other tools...
As some of you (especially those who saw my boss's talk at GCC) already know, I'm working on a tool that lets galaxy users add web service operations as tools to Galaxy. We have a general purpose client for invoking web services, and we have a tool that let's a user enter a WSDL/WADL URL, select the operations they want to add, and generates tool config XML files for Galaxy.
Right now, I'm in the process of making our tool "tool shed" compatible, and that's going along great (I especially like the tool dependency installation stuff), but I'm trying to determine the best course of action on the following:
1. Where is the best place to put the tool config (XML) files that are generated by my tool? My predecessors just placed them in a subdirectory under our tool's directory. I'm ok with this, but does that mean that all the generated files will get removed if they upgrade our tool via the tool shed (haven't taken the time to see if Galaxy just does an "hg pull" or if it's more complicated than that)?
It is mainly a hg pull. But if you deinstall that repository and reinstall it, all your files are lost.
Would it be better to have my tool create a local tool shed repository (just for these generated tools), and if so, can the tool shed API let my tool install this repository? Having not looked at install_tool_shed_repositories.py yet, it's unclear from the wiki documentation whether or not POST /api/tool_shed_repositories/install_repository_revision requires that the tool shed already registered in tool_sheds_conf.xml)?
I think that is a better way to do it. The API can be used to install your tools. But it need to be configured beforehand from the user, administrator.
2. What's the best way to add the generated tools to Galaxy? My predecessors modified the tool_config.xml directly and required users to restart Galaxy. Now, If these tools are in a tool shed repo, I know that I can use the add_to_tool_panel() method in lib/tool_shed/util/tool_util.py if these tools are in a tool shed repo, however, if that's not the recommended course of action, then I'd like some advice on what to do?
There is a feature request, that you can specify in your toolshed-repository the preferred location in the tool panel. For now I would ignore it and just install the tools somewhere and hope that feature request is quiet high in Gregs ToDo list :)
Cheers, Bjoern
participants (4)
-
Björn Grüning -
Michael E. Cotterell -
Peter Cock -
Ross