AFAIK it's not yet possible, but a tool shed API to expose subversion hooks for repository creation and update is a logical development which is already on the radar AFAIK. For the bigger picture of complexity for the user, I think some key clarifications required include: which 'user' is experiencing complexity; how big a barrier it is to generating tools; how readily can we make it go away; how much will it help? Tool builders (even 'users' of the TF) might reasonably be expected to have a capacity for substantial complexity that the end user of the generated tools should not need to possess - after all, once a generated tool is installed in Galaxy, the Galaxy user can't tell the difference and sees no more complexity from the generated tool than a hand written one. On absent local tool sheds: it's trivial to run a toolshed on a dev laptop and not hard to run in production - but tool makers can always use the public test and main TS if they want to distribute new tools so I'm not sure how big a barrier a missing local toolshed really is. On Thu, Jul 18, 2013 at 12:08 PM, Michael E. Cotterell < mepcotterell@gmail.com> wrote:
I didn't mean to imply it was unclear. It definitely made sense.
I completely agree with you. However, is there for a way (maybe via the toolshed api) for my tool to push these tools to a TS repo? Also, doesn't that add some more complexity for the user? Not everyone runs a local tool shed.
Sincerely, Michael E. Cotterell
Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Faculty Liaison, CS Graduate Student Association, University of Georgia mepcotterell@gmail.com (mailto:mepcotterell@gmail.com) mepcott@uga.edu (mailto:mepcott@uga.edu) mec@cs.uga.edu (mailto:mec@cs.uga.edu) http://michaelcotterell.com/
On Wednesday, July 17, 2013 at 10:05 PM, Ross wrote:
Sorry if my post was rambling and unclear.
Here's the executive summary for what it's worth: IMHO, a Galaxy tool that generates tools should emit them as tool shed compatible artefacts, ready for uploading to a TS repository for automated installation to any target instance.
Ross,
Thanks for the comments.
I've already made my main tool "tool_shed" compatible. What I'm wondering about is what do I do with the tools that my tool creates?
Sincerely, Michael E. Cotterell
Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Faculty Liaison, CS Graduate Student Association, University of Georgia mepcotterell@gmail.com (mailto:mepcotterell@gmail.com) (mailto: mepcotterell@gmail.com) mepcott@uga.edu (mailto:mepcott@uga.edu) (mailto:mepcott@uga.edu) mec@cs.uga.edu (mailto:mec@cs.uga.edu) (mailto:mec@cs.uga.edu) http://michaelcotterell.com/
On Wednesday, July 17, 2013 at 7:44 PM, Ross wrote:
Hi, Michael,
As others have said, in the long term, tool wrappers and their dependencies will be distributed and installed through tool sheds rather
The format for tool shed repo uploads and syntax for the tags used
to define dependencies is very well documented in the tool shed section of
On Thu, Jul 18, 2013 at 5:30 AM, Björn Grüning <
bjoern.gruening@pharmazie.uni-freiburg.de (mailto: bjoern.gruening@pharmazie.uni-freiburg.de) (mailto: bjoern.gruening@pharmazie.uni-freiburg.de)> wrote:
Hi Michael,
I think you will enter new ground with your tool. The closest tool
will do something similar is Ross toolfactory, I think:
http://toolshed.g2.bx.psu.edu/view/fubar/toolfactory
For me one question is, do you really want to offer that service to every user and is it save?
TL;DR I have a tool that generates other tools. Where should
I apologize, in advance, for the wall of text. I would like some
advice on what the best way to go about setting up a tool that generates other tools...
As some of you (especially those who saw my boss's talk at GCC)
already know, I'm working on a tool that lets galaxy users add web service operations as tools to Galaxy.
We have a general purpose client for invoking web services, and we have a tool that let's a user enter a WSDL/WADL URL, select the operations they want to add, and generates tool config XML files for Galaxy.
Right now, I'm in the process of making our tool "tool shed" compatible, and that's going along great (I especially like the tool dependency installation stuff), but I'm trying to determine the best course of action on the following:
1. Where is the best place to put the tool config (XML) files
I'm ok with this, but does that mean that all the generated files will get removed if they upgrade our tool via the tool shed (haven't taken the time to see if Galaxy just does an "hg pull" or if it's more complicated than
It is mainly a hg pull. But if you deinstall that repository and reinstall it, all your files are lost.
Would it be better to have my tool create a local tool shed
repository
(just for these generated tools), and if so, can the tool shed API let my tool install this repository? Having not looked at install_tool_shed_repositories.py yet, it's unclear from the wiki documentation whether or not POST /api/tool_shed_repositories/install_repository_revision requires that the tool shed already registered in tool_sheds_conf.xml)?
I think that is a better way to do it. The API can be used to install your tools. But it need to be configured beforehand from the user, administrator.
2. What's the best way to add the generated tools to Galaxy? My
Now, If these tools are in a tool shed repo, I know that I can use the add_to_tool_panel() method in lib/tool_shed/util/tool_util.py if
On Thu, Jul 18, 2013 at 11:25 AM, Michael E. Cotterell < mepcotterell@gmail.com (mailto:mepcotterell@gmail.com)> wrote: than being distributed with Galaxy source, so you might want to plan to generate and upload tool shed compatible archives from the get-go. The test and main tool sheds contain plenty of examples and Björn's tools exercise pretty much all of the functionality - you can browse the repo structure which is identical to the gz archive structure you need to upload to a new repo, or download the repo as a gz, unpack it and get exactly the kind of directory structure and contents you need to emulate for your tools. Once you have a working tool packaging the archive up is straightforward. the wiki. As Björn points out, the tool factory python wrapper might be a useful source of ideas and perhaps code. Your tool generator will need to do something similar to write the content and generate complete tool shed archives. When generating a new tool, the TF uses an ugly XML wrapper generator (contributed improvements would be very welcome!) and (probably more usefully) the few lines of code you need to package up the functional test data and the XML and wrapper if you need one in toolshed archive format. that those tools go, and how should I add them (without modifying Galaxy code)? that are generated by my tool? My predecessors just placed them in a subdirectory under our tool's directory. that)? predecessors modified the tool_config.xml directly and required users to restart Galaxy. these tools are in a tool shed repo, however,
if that's not the recommended course of action, then I'd like some advice on what to do?
There is a feature request, that you can specify in your toolshed-repository the preferred location in the tool panel. For now I would ignore it and just install the tools somewhere and hope that feature request is quiet high in Gregs ToDo list :)
Cheers, Bjoern
-- Ross Lazarus MBBS MPH; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444 http://scholar.google.com/citations?hl=en&user=UCUuEM4AAAAJ
This doesn't seem to fit into the use case that my colleagues have described to me. Our tool, used within Galaxy, creates/builds/generates new tools for Web service operations. Once a user uses our tool to generate a Web service tool, it should be visible and usable to them. Currently, in order to accomplish this, we have to manually modify the tool_config.xml file, however, I don't think we should release a tool that does that. Sincerely, Michael E. Cotterell Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Faculty Liaison, CS Graduate Student Association, University of Georgia mepcotterell@gmail.com (mailto:mepcotterell@gmail.com) mepcott@uga.edu (mailto:mepcott@uga.edu) mec@cs.uga.edu (mailto:mec@cs.uga.edu) http://michaelcotterell.com/ On Wednesday, July 17, 2013 at 10:26 PM, Ross wrote:
AFAIK it's not yet possible, but a tool shed API to expose subversion hooks for repository creation and update is a logical development which is already on the radar AFAIK.
For the bigger picture of complexity for the user, I think some key clarifications required include: which 'user' is experiencing complexity; how big a barrier it is to generating tools; how readily can we make it go away; how much will it help?
Tool builders (even 'users' of the TF) might reasonably be expected to have a capacity for substantial complexity that the end user of the generated tools should not need to possess - after all, once a generated tool is installed in Galaxy, the Galaxy user can't tell the difference and sees no more complexity from the generated tool than a hand written one.
On absent local tool sheds: it's trivial to run a toolshed on a dev laptop and not hard to run in production - but tool makers can always use the public test and main TS if they want to distribute new tools so I'm not sure how big a barrier a missing local toolshed really is.
On Thu, Jul 18, 2013 at 12:08 PM, Michael E. Cotterell <mepcotterell@gmail.com (mailto:mepcotterell@gmail.com)> wrote:
I didn't mean to imply it was unclear. It definitely made sense.
I completely agree with you. However, is there for a way (maybe via the toolshed api) for my tool to push these tools to a TS repo? Also, doesn't that add some more complexity for the user? Not everyone runs a local tool shed.
Sincerely, Michael E. Cotterell
Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Faculty Liaison, CS Graduate Student Association, University of Georgia mepcotterell@gmail.com (mailto:mepcotterell@gmail.com) (mailto:mepcotterell@gmail.com) mepcott@uga.edu (mailto:mepcott@uga.edu) (mailto:mepcott@uga.edu) mec@cs.uga.edu (mailto:mec@cs.uga.edu) (mailto:mec@cs.uga.edu) http://michaelcotterell.com/
On Wednesday, July 17, 2013 at 10:05 PM, Ross wrote:
Sorry if my post was rambling and unclear.
Here's the executive summary for what it's worth: IMHO, a Galaxy tool that generates tools should emit them as tool shed compatible artefacts, ready for uploading to a TS repository for automated installation to any target instance.
On Thu, Jul 18, 2013 at 11:25 AM, Michael E. Cotterell <mepcotterell@gmail.com (mailto:mepcotterell@gmail.com) (mailto:mepcotterell@gmail.com)> wrote:
Ross,
Thanks for the comments.
I've already made my main tool "tool_shed" compatible. What I'm wondering about is what do I do with the tools that my tool creates?
Sincerely, Michael E. Cotterell
Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Faculty Liaison, CS Graduate Student Association, University of Georgia mepcotterell@gmail.com (mailto:mepcotterell@gmail.com) (mailto:mepcotterell@gmail.com) (mailto:mepcotterell@gmail.com) mepcott@uga.edu (mailto:mepcott@uga.edu) (mailto:mepcott@uga.edu) (mailto:mepcott@uga.edu) mec@cs.uga.edu (mailto:mec@cs.uga.edu) (mailto:mec@cs.uga.edu) (mailto:mec@cs.uga.edu) http://michaelcotterell.com/
On Wednesday, July 17, 2013 at 7:44 PM, Ross wrote:
Hi, Michael,
As others have said, in the long term, tool wrappers and their dependencies will be distributed and installed through tool sheds rather than being distributed with Galaxy source, so you might want to plan to generate and upload tool shed compatible archives from the get-go. The test and main tool sheds contain plenty of examples and Björn's tools exercise pretty much all of the functionality - you can browse the repo structure which is identical to the gz archive structure you need to upload to a new repo, or download the repo as a gz, unpack it and get exactly the kind of directory structure and contents you need to emulate for your tools. Once you have a working tool packaging the archive up is straightforward.
The format for tool shed repo uploads and syntax for the tags used to define dependencies is very well documented in the tool shed section of the wiki. As Björn points out, the tool factory python wrapper might be a useful source of ideas and perhaps code. Your tool generator will need to do something similar to write the content and generate complete tool shed archives. When generating a new tool, the TF uses an ugly XML wrapper generator (contributed improvements would be very welcome!) and (probably more usefully) the few lines of code you need to package up the functional test data and the XML and wrapper if you need one in toolshed archive format.
On Thu, Jul 18, 2013 at 5:30 AM, Björn Grüning <bjoern.gruening@pharmazie.uni-freiburg.de (mailto:bjoern.gruening@pharmazie.uni-freiburg.de) (mailto:bjoern.gruening@pharmazie.uni-freiburg.de) (mailto:bjoern.gruening@pharmazie.uni-freiburg.de)> wrote:
Hi Michael,
I think you will enter new ground with your tool. The closest tool that will do something similar is Ross toolfactory, I think:
http://toolshed.g2.bx.psu.edu/view/fubar/toolfactory
For me one question is, do you really want to offer that service to every user and is it save?
> TL;DR I have a tool that generates other tools. Where should those tools go, and how should I add them (without modifying Galaxy code)? > > I apologize, in advance, for the wall of text. I would like some advice on what the best way to go about setting up a tool that generates other tools... > > As some of you (especially those who saw my boss's talk at GCC) already know, I'm working on a tool that lets galaxy users add web service operations as tools to Galaxy. > We have a general purpose client for invoking web services, and we have a tool that let's a user enter a WSDL/WADL URL, select the operations they want to add, and generates tool config XML files for Galaxy. > > Right now, I'm in the process of making our tool "tool shed" compatible, and that's going along great > (I especially like the tool dependency installation stuff), but I'm trying to determine the best course of action on the following: > > 1. Where is the best place to put the tool config (XML) files that are generated by my tool? My predecessors just placed them in a subdirectory under our tool's directory. > I'm ok with this, but does that mean that all the generated files will get removed if they upgrade our tool via the tool shed (haven't taken the time to see if > Galaxy just does an "hg pull" or if it's more complicated than that)?
It is mainly a hg pull. But if you deinstall that repository and reinstall it, all your files are lost.
> Would it be better to have my tool create a local tool shed repository > (just for these generated tools), and if so, can the tool shed API let my tool install this repository? Having not looked at install_tool_shed_repositories.py yet, > it's unclear from the wiki documentation whether or not POST /api/tool_shed_repositories/install_repository_revision requires that the tool shed already registered in tool_sheds_conf.xml)?
I think that is a better way to do it. The API can be used to install your tools. But it need to be configured beforehand from the user, administrator.
> 2. What's the best way to add the generated tools to Galaxy? My predecessors modified the tool_config.xml directly and required users to restart Galaxy. > Now, If these tools are in a tool shed repo, I know that I can use the add_to_tool_panel() method in lib/tool_shed/util/tool_util.py if these tools are in a tool shed repo, however, > if that's not the recommended course of action, then I'd like some advice on what to do?
There is a feature request, that you can specify in your toolshed-repository the preferred location in the tool panel. For now I would ignore it and just install the tools somewhere and hope that feature request is quiet high in Gregs ToDo list :)
Cheers, Bjoern
-- Ross Lazarus MBBS MPH; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444 http://scholar.google.com/citations?hl=en&user=UCUuEM4AAAAJ ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Ah. Forgive me for not paying attention to that requirement. Tools usually run as the Galaxy user so only local admins can do that at present because allowing a non-admin user to install a newly generated tool, especially on a public site opens up some interesting security challenges. I think it would be hard to sell as a new feature unless you've figured out how to prevent potential mischief from miscreant users? On Thu, Jul 18, 2013 at 11:33 PM, Michael E. Cotterell < mepcotterell@gmail.com> wrote:
This doesn't seem to fit into the use case that my colleagues have described to me. Our tool, used within Galaxy, creates/builds/generates new tools for Web service operations. Once a user uses our tool to generate a Web service tool, it should be visible and usable to them.
Currently, in order to accomplish this, we have to manually modify the tool_config.xml file, however, I don't think we should release a tool that does that.
Sincerely, Michael E. Cotterell
Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Faculty Liaison, CS Graduate Student Association, University of Georgia mepcotterell@gmail.com (mailto:mepcotterell@gmail.com) mepcott@uga.edu (mailto:mepcott@uga.edu) mec@cs.uga.edu (mailto:mec@cs.uga.edu) http://michaelcotterell.com/
On Wednesday, July 17, 2013 at 10:26 PM, Ross wrote:
AFAIK it's not yet possible, but a tool shed API to expose subversion hooks for repository creation and update is a logical development which is already on the radar AFAIK.
For the bigger picture of complexity for the user, I think some key clarifications required include: which 'user' is experiencing complexity; how big a barrier it is to generating tools; how readily can we make it go away; how much will it help?
Tool builders (even 'users' of the TF) might reasonably be expected to have a capacity for substantial complexity that the end user of the generated tools should not need to possess - after all, once a generated tool is installed in Galaxy, the Galaxy user can't tell the difference and sees no more complexity from the generated tool than a hand written one.
On absent local tool sheds: it's trivial to run a toolshed on a dev laptop and not hard to run in production - but tool makers can always use the public test and main TS if they want to distribute new tools so I'm not sure how big a barrier a missing local toolshed really is.
I didn't mean to imply it was unclear. It definitely made sense.
I completely agree with you. However, is there for a way (maybe via
On Thu, Jul 18, 2013 at 12:08 PM, Michael E. Cotterell < mepcotterell@gmail.com (mailto:mepcotterell@gmail.com)> wrote: the toolshed api) for my tool to push these tools to a TS repo? Also, doesn't that add some more complexity for the user? Not everyone runs a local tool shed.
Sincerely, Michael E. Cotterell
Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Faculty Liaison, CS Graduate Student Association, University of Georgia mepcotterell@gmail.com (mailto:mepcotterell@gmail.com) (mailto:
mepcott@uga.edu (mailto:mepcott@uga.edu) (mailto:mepcott@uga.edu) mec@cs.uga.edu (mailto:mec@cs.uga.edu) (mailto:mec@cs.uga.edu) http://michaelcotterell.com/
On Wednesday, July 17, 2013 at 10:05 PM, Ross wrote:
Sorry if my post was rambling and unclear.
Here's the executive summary for what it's worth: IMHO, a Galaxy tool that generates tools should emit them as tool shed compatible artefacts, ready for uploading to a TS repository for automated installation to any target instance.
On Thu, Jul 18, 2013 at 11:25 AM, Michael E. Cotterell < mepcotterell@gmail.com (mailto:mepcotterell@gmail.com) (mailto: mepcotterell@gmail.com)> wrote:
Ross,
Thanks for the comments.
I've already made my main tool "tool_shed" compatible. What I'm wondering about is what do I do with the tools that my tool creates?
Sincerely, Michael E. Cotterell
Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Faculty Liaison, CS Graduate Student Association, University of Georgia mepcotterell@gmail.com (mailto:mepcotterell@gmail.com) (mailto: mepcotterell@gmail.com) (mailto:mepcotterell@gmail.com) mepcott@uga.edu (mailto:mepcott@uga.edu) (mailto:mepcott@uga.edu) (mailto:mepcott@uga.edu) mec@cs.uga.edu (mailto:mec@cs.uga.edu) (mailto:mec@cs.uga.edu) (mailto:mec@cs.uga.edu) http://michaelcotterell.com/
On Wednesday, July 17, 2013 at 7:44 PM, Ross wrote:
Hi, Michael,
As others have said, in the long term, tool wrappers and their dependencies will be distributed and installed through tool sheds rather
The format for tool shed repo uploads and syntax for the tags
used to define dependencies is very well documented in the tool shed
On Thu, Jul 18, 2013 at 5:30 AM, Björn Grüning <
bjoern.gruening@pharmazie.uni-freiburg.de (mailto: bjoern.gruening@pharmazie.uni-freiburg.de) (mailto: bjoern.gruening@pharmazie.uni-freiburg.de) (mailto: bjoern.gruening@pharmazie.uni-freiburg.de)> wrote:
> Hi Michael, > > I think you will enter new ground with your tool. The closest tool that > will do something similar is Ross toolfactory, I think: > > http://toolshed.g2.bx.psu.edu/view/fubar/toolfactory > > For me one question is, do you really want to offer that service to > every user and is it save? > > > TL;DR I have a tool that generates other tools. Where should
> > > > I apologize, in advance, for the wall of text. I would like some advice on what the best way to go about setting up a tool that generates other tools... > > > > As some of you (especially those who saw my boss's talk at GCC) already know, I'm working on a tool that lets galaxy users add web service operations as tools to Galaxy. > > We have a general purpose client for invoking web services, and we have a tool that let's a user enter a WSDL/WADL URL, select the operations they want to add, and generates tool config XML files for Galaxy. > > > > Right now, I'm in the process of making our tool "tool shed" compatible, and that's going along great > > (I especially like the tool dependency installation stuff), but I'm trying to determine the best course of action on the following: > > > > 1. Where is the best place to put the tool config (XML) files that are generated by my tool? My predecessors just placed them in a subdirectory under our tool's directory. > > I'm ok with this, but does that mean that all the generated files will get removed if they upgrade our tool via the tool shed (haven't taken the time to see if > > Galaxy just does an "hg pull" or if it's more complicated
> > > > > It is mainly a hg pull. But if you deinstall that repository and > reinstall it, all your files are lost. > > > Would it be better to have my tool create a local tool shed repository > > (just for these generated tools), and if so, can the tool shed API let my tool install this repository? Having not looked at install_tool_shed_repositories.py yet, > > it's unclear from the wiki documentation whether or not POST /api/tool_shed_repositories/install_repository_revision requires that the tool shed already registered in tool_sheds_conf.xml)? > > > > > I think that is a better way to do it. The API can be used to install > your tools. But it need to be configured beforehand from the user, > administrator. > > > 2. What's the best way to add the generated tools to Galaxy? My predecessors modified the tool_config.xml directly and required users to restart Galaxy. > > Now, If these tools are in a tool shed repo, I know that I can use the add_to_tool_panel() method in lib/tool_shed/util/tool_util.py if these tools are in a tool shed repo, however, > > if that's not the recommended course of action, then I'd
> > > > > There is a feature request, that you can specify in your > toolshed-repository the preferred location in the tool panel. For now I > would ignore it and just install the tools somewhere and hope
mepcotterell@gmail.com) than being distributed with Galaxy source, so you might want to plan to generate and upload tool shed compatible archives from the get-go. The test and main tool sheds contain plenty of examples and Björn's tools exercise pretty much all of the functionality - you can browse the repo structure which is identical to the gz archive structure you need to upload to a new repo, or download the repo as a gz, unpack it and get exactly the kind of directory structure and contents you need to emulate for your tools. Once you have a working tool packaging the archive up is straightforward. section of the wiki. As Björn points out, the tool factory python wrapper might be a useful source of ideas and perhaps code. Your tool generator will need to do something similar to write the content and generate complete tool shed archives. When generating a new tool, the TF uses an ugly XML wrapper generator (contributed improvements would be very welcome!) and (probably more usefully) the few lines of code you need to package up the functional test data and the XML and wrapper if you need one in toolshed archive format. those tools go, and how should I add them (without modifying Galaxy code)? than that)? like some advice on what to do? that
> feature request is quiet high in Gregs ToDo list :) > > Cheers, > Bjoern
-- Ross Lazarus MBBS MPH; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444 http://scholar.google.com/citations?hl=en&user=UCUuEM4AAAAJ ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
-- Ross Lazarus MBBS MPH; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444 http://scholar.google.com/citations?hl=en&user=UCUuEM4AAAAJ
participants (2)
-
Michael E. Cotterell -
Ross