Implemented adding tools via admin console
Hi all, I've implemented functionality for adding tools via the admin menu. Let me just briefly explain what I did. You can upload a binary/script file to a specific folder (choosable). Then this file is analyzed. The analysis is at the moment only done for Perl/Python script, as those are mostly used in our institute. First, it is checked in what language the script is written in (if it is not a binary), then a syntax check is performed (for python pyflakes must be installed), and then it is checked how many arguments are in the script. Once you uploaded a file, you can add the XML file for this script. The XML file is analyzed as well, in the following matter. First it is checked that the XML is well formed. Then it is checked if the mandatory tags are in the XML ( I choosed for mandatory tags: root = tool, command, input, help ) Then it is checked if script and XML are "compatible" by checking if they have the same number of arguments. I also wrote unit tests for the main methods. They can be executed by "python lib/galaxy/util/test_tool_analysis.py" The main files I changed are lib/galaxy/web/base/controller.py lib/galaxy/util/tool_analysis.py lib/galaxy/util/helper.py lib/galaxy/util/tool_conf_helper.py and the templates for the admin in templates/admin The "update tools" is only done for one web process, there is no hup signal sent at the moment. If you have any questions or something is not working for you, please let me know, I probably have some time to check it. There are still some things that can be / should be implemented later on but I think this can be a good start for you guys. I forked galaxy-central, so you can check it out by cloning https://bitbucket.org/Bodolski/galaxy-central-adding-tools hg clone https://Bodolski@bitbucket.org/Bodolski/galaxy-central-adding-tools If you have any comments or question please do not hesitate to contact me, I'm open for every kind of critics. Best wishes from Germany, Bodo Vossen
On Fri, May 27, 2011 at 11:21 AM, Vossen, Bodo <Bodo.Vossen@mpi-bn.mpg.de> wrote:
Hi all,
I've implemented functionality for adding tools via the admin menu.
That sounds very useful in principle, although perhaps the less flexible alternative of reloading tool_conf.xml would achieve the same aim?
... Then it is checked if script and XML are "compatible" by checking if they have the same number of arguments.
This seems very fragile, and having looked at the code for detecting the number of arguments in a Python script it looks like it will get it wrong in many situations. Even for something quite simple like this: #sys.argv[0] is the script file itself assert len(sys,argv)==4, "Expect 3 arguments" arg1, arg2, arg3 = sys.argv[1:] Also, what about conditional code like this? assert len(sys.argv)==4, "Expect 3 arguments" if sys.argv[1] == "db": database = sys.argv[2] filename = None else: database = None filename = sys.argv[2] out_filename = sys.argv[3] If I have read your get_parameter_number code right, it would claim there are four arguments since there are four lines of the basic form variable = sys.argv[i] Also, it cannot possibly work when there are a varying number of arguments, for example a repeat argument. If you want a test case, my Venn Diagram tool on the Tool Shed should be relevant (its a python script).
If you have any comments or question please do not hesitate to contact me, I'm open for every kind of critics.
I would remove the attempt to check the number of arguments. It is practically impossible to get this right in all cases, so I don't think it is worth doing. Having valid script/xml combinations rejected would be quite annoying. Peter
Hi Peter, thanks a lot for your comments. I think it is more comfortable to be able to upload tools and xml via the webbrowser than to open a terminal and copy them, that was the idea behind my work. Especially for our institute this is very useful. You're right about the check_argument methods. I wasn't sure to add this kind of functionality in the first place, because it can become very tricky. What I did should just be the very first step for someone who wants to continue the work (if this makes sense). That's why I added a checkbox that disable this check, so that you still can upload your tool/xml even the argument test would fail. Do you have some comments about the other parts ? Is this something you would be interested in using ? Best wishes, Bodo Vossen -----Original Message----- From: Peter Cock [mailto:p.j.a.cock@googlemail.com] Sent: Sat 5/28/2011 10:19 PM To: Vossen, Bodo Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Implemented adding tools via admin console On Fri, May 27, 2011 at 11:21 AM, Vossen, Bodo <Bodo.Vossen@mpi-bn.mpg.de> wrote:
Hi all,
I've implemented functionality for adding tools via the admin menu.
That sounds very useful in principle, although perhaps the less flexible alternative of reloading tool_conf.xml would achieve the same aim?
... Then it is checked if script and XML are "compatible" by checking if they have the same number of arguments.
This seems very fragile, and having looked at the code for detecting the number of arguments in a Python script it looks like it will get it wrong in many situations. Even for something quite simple like this: #sys.argv[0] is the script file itself assert len(sys,argv)==4, "Expect 3 arguments" arg1, arg2, arg3 = sys.argv[1:] Also, what about conditional code like this? assert len(sys.argv)==4, "Expect 3 arguments" if sys.argv[1] == "db": database = sys.argv[2] filename = None else: database = None filename = sys.argv[2] out_filename = sys.argv[3] If I have read your get_parameter_number code right, it would claim there are four arguments since there are four lines of the basic form variable = sys.argv[i] Also, it cannot possibly work when there are a varying number of arguments, for example a repeat argument. If you want a test case, my Venn Diagram tool on the Tool Shed should be relevant (its a python script).
If you have any comments or question please do not hesitate to contact me, I'm open for every kind of critics.
I would remove the attempt to check the number of arguments. It is practically impossible to get this right in all cases, so I don't think it is worth doing. Having valid script/xml combinations rejected would be quite annoying. Peter
On Mon, May 30, 2011 at 9:54 AM, Vossen, Bodo wrote:
Hi Peter,
thanks a lot for your comments.
Hi Bodo, No problem :)
I think it is more comfortable to be able to upload tools and xml via the webbrowser than to open a terminal and copy them, that was the idea behind my work. Especially for our institute this is very useful.
For me the command vs the web interface is not important. What I was excited about was adding a tool to Galaxy without having to restart Galaxy and interrupt any users.
You're right about the check_argument methods. I wasn't sure to add this kind of functionality in the first place, because it can become very tricky. What I did should just be the very first step for someone who wants to continue the work (if this makes sense). That's why I added a checkbox that disable this check, so that you still can upload your tool/xml even the argument test would fail.
It might be useful then.
Do you have some comments about the other parts ? Is this something you would be interested in using ?
I've not had a chance to try the code yet, but it could be useful. Peter
Peter, if you were just interested in having a dynamic refresh.... it was posted long time ago and we have it operational. It comes as a tool in your tool list and refreshes all from the tool_conf.xml. Not the loc filtes though! As far as i remember it doesn't kill running jobs. It requires two lines of code in galaxy source and the addition of a tool to the tools section. I pasted it from my archive below. But it is quite self explanatory. Alex Galaxy source adapt for tool refresh
Changes to Galaxy to add refresh toolbox ability: 1. change the execute method of the class tool in lib/galaxy/tools/__init__.py to
def execute( self, trans, incoming={}, set_output_hid=True ): """ Execute the tool using parameter values in `incoming`. This just dispatches to the `ToolAction` instance specified by `self.tool_action`. In general this will create a `Job` that when run will build the tool's outputs, e.g. `DefaultToolAction`. """
############# #add code to verify if user has administrative privileges #added by sumedha ganjoo, sganjoo at uga.edu: line 1104 to 1109 - to refresh toolbox if self.id == 'REFRESH_ID': self.app.refreshToolBox() #############
return self.tool_action.execute( self, trans, incoming=incoming, set_output_hid=set_output_hid )
2. add refreshToolBox method (find below) to UniverseApplication class in lib/galaxy/app.py
########### # added by sumedha ganjoo,sganjoo at uga.edu, line 86-90 def refreshToolBox( self): self.toolbox = tools.ToolBox( self.config.tool_config, self.config.tool_path, self ) ###########
3. Add Refresh tool. Attached.
Thanks, Sumedha
<tool id="REFRESH_ID" name="Refresh" Version="1.0.0"> <description> :Refresh Galaxy</description> <command interpreter="python"> refreshTool.py $input $output </command> <inputs> <param name="input" type="select" display="radio" size="250" label="Execute this tool by clicking on Execute button"> <option value="refresh">Refresh Galaxy</option> </param>
</inputs> <outputs> <data format="tabular" name="output" /> </outputs> <tests> </tests> <help>
**Click on "Galaxy" on the top left corner of this window to refresh the page.**
</help> </tool>
________________________________________ Van: galaxy-dev-bounces@lists.bx.psu.edu [galaxy-dev-bounces@lists.bx.psu.edu] namens Peter Cock [p.j.a.cock@googlemail.com] Verzonden: maandag 30 mei 2011 21:09 Aan: Vossen, Bodo CC: galaxy-dev@lists.bx.psu.edu Onderwerp: Re: [galaxy-dev] Implemented adding tools via admin console On Mon, May 30, 2011 at 9:54 AM, Vossen, Bodo wrote:
Hi Peter,
thanks a lot for your comments.
Hi Bodo, No problem :)
I think it is more comfortable to be able to upload tools and xml via the webbrowser than to open a terminal and copy them, that was the idea behind my work. Especially for our institute this is very useful.
For me the command vs the web interface is not important. What I was excited about was adding a tool to Galaxy without having to restart Galaxy and interrupt any users.
You're right about the check_argument methods. I wasn't sure to add this kind of functionality in the first place, because it can become very tricky. What I did should just be the very first step for someone who wants to continue the work (if this makes sense). That's why I added a checkbox that disable this check, so that you still can upload your tool/xml even the argument test would fail.
It might be useful then.
Do you have some comments about the other parts ? Is this something you would be interested in using ?
I've not had a chance to try the code yet, but it could be useful. Peter ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
On Mon, May 30, 2011 at 10:42 PM, Bossers, Alex wrote:
Peter, if you were just interested in having a dynamic refresh.... it was posted long time ago and we have it operational. It comes as a tool in your tool list and refreshes all from the tool_conf.xml.
I'd like to see something like this functionality built into Galaxy, on the admin page next to reload a single tool. Having an extra standard tool (if world visible) is not an ideal solution. If that is what you meant? I'll have to try it and see.
Not the loc filtes though!
Reloading loc files is also on my wish list :)
As far as i remember it doesn't kill running jobs.
Good - because otherwise you might as well restart Galaxy.
It requires two lines of code in galaxy source and the addition of a tool to the tools section. I pasted it from my archive below. But it is quite self explanatory. Alex
Thanks, now I have two solutions to play with ;) Peter
Vossen, Bodo wrote:
Hi Peter,
thanks a lot for your comments. I think it is more comfortable to be able to upload tools and xml via the webbrowser than to open a terminal and copy them, that was the idea behind my work. Especially for our institute this is very useful.
You're right about the check_argument methods. I wasn't sure to add this kind of functionality in the first place, because it can become very tricky. What I did should just be the very first step for someone who wants to continue the work (if this makes sense). That's why I added a checkbox that disable this check, so that you still can upload your tool/xml even the argument test would fail.
Do you have some comments about the other parts ? Is this something you would be interested in using ?
Hi Bodo, My thoughts on the argument checking are similar to those of Peter's, although I like that you can essentially use these as a sort of sanity check warning. We will soon be working on simplifying tool management and when that work gets started we can have a look at your code to see how much overlap it has with the work we're doing. Thanks for your contribution! --nate
Best wishes,
Bodo Vossen
-----Original Message----- From: Peter Cock [mailto:p.j.a.cock@googlemail.com] Sent: Sat 5/28/2011 10:19 PM To: Vossen, Bodo Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Implemented adding tools via admin console
On Fri, May 27, 2011 at 11:21 AM, Vossen, Bodo <Bodo.Vossen@mpi-bn.mpg.de> wrote:
Hi all,
I've implemented functionality for adding tools via the admin menu.
That sounds very useful in principle, although perhaps the less flexible alternative of reloading tool_conf.xml would achieve the same aim?
... Then it is checked if script and XML are "compatible" by checking if they have the same number of arguments.
This seems very fragile, and having looked at the code for detecting the number of arguments in a Python script it looks like it will get it wrong in many situations. Even for something quite simple like this:
#sys.argv[0] is the script file itself assert len(sys,argv)==4, "Expect 3 arguments" arg1, arg2, arg3 = sys.argv[1:]
Also, what about conditional code like this?
assert len(sys.argv)==4, "Expect 3 arguments" if sys.argv[1] == "db": database = sys.argv[2] filename = None else: database = None filename = sys.argv[2] out_filename = sys.argv[3]
If I have read your get_parameter_number code right, it would claim there are four arguments since there are four lines of the basic form variable = sys.argv[i]
Also, it cannot possibly work when there are a varying number of arguments, for example a repeat argument. If you want a test case, my Venn Diagram tool on the Tool Shed should be relevant (its a python script).
If you have any comments or question please do not hesitate to contact me, I'm open for every kind of critics.
I would remove the attempt to check the number of arguments. It is practically impossible to get this right in all cases, so I don't think it is worth doing. Having valid script/xml combinations rejected would be quite annoying.
Peter
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
participants (4)
-
Bossers, Alex
-
Nate Coraor
-
Peter Cock
-
Vossen, Bodo