Currently I feel that a tool dependency definition should allow for only a singe download of some item, and if that item requires other items. they should be defined as separate dependencies that are installed using the prior_installation_required=True attribute. Is there a case where this is not possible? If so, I'm willing to enable downloading more than 1 item per dependency package definition, but I think we have to be careful in how we go about this. In any case, wget should not be used yet as it requires wget to be on the installation server, and we have not yet decided what 3rd party utilities we can assume exist in that environment. If there is justification for allowing download_by_url multiple times in the same package dependency definition, then I would rather enhance the tool shed framework to support that. Greg Von Kuster On May 9, 2013, at 12:15 PM, Peter Cock wrote:

On Thu, May 9, 2013 at 4:57 PM, Björn Grüning <bjoern.gruening@pharmazie.uni-freiburg.de> wrote:

...

Hi Peter,

Afaik "download_by_url" is only supported as first action. To work around that limitation, until its fixed, I used wget to fetch any additional file.

Ciao, Björn

Interesting tip - I don't see that on the wiki but it sounds believable. This would resolve by concerns about how this altered the current working directory since (assuming I understand correctly) after decompressing a tar ball, it changes into that new directory.

I'll try the wget alternative - thanks!

Peter

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

On Thu, May 9, 2013 at 6:21 PM, Björn Grüning <bjoern.gruening@pharmazie.uni-freiburg.de> wrote:

Hi Greg and Peter,

two examples from my side.

One of my tools (osra) has 7 dependencies. A few of them, GraphicsMagick for example, makes sense to include as separate repo. But I feel that the other are too special.

I made the same judgement call with clinod (command line NoD), which needs a single binary tool 'batchman' (either on the path or next to the clinod install) from a 3rd party package, the Stuttgart Neural Network Simulator (SNNS). http://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod/ I could have tried to work out how to download and install the whole of the SNNS package, but that in turn would trigger more dependencies. Instead I am just installing batchman by putting it in the clinod $INSTALL_DIR - this is far simpler and less likely to break. (If it turns out in six months time that some other Galaxy Tools also use bits of SNNS, then I'd reconsider this choice.) Peter

Björn, Peter, I've created a trello card (https://trello.com/c/QY0Z7L7p) for this requirement, and will be working on enhancing that feature as my next priority. --Dave B. On 5/10/13 09:59:37.000, Björn Grüning wrote:

Hi Peter,

...

...

For defining the 'install script' for some of my tool wrappers, I have mainly following http://wiki.galaxyproject.org/ToolShedToolFeatures and looking at one or two examples.

I've got some of my tools to work, but not all. For instance, this one appears not to be downloading or moving JAR files correctly: http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/5644c28cf965

...

Here my Python wrapper script has evidently read the $EFFECTIVET3 environment variable (defined in the tool_dependencies.xml to point to the tool's $INSTALL_DIR), but the JAR file isn't there:

Effective T3 JAR file not found: /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmpYR9FXD/effectiveT3/1.0.1/peterjc/effectivet3/5644c28cf965/TTSS_GUI-1.0.1.jar

My previous revision set the $EFFECTIVET3 environment variable at the end of the tool_dependencies.xml file, and from the test failure it appeared not to be getting set. My hunch is something breaks part way through the installation but the Tool Shed isn't reporting this - or am I just not looking in the right place? It would make sense to show any installation issues under the "Tool Dependencies" entry on the main tool page.

Related to this, I'd like to suggest another couple of assertion action types, used for checking if directories or files exist, which would abort the installation with a clear error if the check fails:

<action type="assert_dir_exists">$INSTALL_DIR/module/</action> <action type="assert_file_exists">$INSTALL_DIR/module/TTSS_STD-1.0.1.jar</action>

An obvious application of these would be immediately after an <action type="download_by_url"> line to sanity test the download (and decompression) worked as expected - and didn't for example just download an HTML error page for instance.

(In this case I am worried that perhaps the download_by_url action has been overly enthusiastic and unzipped the JAR file, perhaps based on the mime-type?).

Thanks,

Peter

Thanks for the input Björn and Greg about not being able to make multiple calls to download_by_url - I've tried tweaking both the effectivet3 and clinod tool_dependencies.xml files to use wget but they are still failing:

http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/c70df461aae5 http://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod/20e44218fa63

I'm not sure, but can you try to fetch the first jar file with download_by_url. The last time I was struggling with that the download_by_url was required, if I remember correctly.

...

I believe that either I have a simple error in both install descriptions, leading to the install to 'work' but not put things where I expect them (which is my problem), or the install is really failing yet there is no feedback about this (which is a Galaxy Tool Shed problem).

...

The idea of my new <action> types for checking if files and folders exist is to catch some types of install failure as early as possible - in this case verify the install put the relevant files where it was intended to.

Could you direct me at the source code which handles running the tool_dependencies.xm install scripts? I'd like to look at what kind of error handling it does (e.g. I'd hope non-zero return codes from a shell_command <action> would be treated as an error), and how easy it would be to add the new actions myself.

Have a look at:

lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py

Hope that helps! Björn

...

Thanks,

Peter

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

Peter, Yes, jar files are just a directory of .class (and other) files packaged with the zip algorithm, which has its benefits and drawbacks, as you've discovered. In the new action type I'm working on, I do intend to include the option to extract a downloaded file, or leave it as-is, which will allow the download of jar files needed at build time or run time, as well as any other compressed file that should not be automatically extracted. --Dave B. On 5/10/13 11:13:36.000, Peter Cock wrote:

On Fri, May 10, 2013 at 2:59 PM, Björn Grüning <bjoern.gruening@pharmazie.uni-freiburg.de> wrote:

...

Hi Peter,

...

I believe that either I have a simple error in both install descriptions, leading to the install to 'work' but not put things where I expect them (which is my problem), or the install is really failing yet there is no feedback about this (which is a Galaxy Tool Shed problem).

...

The idea of my new <action> types for checking if files and folders exist is to catch some types of install failure as early as possible - in this case verify the install put the relevant files where it was intended to.

Could you direct me at the source code which handles running the tool_dependencies.xm install scripts? I'd like to look at what kind of error handling it does (e.g. I'd hope non-zero return codes from a shell_command <action> would be treated as an error), and how easy it would be to add the new actions myself.

Have a look at:

lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py

Hope that helps! Björn

That is educational, thanks Björn :)

The good news is that the shell_command handling code does check the return code and record an error if it failed as part of the tool_dependency object (status and error_message).

The bad news is that in general install_and_build_package doesn't seem to catch any other errors - for instance a failing download, corrupt zip file, invalid path name, etc.

Further bad news, it looks like by guess about the JAR files wrongly being unzipped was accurate - digging down the imports, the common_util.iszip function seems to just be doing this:

...

...

...

import zipfile zipfile.is_zipfile("/opt/clinod/clinod-1.3.jar") True

Therefore (right now) as I feared, we cannot use the download_by_url action with JAR files because it will unzip them. As suggested earlier, one fix would be an option for controlling any decompression (defaulting to automatic as now, but allowing it to be explicitly set to true or false?).

Peter

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

Hi Dave, that was fast. Thanks a lot! Björn

<?xml version="1.0"?> <tool_dependency> <package name="effectiveT3" version="1.0.1"> <install version="1.0"> <actions> <!-- Download jar files required by this tool. --> <action type="download_file">http://effectors.org/download/version/TTSS_GUI-1.0.1.jar</action> <action type="move_file"><source>TTSS_GUI-1.0.1.jar</source><destination>$INSTALL_DIR/</destination></action> <action type="make_directory">$INSTALL_DIR/module</action> <action type="download_file">http://effectors.org/download/module/TTSS_ANIMAL-1.0.1.jar</action> <action type="download_file">http://effectors.org/download/module/TTSS_PLANT-1.0.1.jar</action> <action type="download_file">http://effectors.org/download/module/TTSS_STD-1.0.1.jar</action> <action type="move_file"><source>TTSS_PLANT-1.0.1.jar</source><destination>$INSTALL_DIR/module/</destination></action> <action type="move_file"><source>TTSS_STD-1.0.1.jar</source><destination>$INSTALL_DIR/module/</destination></action> <action type="move_file"><source>TTSS_ANIMAL-1.0.1.jar</source><destination>$INSTALL_DIR/module/</destination></action> <!-- Set environment variable so Python script knows where to look --> <action type="set_environment"> <environment_variable name="EFFECTIVET3" action="set_to">$INSTALL_DIR</environment_variable> </action> </actions> </install> <readme> Downloads effectiveT3 v1.0.1 and the three models from http://effectors.org/ </readme> </package> </tool_dependency>

An enhancement I intend to make on Monday is the option to specify a target path and filename, which will eliminate the need for the move_file steps you see in the above XML.

As this is a new feature, if you discover any bugs or anomalous behavior, please feel free to let me know, and I'll get that resolved as soon as I can.

--Dave B.

On 5/10/13 11:13:36.000, Peter Cock wrote:

...

On Fri, May 10, 2013 at 2:59 PM, Björn Grüning <bjoern.gruening@pharmazie.uni-freiburg.de> wrote:

...

Hi Peter,

...

I believe that either I have a simple error in both install descriptions, leading to the install to 'work' but not put things where I expect them (which is my problem), or the install is really failing yet there is no feedback about this (which is a Galaxy Tool Shed problem).

...

The idea of my new <action> types for checking if files and folders exist is to catch some types of install failure as early as possible - in this case verify the install put the relevant files where it was intended to.

Could you direct me at the source code which handles running the tool_dependencies.xm install scripts? I'd like to look at what kind of error handling it does (e.g. I'd hope non-zero return codes from a shell_command <action> would be treated as an error), and how easy it would be to add the new actions myself.

Have a look at:

lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py

Hope that helps! Björn

That is educational, thanks Björn :)

The good news is that the shell_command handling code does check the return code and record an error if it failed as part of the tool_dependency object (status and error_message).

The bad news is that in general install_and_build_package doesn't seem to catch any other errors - for instance a failing download, corrupt zip file, invalid path name, etc.

Further bad news, it looks like by guess about the JAR files wrongly being unzipped was accurate - digging down the imports, the common_util.iszip function seems to just be doing this:

...

...

...

import zipfile zipfile.is_zipfile("/opt/clinod/clinod-1.3.jar") True

Therefore (right now) as I feared, we cannot use the download_by_url action with JAR files because it will unzip them. As suggested earlier, one fix would be an option for controlling any decompression (defaulting to automatic as now, but allowing it to be explicitly set to true or false?).

Peter

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

On Mon, May 13, 2013 at 10:34 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:

On Fri, May 10, 2013 at 8:11 PM, Björn Grüning <bjoern.gruening@pharmazie.uni-freiburg.de> wrote:

...

Hi Dave,

that was fast.

Thanks a lot! Björn

Thanks Dave,

I see this also should fix handling of JAR files in download_by_url :) https://bitbucket.org/galaxy/galaxy-central/commits/b4733e42a2c98c42401bca5e...

Could I suggest an enhancement to abort on undefined action types, rough patch below (untested)? This will help give a clear error if a new command is used but the Galaxy host is too old to implement it (which is exactly the situation we'd see right now on the main Tool Shed if the new download_file action is used).

Regards,

Peter

Another data point, with my old wget based tool_wrapper.xml file: http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/b863c4bec840

From the test failure of the previous commit I suspected this, but this revision checks specifically for the 'module' subdirectory and confirms it is not being created:

Effective T3 module folder not found: '/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmpAzkY3r/effectiveT3/1.0.1/peterjc/effectivet3/b863c4bec840/module' I infer the 'install script' is failing silently after setting the environment variable but before or when creating the sub-directory $INSTALL_DIR/module, <actions> <!-- Set environment variable so Python script knows where to look --> <action type="set_environment"> <environment_variable name="EFFECTIVET3" action="set_to">$INSTALL_DIR</environment_variable> </action> <!-- Main JAR file --> <action type="shell_command">wget http://effectors.org/download/version/TTSS_GUI-1.0.1.jar</action> <action type="move_file"><source>TTSS_GUI-1.0.1.jar</source><destination>$INSTALL_DIR/</destination></action> <!-- Three model JAR files --> <action type="make_directory">$INSTALL_DIR/module</action> <action type="shell_command">wget http://effectors.org/download/module/TTSS_ANIMAL-1.0.1.jar</action> <action type="move_file"><source>TTSS_ANIMAL-1.0.1.jar</source><destination>$INSTALL_DIR/module/</destination></action> <action type="shell_command">wget http://effectors.org/download/module/TTSS_PLANT-1.0.1.jar</action> <action type="move_file"><source>TTSS_PLANT-1.0.1.jar</source><destination>$INSTALL_DIR/module/</destination></action> <action type="shell_command">wget http://effectors.org/download/module/TTSS_STD-1.0.1.jar</action> <action type="move_file"><source>TTSS_STD-1.0.1.jar</source><destination>$INSTALL_DIR/module/</destination></action> </actions> The first possible cause of failure is the set_environment action, but from the limited information from the failing test that worked. The second possible causes of failure is the wget call, perhaps that command line tool isn't even installed? However, from the limited information reported via the failing test I know the JAR file was downloaded (and moved into place). The third possible failure is the move_file action, and again I know from the limited information reported via the failing test that the TTSS_GUI-1.0.1.jar file was moved into place. My suspicion therefore falls on the make_directory action - my debug output from the failing test indicates the expected directory was not created. However, any exception within fabric_util.install_and_build_package should be caught by install_util.install_and_build_package_via_fabric and recorded in the tool_dependency object. Tracing up the call tree, I found this puzzling bit in the common_install_util.py file: https://bitbucket.org/galaxy/galaxy-central/src/65a81aead95e147f709ee3969d49... def handle_tool_dependencies( app, tool_shed_repository, tool_dependencies_config, tool_dependencies ): ... if tool_dependency.can_install: try: tool_dependency = install_package( app, elem, tool_shed_repository, tool_dependencies=tool_dependencies ) except Exception, e: error_message = "Error installing tool dependency %s version %s: %s" % ( str( package_name ), str( package_version ), str( e ) ) log.debug( error_message ) if tool_dependency: tool_dependency.status = app.model.ToolDependency.installation_status.ERROR tool_dependency.error_message = error_message sa_session.add( tool_dependency ) sa_session.flush() if tool_dependency and tool_dependency.status in [ app.model.ToolDependency.installation_status.INSTALLED, app.model.ToolDependency.installation_status.ERROR ]: installed_tool_dependencies.append( tool_dependency ) It seems even if install_package fails, the tool_dependency is still considered to be installed? If so, why? This would explain the behaviour I am seeing with effectivet3 and other tools on the Tool Shed - despite the tool_dependencies.xml seeming to fail part way though, the tests are run anyway. (I would instead expect the Tool Shed to tell me the install failed, and not bother trying to run the tests, which are probably going to fail anyway). Regards, Peter

On Mon, May 13, 2013 at 8:35 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:

On Mon, May 13, 2013 at 11:42 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:

...

...

This would explain the behaviour I am seeing with effectivet3 and other tools on the Tool Shed - despite the tool_dependencies.xml seeming to fail part way though, the tests are run anyway.

(I would instead expect the Tool Shed to tell me the install failed, and not bother trying to run the tests, which are probably going to fail anyway).

It looks like this is coming soon - including displaying something about the install failure reasons if I'm reading this correctly :) https://bitbucket.org/galaxy/galaxy-central/commits/ef2b705c7de3

I'm seeing the new install failures section on some tools now :) (I've just updated the tool concerned, so hopefully that install error will go away). Thanks, Peter