Tool Shed upload/migration revision checking bug?
Hi Bjoern & Greg, The Biopython package looks OK on the test tool shed, http://testtoolshed.g2.bx.psu.edu/view/biopython/package_biopython_1_61 This includes the lines: <package name="numpy" version="1.7.1"> <repository changeset_revision="ad6ebe2c75ef" name="package_numpy_1_7" owner="bgruening" prior_installation_required="True" toolshed="http://testtoolshed.g2.bx.psu.edu" /> </package> <package name="matplotlib" version="1.2.1"> <repository changeset_revision="c888aa8ed318" name="package_matplotlib_1_2" owner="bgruening" prior_installation_required="True" toolshed="http://testtoolshed.g2.bx.psu.edu" /> </package> Note that we plan to update those dependencies to point at the IUC account, not Bjoern's personal account. However, on the main Tool Shed something is broken with the dependencies lacking relevant revisions: http://toolshed.g2.bx.psu.edu/view/biopython/package_biopython_1_61 <package name="numpy" version="1.7.1"> <repository name="package_numpy_1_7" owner="bgruening" prior_installation_required="True" toolshed="http://toolshed.g2.bx.psu.edu" /> </package> <package name="matplotlib" version="1.2.1"> <repository name="package_matplotlib_1_2" owner="bgruening" prior_installation_required="True" toolshed="http://toolshed.g2.bx.psu.edu" /> </package> My guess is the revisions failed because there is no such repository on the main Tool Shed (they do exist under the new IUC user through, while on the Test Tool Shed they exist under both usernames for now). Bjoern, do you remember if this was a manual upload to the Main Tool Shed, or done via the migration script? Greg, if this was uploaded manually, the reference to non-existent repositories should have shown up as an error. If this was migrated, was there no error message? Bjoern, are you happy to update both the test & main repositories to point at the IUC packages now? Or should I do that myself? Thanks, Peter
Hi,
Hi Bjoern & Greg,
The Biopython package looks OK on the test tool shed, http://testtoolshed.g2.bx.psu.edu/view/biopython/package_biopython_1_61
This includes the lines:
<package name="numpy" version="1.7.1"> <repository changeset_revision="ad6ebe2c75ef" name="package_numpy_1_7" owner="bgruening" prior_installation_required="True" toolshed="http://testtoolshed.g2.bx.psu.edu" /> </package> <package name="matplotlib" version="1.2.1"> <repository changeset_revision="c888aa8ed318" name="package_matplotlib_1_2" owner="bgruening" prior_installation_required="True" toolshed="http://testtoolshed.g2.bx.psu.edu" /> </package>
Note that we plan to update those dependencies to point at the IUC account, not Bjoern's personal account.
However, on the main Tool Shed something is broken with the dependencies lacking relevant revisions: http://toolshed.g2.bx.psu.edu/view/biopython/package_biopython_1_61
<package name="numpy" version="1.7.1"> <repository name="package_numpy_1_7" owner="bgruening" prior_installation_required="True" toolshed="http://toolshed.g2.bx.psu.edu" /> </package> <package name="matplotlib" version="1.2.1"> <repository name="package_matplotlib_1_2" owner="bgruening" prior_installation_required="True" toolshed="http://toolshed.g2.bx.psu.edu" /> </package>
Upps ... :(
My guess is the revisions failed because there is no such repository on the main Tool Shed (they do exist under the new IUC user through, while on the Test Tool Shed they exist under both usernames for now).
Bjoern, do you remember if this was a manual upload to the Main Tool Shed, or done via the migration script?
It was done manually.
Greg, if this was uploaded manually, the reference to non-existent repositories should have shown up as an error. If this was migrated, was there no error message?
Bjoern, are you happy to update both the test & main repositories to point at the IUC packages now? Or should I do that myself?
Done! Hope it works now. Thanks for pointing that out! Bjoern
Thanks,
Peter
On Mon, Aug 12, 2013 at 12:27 PM, Bjoern Gruening <bjoern.gruening@gmail.com> wrote:
Hi,
Hi Bjoern & Greg,
The Biopython package looks OK on the test tool shed, http://testtoolshed.g2.bx.psu.edu/view/biopython/package_biopython_1_61
This includes the lines:
<package name="numpy" version="1.7.1"> <repository changeset_revision="ad6ebe2c75ef" name="package_numpy_1_7" owner="bgruening" prior_installation_required="True" toolshed="http://testtoolshed.g2.bx.psu.edu" /> </package> <package name="matplotlib" version="1.2.1"> <repository changeset_revision="c888aa8ed318" name="package_matplotlib_1_2" owner="bgruening" prior_installation_required="True" toolshed="http://testtoolshed.g2.bx.psu.edu" /> </package>
Note that we plan to update those dependencies to point at the IUC account, not Bjoern's personal account.
However, on the main Tool Shed something is broken with the dependencies lacking relevant revisions: http://toolshed.g2.bx.psu.edu/view/biopython/package_biopython_1_61
<package name="numpy" version="1.7.1"> <repository name="package_numpy_1_7" owner="bgruening" prior_installation_required="True" toolshed="http://toolshed.g2.bx.psu.edu" /> </package> <package name="matplotlib" version="1.2.1"> <repository name="package_matplotlib_1_2" owner="bgruening" prior_installation_required="True" toolshed="http://toolshed.g2.bx.psu.edu" /> </package>
Upps ... :(
My guess is the revisions failed because there is no such repository on the main Tool Shed (they do exist under the new IUC user through, while on the Test Tool Shed they exist under both usernames for now).
Bjoern, do you remember if this was a manual upload to the Main Tool Shed, or done via the migration script?
It was done manually.
Greg, if this was uploaded manually, the reference to non-existent repositories should have shown up as an error. If this was migrated, was there no error message?
Bjoern, are you happy to update both the test & main repositories to point at the IUC packages now? Or should I do that myself?
Done! Hope it works now.
Thanks for pointing that out! Bjoern
Great - that seems to have solve the Biopython issue, although I'm curious if there is a glitch in the Tool Shed failing to issue a warning in a case like this? Thanks, Peter
On Mon, 2013-08-12 at 12:41 +0100, Peter Cock wrote:
On Mon, Aug 12, 2013 at 12:27 PM, Bjoern Gruening <bjoern.gruening@gmail.com> wrote:
Hi,
Hi Bjoern & Greg,
The Biopython package looks OK on the test tool shed, http://testtoolshed.g2.bx.psu.edu/view/biopython/package_biopython_1_61
This includes the lines:
<package name="numpy" version="1.7.1"> <repository changeset_revision="ad6ebe2c75ef" name="package_numpy_1_7" owner="bgruening" prior_installation_required="True" toolshed="http://testtoolshed.g2.bx.psu.edu" /> </package> <package name="matplotlib" version="1.2.1"> <repository changeset_revision="c888aa8ed318" name="package_matplotlib_1_2" owner="bgruening" prior_installation_required="True" toolshed="http://testtoolshed.g2.bx.psu.edu" /> </package>
Note that we plan to update those dependencies to point at the IUC account, not Bjoern's personal account.
However, on the main Tool Shed something is broken with the dependencies lacking relevant revisions: http://toolshed.g2.bx.psu.edu/view/biopython/package_biopython_1_61
<package name="numpy" version="1.7.1"> <repository name="package_numpy_1_7" owner="bgruening" prior_installation_required="True" toolshed="http://toolshed.g2.bx.psu.edu" /> </package> <package name="matplotlib" version="1.2.1"> <repository name="package_matplotlib_1_2" owner="bgruening" prior_installation_required="True" toolshed="http://toolshed.g2.bx.psu.edu" /> </package>
Upps ... :(
My guess is the revisions failed because there is no such repository on the main Tool Shed (they do exist under the new IUC user through, while on the Test Tool Shed they exist under both usernames for now).
Bjoern, do you remember if this was a manual upload to the Main Tool Shed, or done via the migration script?
It was done manually.
Greg, if this was uploaded manually, the reference to non-existent repositories should have shown up as an error. If this was migrated, was there no error message?
Bjoern, are you happy to update both the test & main repositories to point at the IUC packages now? Or should I do that myself?
Done! Hope it works now.
Thanks for pointing that out! Bjoern
Great - that seems to have solve the Biopython issue, although I'm curious if there is a glitch in the Tool Shed failing to issue a warning in a case like this?
Yes, a waning would be really great. I totally missed it :(
Thanks,
Peter
Hello Peter and Bjoern, I believe this issue was handled in the August Galaxy release, but the main tool shed had not yet been upgraded to that release when you were seeing this behavior. Unless I misunderstand the steps necessary to reproduce the behavior, I am not able to do so. Here is what I've tried in my local tool shed: 1) Create a repopsitory named package_atlas_3_10 owned by username "test" with repository type "tool_dependency_definition". 2 Populate the package_atlas_3_10 repository with a tool_dependencies.xml file that looks like this: <tool_dependency> <package name="atlas" version="3.10.1"> <install version="1.0"> <actions> <!-- first action is always downloading --> <action type="download_by_url">file:///var/chemicaltoolbox/atlas.tar</action> <action type="move_directory_files"> <source_directory>.</source_directory> <destination_directory>$INSTALL_DIR</destination_directory> </action> <action type="set_environment"> <environment_variable name="ATLAS_LIB_DIR" action="set_to">$INSTALL_DIR/lib</environment_variable> <environment_variable name="ATLAS_INCLUDE_DIR" action="set_to">$INSTALL_DIR/include</environment_variable> </action> </actions> </install> <readme>ATLAS_LIB_DIR and ATLAS_INCLUDE_DIR will be set (including libatlas.a). During ATLAS library compilation, ATLAS performs code efficiency checks. These checks can only provide optimal results, if "frequency scaling" is disabled on the CPU, and no other load-intense processes are running. Ideally, you should compile on an empty cluster node with CPU frequency scaling disabled (see "cpufreq-selector" or "cpufreq-set"). </readme> </package> </tool_dependency> 3) Create a repository named package_confab_1_0_1owned by "test" with repository type "tool_dependency_definition". 4) Attempt to upload a tool_dependencies.xml file to package_confab_1_0_1that looks like this - notice that the owner "hello" is invalid since the required package_atlas_3_10 repository is actually owned by "test". <tool_dependency> <package name="atlas" version="3.10.1"> <repository name="package_atlas_3_10" owner="hello" prior_installation_required="True" /> </package> <package name="confab" version="1.0.1"> <install version="1.0"> <actions> <action type="download_by_url">http://confab.googlecode.com/files/Confab-1.0.1.tar.gz</action> <!-- populate the environment variables from the dependent repos --> <action type="set_environment_for_install"> <repository name="package_atlas_3_10" owner="hello"> <package name="atlas" version="3.10.1" /> </repository> </action> <action type="shell_command">cmake . -DCMAKE_INSTALL_PREFIX=$INSTALL_DIR -DEIGEN2_INCLUDE_DIR=$EIGEN_SOURCE_PATH/</action> <action type="shell_command">make</action> <action type="shell_command">make install</action> <action type="set_environment"> <environment_variable name="PATH" action="prepend_to">$INSTALL_DIR/bin</environment_variable> </action> </actions> </install> <readme>Compiling Confab requires g++, CMake 2.4+ and Eigen version 2 (libeigen2-dev). Optional but required for a few features is libxml2 and zlib.</readme> </package> </tool_dependency> The above upload raises a server error, and the following stack trace is displayed in the borwser. Internal Server Error Galaxy was unable to sucessfully complete your request URL: http://localhost:9009/upload/upload?repository_id=cd0d8ada19d98f27 Module galaxy.web.framework.middleware.error:149 in __call__
app_iter = self.application(environ, sr_checker) Module paste.debug.prints:106 in __call__ environ, self.app) Module paste.wsgilib:543 in intercept_output app_iter = application(environ, replacement_start_response) Module paste.recursive:84 in __call__ return self.application(environ, start_response) Module paste.httpexceptions:633 in __call__ return self.application(environ, start_response) Module galaxy.web.framework.base:132 in __call__ return self.handle_request( environ, start_response ) Module galaxy.web.framework.base:190 in handle_request body = method( trans, **kwargs ) Module galaxy.web.framework:98 in decorator return func( self, trans, *args, **kwargs ) Module galaxy.webapps.tool_shed.controllers.upload:147 in upload altered, root_elem = commit_util.handle_tool_dependencies_definition( trans, uploaded_file_name ) Module tool_shed.util.commit_util:321 in handle_tool_dependencies_definition raise Exception( exception_message ) Exception: The tool_dependencies.xml file contains an invalid <repository> tag. Unable to locate repository with name package_atlas_3_10 and owner hello.
The raised exception did not allow the upload to succeed, and the "package_confab_1_0_1" repository remains empty with no changeset revisions. Revision: -1:000000000000 Please let me know if I've misunderstood the steps you used to produce the behavior you saw. Thanks very much, Greg Von Kuster On Aug 12, 2013, at 7:44 AM, Bjoern Gruening <bjoern.gruening@gmail.com> wrote:
On Mon, 2013-08-12 at 12:41 +0100, Peter Cock wrote:
On Mon, Aug 12, 2013 at 12:27 PM, Bjoern Gruening <bjoern.gruening@gmail.com> wrote:
Hi,
Hi Bjoern & Greg,
The Biopython package looks OK on the test tool shed, http://testtoolshed.g2.bx.psu.edu/view/biopython/package_biopython_1_61
This includes the lines:
<package name="numpy" version="1.7.1"> <repository changeset_revision="ad6ebe2c75ef" name="package_numpy_1_7" owner="bgruening" prior_installation_required="True" toolshed="http://testtoolshed.g2.bx.psu.edu" /> </package> <package name="matplotlib" version="1.2.1"> <repository changeset_revision="c888aa8ed318" name="package_matplotlib_1_2" owner="bgruening" prior_installation_required="True" toolshed="http://testtoolshed.g2.bx.psu.edu" /> </package>
Note that we plan to update those dependencies to point at the IUC account, not Bjoern's personal account.
However, on the main Tool Shed something is broken with the dependencies lacking relevant revisions: http://toolshed.g2.bx.psu.edu/view/biopython/package_biopython_1_61
<package name="numpy" version="1.7.1"> <repository name="package_numpy_1_7" owner="bgruening" prior_installation_required="True" toolshed="http://toolshed.g2.bx.psu.edu" /> </package> <package name="matplotlib" version="1.2.1"> <repository name="package_matplotlib_1_2" owner="bgruening" prior_installation_required="True" toolshed="http://toolshed.g2.bx.psu.edu" /> </package>
Upps ... :(
My guess is the revisions failed because there is no such repository on the main Tool Shed (they do exist under the new IUC user through, while on the Test Tool Shed they exist under both usernames for now).
Bjoern, do you remember if this was a manual upload to the Main Tool Shed, or done via the migration script?
It was done manually.
Greg, if this was uploaded manually, the reference to non-existent repositories should have shown up as an error. If this was migrated, was there no error message?
Bjoern, are you happy to update both the test & main repositories to point at the IUC packages now? Or should I do that myself?
Done! Hope it works now.
Thanks for pointing that out! Bjoern
Great - that seems to have solve the Biopython issue, although I'm curious if there is a glitch in the Tool Shed failing to issue a warning in a case like this?
Yes, a waning would be really great. I totally missed it :(
Thanks,
Peter
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Morning Greg, seems to be also fixed in latest release. Will reopen the thread if I encounter it again. Thanks a lot, Bjoern
Hello Peter and Bjoern,
I believe this issue was handled in the August Galaxy release, but the main tool shed had not yet been upgraded to that release when you were seeing this behavior. Unless I misunderstand the steps necessary to reproduce the behavior, I am not able to do so.
Here is what I've tried in my local tool shed:
1) Create a repopsitory named package_atlas_3_10 owned by username "test" with repository type "tool_dependency_definition".
2 Populate the package_atlas_3_10 repository with a tool_dependencies.xml file that looks like this:
<tool_dependency> <package name="atlas" version="3.10.1"> <install version="1.0"> <actions> <!-- first action is always downloading --> <action type="download_by_url">file:///var/chemicaltoolbox/atlas.tar</action>
<action type="move_directory_files"> <source_directory>.</source_directory> <destination_directory>$INSTALL_DIR</destination_directory> </action>
<action type="set_environment"> <environment_variable name="ATLAS_LIB_DIR" action="set_to">$INSTALL_DIR/lib</environment_variable> <environment_variable name="ATLAS_INCLUDE_DIR" action="set_to">$INSTALL_DIR/include</environment_variable> </action> </actions> </install> <readme>ATLAS_LIB_DIR and ATLAS_INCLUDE_DIR will be set (including libatlas.a). During ATLAS library compilation, ATLAS performs code efficiency checks. These checks can only provide optimal results, if "frequency scaling" is disabled on the CPU, and no other load-intense processes are running. Ideally, you should compile on an empty cluster node with CPU frequency scaling disabled (see "cpufreq-selector" or "cpufreq-set"). </readme> </package> </tool_dependency>
3) Create a repository named package_confab_1_0_1owned by "test" with repository type "tool_dependency_definition".
4) Attempt to upload a tool_dependencies.xml file to package_confab_1_0_1that looks like this - notice that the owner "hello" is invalid since the required package_atlas_3_10 repository is actually owned by "test".
<tool_dependency> <package name="atlas" version="3.10.1"> <repository name="package_atlas_3_10" owner="hello" prior_installation_required="True" /> </package> <package name="confab" version="1.0.1"> <install version="1.0"> <actions> <action type="download_by_url">http://confab.googlecode.com/files/Confab-1.0.1.tar.gz</action>
<!-- populate the environment variables from the dependent repos --> <action type="set_environment_for_install"> <repository name="package_atlas_3_10" owner="hello"> <package name="atlas" version="3.10.1" /> </repository> </action>
<action type="shell_command">cmake . -DCMAKE_INSTALL_PREFIX=$INSTALL_DIR -DEIGEN2_INCLUDE_DIR= $EIGEN_SOURCE_PATH/</action> <action type="shell_command">make</action> <action type="shell_command">make install</action> <action type="set_environment"> <environment_variable name="PATH" action="prepend_to">$INSTALL_DIR/bin</environment_variable> </action> </actions> </install> <readme>Compiling Confab requires g++, CMake 2.4+ and Eigen version 2 (libeigen2-dev). Optional but required for a few features is libxml2 and zlib.</readme> </package> </tool_dependency>
app_iter = self.application(environ, sr_checker) Module paste.debug.prints:106 in __call__ environ, self.app) Module paste.wsgilib:543 in intercept_output app_iter = application(environ, replacement_start_response) Module paste.recursive:84 in __call__ return self.application(environ, start_response) Module paste.httpexceptions:633 in __call__ return self.application(environ, start_response) Module galaxy.web.framework.base:132 in __call__ return self.handle_request( environ, start_response ) Module galaxy.web.framework.base:190 in handle_request body = method( trans, **kwargs ) Module galaxy.web.framework:98 in decorator return func( self, trans, *args, **kwargs ) Module galaxy.webapps.tool_shed.controllers.upload:147 in upload altered, root_elem = commit_util.handle_tool_dependencies_definition( trans, uploaded_file_name ) Module tool_shed.util.commit_util:321 in handle_tool_dependencies_definition raise Exception( exception_message ) Exception: The tool_dependencies.xml file contains an invalid <repository> tag. Unable to locate repository with name
The above upload raises a server error, and the following stack trace is displayed in the borwser. Internal Server Error Galaxy was unable to sucessfully complete your request URL: http://localhost:9009/upload/upload?repository_id=cd0d8ada19d98f27 Module galaxy.web.framework.middleware.error:149 in __call__ package_atlas_3_10 and owner hello.
The raised exception did not allow the upload to succeed, and the "package_confab_1_0_1" repository remains empty with no changeset revisions.
Revision: -1:000000000000
Please let me know if I've misunderstood the steps you used to produce the behavior you saw.
Thanks very much,
Greg Von Kuster
On Aug 12, 2013, at 7:44 AM, Bjoern Gruening <bjoern.gruening@gmail.com> wrote:
On Mon, 2013-08-12 at 12:41 +0100, Peter Cock wrote:
On Mon, Aug 12, 2013 at 12:27 PM, Bjoern Gruening <bjoern.gruening@gmail.com> wrote:
Hi,
Hi Bjoern & Greg,
The Biopython package looks OK on the test tool shed, http://testtoolshed.g2.bx.psu.edu/view/biopython/package_biopython_1_61
This includes the lines:
<package name="numpy" version="1.7.1"> <repository changeset_revision="ad6ebe2c75ef" name="package_numpy_1_7" owner="bgruening" prior_installation_required="True" toolshed="http://testtoolshed.g2.bx.psu.edu" /> </package> <package name="matplotlib" version="1.2.1"> <repository changeset_revision="c888aa8ed318" name="package_matplotlib_1_2" owner="bgruening" prior_installation_required="True" toolshed="http://testtoolshed.g2.bx.psu.edu" /> </package>
Note that we plan to update those dependencies to point at the IUC account, not Bjoern's personal account.
However, on the main Tool Shed something is broken with the dependencies lacking relevant revisions: http://toolshed.g2.bx.psu.edu/view/biopython/package_biopython_1_61
<package name="numpy" version="1.7.1"> <repository name="package_numpy_1_7" owner="bgruening" prior_installation_required="True" toolshed="http://toolshed.g2.bx.psu.edu" /> </package> <package name="matplotlib" version="1.2.1"> <repository name="package_matplotlib_1_2" owner="bgruening" prior_installation_required="True" toolshed="http://toolshed.g2.bx.psu.edu" /> </package>
Upps ... :(
My guess is the revisions failed because there is no such repository on the main Tool Shed (they do exist under the new IUC user through, while on the Test Tool Shed they exist under both usernames for now).
Bjoern, do you remember if this was a manual upload to the Main Tool Shed, or done via the migration script?
It was done manually.
Greg, if this was uploaded manually, the reference to non-existent repositories should have shown up as an error. If this was migrated, was there no error message?
Bjoern, are you happy to update both the test & main repositories to point at the IUC packages now? Or should I do that myself?
Done! Hope it works now.
Thanks for pointing that out! Bjoern
Great - that seems to have solve the Biopython issue, although I'm curious if there is a glitch in the Tool Shed failing to issue a warning in a case like this?
Yes, a waning would be really great. I totally missed it :(
Thanks,
Peter
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
participants (3)
-
Bjoern Gruening -
Greg Von Kuster -
Peter Cock