Re: [galaxy-dev] environment variables and paths for toolshed tools
Greg, Pablo, I'm using a SnpEffect repository 'snpeff_with_dep' in http://testtoolshed.g2.bx.psu.edu/ to test using tool_dependencies.xml with the <set_environment> tag. ( You both write access, if you want to use this for correcting my errors or debugging. ) I'm getting an error in prepare_for_install: Modulegalaxy.web.controllers.admin_toolshed:*1174*in|prepare_for_install| |>>>|||tool_dependencies|| |{'set_environment': [{'type': 'environment variable', 'name': 'JAVA_JAR_PATH'}], 'snpEff/3.0f': {'readme': '\n ', 'version': '3.0f', 'type': 'package', 'name': 'snpEff'}}| ... Module_admin_tool_shed_repository_common_mako:*242*in|render_render_tool_dependency_section| |>> <http://localhost:18080/admin_toolshed/prepare_for_install?tool_shed_url=http://testtoolshed.g2.bx.psu.edu/&repository_ids=d01ace51ab1282a5&changeset_revisions=38b3a6b97bd1#>name*=*requirements_dict*[*'name'*]*| *TypeError: list indices must be integers, not str* It appears that the requirements_dict for the set_environment tag is enclosed in an array. I haven't yet looked at the repository code to see how that might happen. Pablo, The http://sourceforge.net/projects/snpeff/files/ repository should probably have versions packaged for download named like: http://sourceforge.net/projects/snpeff/files/snpEff_3_0.tgz $ tar ztvf snpEff_3_0.tgz drwxrwxrwx 0 0 0 0 Aug 23 12:17 snpEff_3_0/ -rwxrwxrwx 0 0 0 14298767 Aug 23 12:17 snpEff_3_0/SnpSift.jar ... That will allow versioning within galaxy, and successive versions of tool_dependencies.xml with only need a change to the action tag: download_by_url Thanks, JJ I'm trying to test a repository that use in tool_depen
Message: 18 Date: Thu, 13 Sep 2012 11:46:32 -0400 From: Greg Von Kuster<greg@bx.psu.edu> To: David Hoover<hooverdm@helix.nih.gov> Cc:galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] environment variables and paths for toolshed tools Message-ID:<F75D6320-3AF8-4BB0-8F31-B0F44AA1306B@bx.psu.edu> Content-Type: text/plain; charset="us-ascii"
Hello David,
I've just committed change set 7656:6aadac8026cb to the Galaxy central repository that provides the ability to do what you need here. It would be great if you could try things out and let me know if you run into any problems.
I've been working with nikhil-joshi's deseq_and_sam2counts repository in the main Galaxy tool shed because his tools require this new feature as well. This example should provide you with the information you'll need to tweak your tool shed repository so that your tool dependencies are located where they get installed rather than attempting to move them to ${GALAXY_DATA_INDEX_DIR} or some other location.
Here is the tool_dependency.xml file entry for locating a directory referred to by an environment variable name R_SCRIPT_PATH.
<?xml version="1.0"?> <tool_dependency> <set_environment version="1.0"> <environment_variable name="R_SCRIPT_PATH" action="set_to">$REPOSITORY_INSTALL_DIR</environment_variable> </set_environment> </tool_dependency>
The <set_enviroment> tag, is still supported inside <package> tag sets, but when defined at the xml root level, it will locate dependencies included in the installed tool shed repository. Of course, the above tool_dependencies.xml file could of course also include entries for tool dependencies that are packages, For example:
<?xml version="1.0"?> <tool_dependency> <set_environment version="1.0"> <environment_variable name="R_SCRIPT_PATH" action="set_to">$REPOSITORY_INSTALL_DIR</environment_variable> </set_environment> <package name="R" version="2.15.1"> <install version="1.0"> <actions> <action type="download_by_url">http://CRAN.R-project.org/src/base/R-2/R-2.15.1.tar.gz</action> <action type="shell_command">./configure --prefix=$INSTALL_DIR</action> <action type="shell_command">make</action> <action type="set_environment"> <environment_variable name="PATH" action="prepend_to">$INSTALL_DIR/bin</environment_variable> </action> </actions> </install> <readme> You need a FORTRAN compiler or perhaps f2c in addition to a C compiler to build R. </readme> </package> </tool_dependency>
These tool dependency definitions are handled as described in the following section of the tool shed wiki:
http://wiki.g2.bx.psu.edu/Tool%20Shed#Automatic_third-party_tool_dependency_...
So in order for the dependencies to be handled when the repository is installed, they must be defined in at least one of the <requirement> tag sets in at least 1 tool config in the repository. So, the <requirements> tag set in the deseq.xml tool config file will look something like this (there are more requirement tags defined here than in the tool_dependencies.xml fiel above, but you should get the idea).
<requirements> <requirement type="set_environment">R_SCRIPT_PATH</requirement> <requirement type="package" version="2.15.1">R</requirement> <requirement type="package" version="2.10">Bioconductor</requirement> <requirement type="package" version="1.8.3">DESeq</requirement> <requirement type="package" version="1.24.0">aroma.light</requirement> <requirement type="package" version="0.20-6">lattice</requirement> </requirements>
The <command> tag set in the deseq.xml tool config is also slightly altered. Here is the way it looks before using this new approach:
<command interpreter="python"> stderr_wrapper.py Rscript ${GALAXY_DATA_INDEX_DIR}/deseq.R $counts $column_types $comparison $top_table $diagnostic_html "$diagnostic_html.files_path" "$counts.name" </command>
To use this new feature, the command string now uses the R_SCRIPT_PATH environment variable ( notice the required backslash to escape the $ )
<command interpreter="python"> stderr_wrapper.py Rscript \$R_SCRIPT_PATH/deseq.R $counts $column_types $comparison $top_table $diagnostic_html "$diagnostic_html.files_path" "$counts.name" </command>
When installed from the tool shed, a tool dependency object named R_SCRIPT_PATH will be created and associated with the installed repository. The dependency will have a pointer to the env.sh file that is created to set the value of the R_SCRIPT_PATH environment variable.
Let me know if you bump into any issues in getting this working for your tools.
Thanks!
Greg Von Kuster
On Sep 12, 2012, at 1:00 PM, David Hoover wrote:
Are there any environment variables that are honored by toolshed installed tools? I tried creating a tool that uses ${GALAXY_DATA_INDEX_DIR} or $GALAXY_HOME, then uploaded it the test toolshed, then installed it automatically. Neither of these resolved to what I expected. I don't want to hard-code the path in the xml tool file, but rather have a default location for other executables and jar files. How should this best be done?
David Hoover Helix Systems Staff ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Greg, I think templates/admin/tool_shed_repository/common.mako needs to bypass the 'set_environment' requirement in the following block. Thanks, JJ $ hg diff templates/admin/tool_shed_repository/common.mako diff -r 65ecf4e0ed28 templates/admin/tool_shed_repository/common.mako --- a/templates/admin/tool_shed_repository/common.mako Mon Sep 17 16:24:27 2012 -0400 +++ b/templates/admin/tool_shed_repository/common.mako Mon Sep 17 16:06:52 2012 -0500 @@ -129,28 +129,30 @@ <% package_header_row_displayed = True %> %endif %for dependency_key, requirements_dict in tool_dependencies.items(): - <% - name = requirements_dict[ 'name' ] - version = requirements_dict[ 'version' ] - type = requirements_dict[ 'type' ] - install_dir = os.path.join( trans.app.config.tool_dependency_dir, - name, - version, - repository_owner, - repository_name, - changeset_revision ) - tool_dependency_readme_text = requirements_dict.get( 'readme', None ) - %> - %if not os.path.exists( install_dir ): - <tr> - <td>${name}</td> - <td>${version}</td> - <td>${type}</td> - <td>${install_dir}</td> - </tr> - %if tool_dependency_readme_text: - <tr><td colspan="4" bgcolor="#FFFFCC">${name} ${version} requirements and installation information</td></tr> - <tr><td colspan="4"><pre>${tool_dependency_readme_text}</pre></td></tr> + %if not dependency_key == 'set_environment': + <% + name = requirements_dict[ 'name' ] + version = requirements_dict[ 'version' ] + type = requirements_dict[ 'type' ] + install_dir = os.path.join( trans.app.config.tool_dependency_dir, + name, + version, + repository_owner, + repository_name, + changeset_revision ) + tool_dependency_readme_text = requirements_dict.get( 'readme', None ) + %> + %if not os.path.exists( install_dir ): + <tr> + <td>${name}</td> + <td>${version}</td> + <td>${type}</td> + <td>${install_dir}</td> + </tr> + %if tool_dependency_readme_text: + <tr><td colspan="4" bgcolor="#FFFFCC">${name} ${version} requirements and installation information</td></tr> + <tr><td colspan="4"><pre>${tool_dependency_readme_text}</pre></td></tr> + %endif %endif %endif %endfor On 9/16/12 8:48 AM, Jim Johnson wrote:
Greg, Pablo,
I'm using a SnpEffect repository 'snpeff_with_dep' in http://testtoolshed.g2.bx.psu.edu/ to test using tool_dependencies.xml with the <set_environment> tag. ( You both write access, if you want to use this for correcting my errors or debugging. )
I'm getting an error in prepare_for_install:
Modulegalaxy.web.controllers.admin_toolshed:*1174*in|prepare_for_install| |>>>|||tool_dependencies|| |{'set_environment': [{'type': 'environment variable', 'name': 'JAVA_JAR_PATH'}], 'snpEff/3.0f': {'readme': '\n ', 'version': '3.0f', 'type': 'package', 'name': 'snpEff'}}| ... Module_admin_tool_shed_repository_common_mako:*242*in|render_render_tool_dependency_section| |>> <http://localhost:18080/admin_toolshed/prepare_for_install?tool_shed_url=http://testtoolshed.g2.bx.psu.edu/&repository_ids=d01ace51ab1282a5&changeset_revisions=38b3a6b97bd1#>name*=*requirements_dict*[*'name'*]*| *TypeError: list indices must be integers, not str*
It appears that the requirements_dict for the set_environment tag is enclosed in an array. I haven't yet looked at the repository code to see how that might happen.
Pablo,
The http://sourceforge.net/projects/snpeff/files/ repository should probably have versions packaged for download named like:
http://sourceforge.net/projects/snpeff/files/snpEff_3_0.tgz
$ tar ztvf snpEff_3_0.tgz drwxrwxrwx 0 0 0 0 Aug 23 12:17 snpEff_3_0/ -rwxrwxrwx 0 0 0 14298767 Aug 23 12:17 snpEff_3_0/SnpSift.jar ...
That will allow versioning within galaxy, and successive versions of tool_dependencies.xml with only need a change to the action tag: download_by_url
Thanks,
JJ
I'm trying to test a repository that use in tool_depen
Message: 18 Date: Thu, 13 Sep 2012 11:46:32 -0400 From: Greg Von Kuster<greg@bx.psu.edu> To: David Hoover<hooverdm@helix.nih.gov> Cc:galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] environment variables and paths for toolshed tools Message-ID:<F75D6320-3AF8-4BB0-8F31-B0F44AA1306B@bx.psu.edu> Content-Type: text/plain; charset="us-ascii"
Hello David,
I've just committed change set 7656:6aadac8026cb to the Galaxy central repository that provides the ability to do what you need here. It would be great if you could try things out and let me know if you run into any problems.
I've been working with nikhil-joshi's deseq_and_sam2counts repository in the main Galaxy tool shed because his tools require this new feature as well. This example should provide you with the information you'll need to tweak your tool shed repository so that your tool dependencies are located where they get installed rather than attempting to move them to ${GALAXY_DATA_INDEX_DIR} or some other location.
Here is the tool_dependency.xml file entry for locating a directory referred to by an environment variable name R_SCRIPT_PATH.
<?xml version="1.0"?> <tool_dependency> <set_environment version="1.0"> <environment_variable name="R_SCRIPT_PATH" action="set_to">$REPOSITORY_INSTALL_DIR</environment_variable> </set_environment> </tool_dependency>
The <set_enviroment> tag, is still supported inside <package> tag sets, but when defined at the xml root level, it will locate dependencies included in the installed tool shed repository. Of course, the above tool_dependencies.xml file could of course also include entries for tool dependencies that are packages, For example:
<?xml version="1.0"?> <tool_dependency> <set_environment version="1.0"> <environment_variable name="R_SCRIPT_PATH" action="set_to">$REPOSITORY_INSTALL_DIR</environment_variable> </set_environment> <package name="R" version="2.15.1"> <install version="1.0"> <actions> <action type="download_by_url">http://CRAN.R-project.org/src/base/R-2/R-2.15.1.tar.gz</action> <action type="shell_command">./configure --prefix=$INSTALL_DIR</action> <action type="shell_command">make</action> <action type="set_environment"> <environment_variable name="PATH" action="prepend_to">$INSTALL_DIR/bin</environment_variable> </action> </actions> </install> <readme> You need a FORTRAN compiler or perhaps f2c in addition to a C compiler to build R. </readme> </package> </tool_dependency>
These tool dependency definitions are handled as described in the following section of the tool shed wiki:
http://wiki.g2.bx.psu.edu/Tool%20Shed#Automatic_third-party_tool_dependency_...
So in order for the dependencies to be handled when the repository is installed, they must be defined in at least one of the <requirement> tag sets in at least 1 tool config in the repository. So, the <requirements> tag set in the deseq.xml tool config file will look something like this (there are more requirement tags defined here than in the tool_dependencies.xml fiel above, but you should get the idea).
<requirements> <requirement type="set_environment">R_SCRIPT_PATH</requirement> <requirement type="package" version="2.15.1">R</requirement> <requirement type="package" version="2.10">Bioconductor</requirement> <requirement type="package" version="1.8.3">DESeq</requirement> <requirement type="package" version="1.24.0">aroma.light</requirement> <requirement type="package" version="0.20-6">lattice</requirement> </requirements>
The <command> tag set in the deseq.xml tool config is also slightly altered. Here is the way it looks before using this new approach:
<command interpreter="python"> stderr_wrapper.py Rscript ${GALAXY_DATA_INDEX_DIR}/deseq.R $counts $column_types $comparison $top_table $diagnostic_html "$diagnostic_html.files_path" "$counts.name" </command>
To use this new feature, the command string now uses the R_SCRIPT_PATH environment variable ( notice the required backslash to escape the $ )
<command interpreter="python"> stderr_wrapper.py Rscript \$R_SCRIPT_PATH/deseq.R $counts $column_types $comparison $top_table $diagnostic_html "$diagnostic_html.files_path" "$counts.name" </command>
When installed from the tool shed, a tool dependency object named R_SCRIPT_PATH will be created and associated with the installed repository. The dependency will have a pointer to the env.sh file that is created to set the value of the R_SCRIPT_PATH environment variable.
Let me know if you bump into any issues in getting this working for your tools.
Thanks!
Greg Von Kuster
On Sep 12, 2012, at 1:00 PM, David Hoover wrote:
Are there any environment variables that are honored by toolshed installed tools? I tried creating a tool that uses ${GALAXY_DATA_INDEX_DIR} or $GALAXY_HOME, then uploaded it the test toolshed, then installed it automatically. Neither of these resolved to what I expected. I don't want to hard-code the path in the xml tool file, but rather have a default location for other executables and jar files. How should this best be done?
David Hoover Helix Systems Staff ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Hi Jim, thanks very much for catching this and providing the fix (I've been away from email for a few days, so am just getting caught up). I've committed this in change set 7705:ba64c2178fbe. Greg Von Kuster On Sep 17, 2012, at 5:13 PM, Jim Johnson wrote:
Greg,
I think templates/admin/tool_shed_repository/common.mako needs to bypass the 'set_environment' requirement in the following block.
Thanks,
JJ
$ hg diff templates/admin/tool_shed_repository/common.mako diff -r 65ecf4e0ed28 templates/admin/tool_shed_repository/common.mako --- a/templates/admin/tool_shed_repository/common.mako Mon Sep 17 16:24:27 2012 -0400 +++ b/templates/admin/tool_shed_repository/common.mako Mon Sep 17 16:06:52 2012 -0500 @@ -129,28 +129,30 @@ <% package_header_row_displayed = True %> %endif %for dependency_key, requirements_dict in tool_dependencies.items(): - <% - name = requirements_dict[ 'name' ] - version = requirements_dict[ 'version' ] - type = requirements_dict[ 'type' ] - install_dir = os.path.join( trans.app.config.tool_dependency_dir, - name, - version, - repository_owner, - repository_name, - changeset_revision ) - tool_dependency_readme_text = requirements_dict.get( 'readme', None ) - %> - %if not os.path.exists( install_dir ): - <tr> - <td>${name}</td> - <td>${version}</td> - <td>${type}</td> - <td>${install_dir}</td> - </tr> - %if tool_dependency_readme_text: - <tr><td colspan="4" bgcolor="#FFFFCC">${name} ${version} requirements and installation information</td></tr> - <tr><td colspan="4"><pre>${tool_dependency_readme_text}</pre></td></tr> + %if not dependency_key == 'set_environment': + <% + name = requirements_dict[ 'name' ] + version = requirements_dict[ 'version' ] + type = requirements_dict[ 'type' ] + install_dir = os.path.join( trans.app.config.tool_dependency_dir, + name, + version, + repository_owner, + repository_name, + changeset_revision ) + tool_dependency_readme_text = requirements_dict.get( 'readme', None ) + %> + %if not os.path.exists( install_dir ): + <tr> + <td>${name}</td> + <td>${version}</td> + <td>${type}</td> + <td>${install_dir}</td> + </tr> + %if tool_dependency_readme_text: + <tr><td colspan="4" bgcolor="#FFFFCC">${name} ${version} requirements and installation information</td></tr> + <tr><td colspan="4"><pre>${tool_dependency_readme_text}</pre></td></tr> + %endif %endif %endif %endfor
On 9/16/12 8:48 AM, Jim Johnson wrote:
Greg, Pablo,
I'm using a SnpEffect repository 'snpeff_with_dep' in http://testtoolshed.g2.bx.psu.edu/ to test using tool_dependencies.xml with the <set_environment> tag. ( You both write access, if you want to use this for correcting my errors or debugging. )
I'm getting an error in prepare_for_install:
Module galaxy.web.controllers.admin_toolshed:1174 in prepare_for_install
tool_dependencies {'set_environment': [{'type': 'environment variable', 'name': 'JAVA_JAR_PATH'}], 'snpEff/3.0f': {'readme': '\n ', 'version': '3.0f', 'type': 'package', 'name': 'snpEff'}} ... Module _admin_tool_shed_repository_common_mako:242 in render_render_tool_dependency_section name = requirements_dict[ 'name' ] TypeError: list indices must be integers, not str
It appears that the requirements_dict for the set_environment tag is enclosed in an array. I haven't yet looked at the repository code to see how that might happen.
Pablo,
The http://sourceforge.net/projects/snpeff/files/ repository should probably have versions packaged for download named like:
http://sourceforge.net/projects/snpeff/files/snpEff_3_0.tgz
$ tar ztvf snpEff_3_0.tgz drwxrwxrwx 0 0 0 0 Aug 23 12:17 snpEff_3_0/ -rwxrwxrwx 0 0 0 14298767 Aug 23 12:17 snpEff_3_0/SnpSift.jar ...
That will allow versioning within galaxy, and successive versions of tool_dependencies.xml with only need a change to the action tag: download_by_url
Thanks,
JJ
I'm trying to test a repository that use in tool_depen
Message: 18 Date: Thu, 13 Sep 2012 11:46:32 -0400 From: Greg Von Kuster <greg@bx.psu.edu> To: David Hoover <hooverdm@helix.nih.gov> Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] environment variables and paths for toolshed tools Message-ID: <F75D6320-3AF8-4BB0-8F31-B0F44AA1306B@bx.psu.edu> Content-Type: text/plain; charset="us-ascii"
Hello David,
I've just committed change set 7656:6aadac8026cb to the Galaxy central repository that provides the ability to do what you need here. It would be great if you could try things out and let me know if you run into any problems.
I've been working with nikhil-joshi's deseq_and_sam2counts repository in the main Galaxy tool shed because his tools require this new feature as well. This example should provide you with the information you'll need to tweak your tool shed repository so that your tool dependencies are located where they get installed rather than attempting to move them to ${GALAXY_DATA_INDEX_DIR} or some other location.
Here is the tool_dependency.xml file entry for locating a directory referred to by an environment variable name R_SCRIPT_PATH.
<?xml version="1.0"?> <tool_dependency> <set_environment version="1.0"> <environment_variable name="R_SCRIPT_PATH" action="set_to">$REPOSITORY_INSTALL_DIR</environment_variable> </set_environment> </tool_dependency>
The <set_enviroment> tag, is still supported inside <package> tag sets, but when defined at the xml root level, it will locate dependencies included in the installed tool shed repository. Of course, the above tool_dependencies.xml file could of course also include entries for tool dependencies that are packages, For example:
<?xml version="1.0"?> <tool_dependency> <set_environment version="1.0"> <environment_variable name="R_SCRIPT_PATH" action="set_to">$REPOSITORY_INSTALL_DIR</environment_variable> </set_environment> <package name="R" version="2.15.1"> <install version="1.0"> <actions> <action type="download_by_url">http://CRAN.R-project.org/src/base/R-2/R-2.15.1.tar.gz</action> <action type="shell_command">./configure --prefix=$INSTALL_DIR</action> <action type="shell_command">make</action> <action type="set_environment"> <environment_variable name="PATH" action="prepend_to">$INSTALL_DIR/bin</environment_variable> </action> </actions> </install> <readme> You need a FORTRAN compiler or perhaps f2c in addition to a C compiler to build R. </readme> </package> </tool_dependency>
These tool dependency definitions are handled as described in the following section of the tool shed wiki:
http://wiki.g2.bx.psu.edu/Tool%20Shed#Automatic_third-party_tool_dependency_...
So in order for the dependencies to be handled when the repository is installed, they must be defined in at least one of the <requirement> tag sets in at least 1 tool config in the repository. So, the <requirements> tag set in the deseq.xml tool config file will look something like this (there are more requirement tags defined here than in the tool_dependencies.xml fiel above, but you should get the idea).
<requirements> <requirement type="set_environment">R_SCRIPT_PATH</requirement> <requirement type="package" version="2.15.1">R</requirement> <requirement type="package" version="2.10">Bioconductor</requirement> <requirement type="package" version="1.8.3">DESeq</requirement> <requirement type="package" version="1.24.0">aroma.light</requirement> <requirement type="package" version="0.20-6">lattice</requirement> </requirements>
The <command> tag set in the deseq.xml tool config is also slightly altered. Here is the way it looks before using this new approach:
<command interpreter="python"> stderr_wrapper.py Rscript ${GALAXY_DATA_INDEX_DIR}/deseq.R $counts $column_types $comparison $top_table $diagnostic_html "$diagnostic_html.files_path" "$counts.name" </command>
To use this new feature, the command string now uses the R_SCRIPT_PATH environment variable ( notice the required backslash to escape the $ )
<command interpreter="python"> stderr_wrapper.py Rscript \$R_SCRIPT_PATH/deseq.R $counts $column_types $comparison $top_table $diagnostic_html "$diagnostic_html.files_path" "$counts.name" </command>
When installed from the tool shed, a tool dependency object named R_SCRIPT_PATH will be created and associated with the installed repository. The dependency will have a pointer to the env.sh file that is created to set the value of the R_SCRIPT_PATH environment variable.
Let me know if you bump into any issues in getting this working for your tools.
Thanks!
Greg Von Kuster
On Sep 12, 2012, at 1:00 PM, David Hoover wrote:
Are there any environment variables that are honored by toolshed installed tools? I tried creating a tool that uses ${GALAXY_DATA_INDEX_DIR} or $GALAXY_HOME, then uploaded it the test toolshed, then installed it automatically. Neither of these resolved to what I expected. I don't want to hard-code the path in the xml tool file, but rather have a default location for other executables and jar files. How should this best be done?
David Hoover Helix Systems Staff ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
<common.mako.diff>___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
participants (2)
-
Greg Von Kuster -
Jim Johnson