Just been reading a bit more about the Galaxy packaging system. Here's a slight modification to what I was suggesting that might fit in a bit better. Apologies for not being more familiar with the existing system before proposing extensions.
Recall that my goal is to support using a system-installed (native) package, at a defined version, which I aim to achieve by loading the appropriate environment module before running a tool.
We still have tool_dependencies.xml defining a package at a particular version, but rather than download and build the source code, there's just a directive that says how to pick up the correct program version at runtime, e.g. which environment module to load.
So instead of the tool_dependencies.xml fragment: <tool_dependency> <package name="bwa" version="0.6.2"> <install version="1.0"> <actions> <action type="download_by_url">http://downloads.sourceforge.net/project/bio-bwa/bwa-0.6.2.tar.bz2</action> <action type="shell_command">make</action> <action type="move_file"> <source>bwa</source> <destination>$INSTALL_DIR/bin</destination> </action> <action type="set_environment"> <environment_variable name="PATH" action="prepend_to">$INSTALL_DIR/bin</environment_variable> </action> </actions> </install> </package> </tool_dependency>
We have something like this (NB: element and attribute names are for illustrative purposes only):
<tool_dependency> <package name="bwa" version="0.6.2"> <use_native> <actions> <action type="module_load">bwa/0.6.2</action> </actions> </use_native> </package> </tool_dependency>
This causes the right thing (module load bwa/0.6.2) to be stuck into the dependencies env.sh file when this package is installed from the toolshed. We could call this toolshed tool native_package_bwa_0_6_2, to avoid confusion with the existing download-and-make one.
We might want a bit of flexibility on what actions are supported (in case we want to support Software Collections, for example).
What do you think?
cheers, Simon
PS: In case it wasn't already clear, solving this problem well is quite important to us here at AgResearch. ;-)
======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================
Hi, has anything changed in galaxy in this regard? Any way to modify an environment before a tools is run?
I now have a tool relying on R-devel and bioconductor devel, both of which I can load in a module. The tool comes from the toolshed with xml like:
<command interpreter="Rscript"> …
I don't want to hack around in the tool itself, but simply load the necessary R-version.
thank you very much, ido
On Sep 13, 2013, at 3:23 AM, "Guest, Simon" Simon.Guest@agresearch.co.nz wrote:
Just been reading a bit more about the Galaxy packaging system. Here's a slight modification to what I was suggesting that might fit in a bit better. Apologies for not being more familiar with the existing system before proposing extensions.
Recall that my goal is to support using a system-installed (native) package, at a defined version, which I aim to achieve by loading the appropriate environment module before running a tool.
We still have tool_dependencies.xml defining a package at a particular version, but rather than download and build the source code, there's just a directive that says how to pick up the correct program version at runtime, e.g. which environment module to load.
So instead of the tool_dependencies.xml fragment:
<tool_dependency> <package name="bwa" version="0.6.2"> <install version="1.0"> <actions> <action type="download_by_url">http://downloads.sourceforge.net/project/bio-bwa/bwa-0.6.2.tar.bz2</action> <action type="shell_command">make</action> <action type="move_file"> <source>bwa</source> <destination>$INSTALL_DIR/bin</destination> </action> <action type="set_environment"> <environment_variable name="PATH" action="prepend_to">$INSTALL_DIR/bin</environment_variable> </action> </actions> </install> </package> </tool_dependency>
We have something like this (NB: element and attribute names are for illustrative purposes only):
<tool_dependency> <package name="bwa" version="0.6.2"> <use_native> <actions> <action type="module_load">bwa/0.6.2</action> </actions> </use_native> </package> </tool_dependency>
This causes the right thing (module load bwa/0.6.2) to be stuck into the dependencies env.sh file when this package is installed from the toolshed. We could call this toolshed tool native_package_bwa_0_6_2, to avoid confusion with the existing download-and-make one.
We might want a bit of flexibility on what actions are supported (in case we want to support Software Collections, for example).
What do you think?
cheers, Simon
PS: In case it wasn't already clear, solving this problem well is quite important to us here at AgResearch. ;-)
======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Ido,
I do not get your question in all detail, but it is possible to define a tool_dependencies.xml with a specific R version, + R libraries and use only that specific version from your tool with <requirement> tags.
For an example please see various R tools from:
https://github.com/bgruening/galaxytools
Cheers, Bjoern
Am 25.04.2014 14:25, schrieb Ido Tamir:
Hi, has anything changed in galaxy in this regard? Any way to modify an environment before a tools is run?
I now have a tool relying on R-devel and bioconductor devel, both of which I can load in a module. The tool comes from the toolshed with xml like:
<command interpreter="Rscript"> …
I don't want to hack around in the tool itself, but simply load the necessary R-version.
thank you very much, ido
On Sep 13, 2013, at 3:23 AM, "Guest, Simon" Simon.Guest@agresearch.co.nz wrote:
Just been reading a bit more about the Galaxy packaging system. Here's a slight modification to what I was suggesting that might fit in a bit better. Apologies for not being more familiar with the existing system before proposing extensions.
Recall that my goal is to support using a system-installed (native) package, at a defined version, which I aim to achieve by loading the appropriate environment module before running a tool.
We still have tool_dependencies.xml defining a package at a particular version, but rather than download and build the source code, there's just a directive that says how to pick up the correct program version at runtime, e.g. which environment module to load.
So instead of the tool_dependencies.xml fragment:
<tool_dependency> <package name="bwa" version="0.6.2"> <install version="1.0"> <actions> <action type="download_by_url">http://downloads.sourceforge.net/project/bio-bwa/bwa-0.6.2.tar.bz2</action> <action type="shell_command">make</action> <action type="move_file"> <source>bwa</source> <destination>$INSTALL_DIR/bin</destination> </action> <action type="set_environment"> <environment_variable name="PATH" action="prepend_to">$INSTALL_DIR/bin</environment_variable> </action> </actions> </install> </package> </tool_dependency>
We have something like this (NB: element and attribute names are for illustrative purposes only):
<tool_dependency> <package name="bwa" version="0.6.2"> <use_native> <actions> <action type="module_load">bwa/0.6.2</action> </actions> </use_native> </package> </tool_dependency>
This causes the right thing (module load bwa/0.6.2) to be stuck into the dependencies env.sh file when this package is installed from the toolshed. We could call this toolshed tool native_package_bwa_0_6_2, to avoid confusion with the existing download-and-make one.
We might want a bit of flexibility on what actions are supported (in case we want to support Software Collections, for example).
What do you think?
cheers, Simon
PS: In case it wasn't already clear, solving this problem well is quite important to us here at AgResearch. ;-)
======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Dear Björn,
maybe I could change the tool, which is not mine, and which I don't want to maintain to use a specific R-version that is already available on our cluster and which I can put into my path with "module load R/3.1.0-devel" ( http://modules.sourceforge.net/)
there even is a requirements in the wrapper which is not fulfilled (these versions are not available at the moment) and still it installed without problems. <requirements> <requirement type="R-module" version="3.5.27">edgeR</requirement> <requirement type="R-module" version="3.18.13">limma</requirement> </requirements>
This requirements tags looks also rather inflexible to me. With an additional level of user configurable indirection it would be possible to make the tools fit to different infrastructures without having to use binaries provided by somebody else, taking up space for just one tool etc...
Currently for jobs that are not run locally there is a general environment_setup_file. There could be one optional environment_setup_file for every job (or destination).
In the end I created yet another wrapper.
best, ido
On Apr 25, 2014, at 2:34 PM, Björn Grüning bjoern.gruening@gmail.com wrote:
Hi Ido,
I do not get your question in all detail, but it is possible to define a tool_dependencies.xml with a specific R version, + R libraries and use only that specific version from your tool with <requirement> tags.
For an example please see various R tools from:
https://github.com/bgruening/galaxytools
Cheers, Bjoern
Am 25.04.2014 14:25, schrieb Ido Tamir:
Hi, has anything changed in galaxy in this regard? Any way to modify an environment before a tools is run?
I now have a tool relying on R-devel and bioconductor devel, both of which I can load in a module. The tool comes from the toolshed with xml like:
<command interpreter="Rscript"> …
I don't want to hack around in the tool itself, but simply load the necessary R-version.
thank you very much, ido
On Sep 13, 2013, at 3:23 AM, "Guest, Simon" Simon.Guest@agresearch.co.nz wrote:
Just been reading a bit more about the Galaxy packaging system. Here's a slight modification to what I was suggesting that might fit in a bit better. Apologies for not being more familiar with the existing system before proposing extensions.
Recall that my goal is to support using a system-installed (native) package, at a defined version, which I aim to achieve by loading the appropriate environment module before running a tool.
We still have tool_dependencies.xml defining a package at a particular version, but rather than download and build the source code, there's just a directive that says how to pick up the correct program version at runtime, e.g. which environment module to load.
So instead of the tool_dependencies.xml fragment:
<tool_dependency> <package name="bwa" version="0.6.2"> <install version="1.0"> <actions> <action type="download_by_url">http://downloads.sourceforge.net/project/bio-bwa/bwa-0.6.2.tar.bz2</action> <action type="shell_command">make</action> <action type="move_file"> <source>bwa</source> <destination>$INSTALL_DIR/bin</destination> </action> <action type="set_environment"> <environment_variable name="PATH" action="prepend_to">$INSTALL_DIR/bin</environment_variable> </action> </actions> </install> </package> </tool_dependency>
We have something like this (NB: element and attribute names are for illustrative purposes only):
<tool_dependency> <package name="bwa" version="0.6.2"> <use_native> <actions> <action type="module_load">bwa/0.6.2</action> </actions> </use_native> </package> </tool_dependency>
This causes the right thing (module load bwa/0.6.2) to be stuck into the dependencies env.sh file when this package is installed from the toolshed. We could call this toolshed tool native_package_bwa_0_6_2, to avoid confusion with the existing download-and-make one.
We might want a bit of flexibility on what actions are supported (in case we want to support Software Collections, for example).
What do you think?
cheers, Simon
PS: In case it wasn't already clear, solving this problem well is quite important to us here at AgResearch. ;-)
======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hello Ido,
Reading through your e-mail it is not clear to me whether the tool has requirements that could be overridden. If it has package requirements explicitly defined - those are completely configurable - you can write arbitrary python plugins to resolve these any you wish - and (if the modules are available on the galaxy server) there is an existing plugin that can be used to resolve these packages as modules. I think the best documentation of this may be on this pull request (https://bitbucket.org/galaxy/galaxy-central/pull-request/228/tool-dependency...).
Additionally (perhaps more pertinently), I have an open pull request (created just this week) to allow setting environment variables on a per destination basis - this is sort of what you are getting at and might be a better way to go - https://bitbucket.org/galaxy/galaxy-central/pull-request/378/allow-specifica.... With this modification you could hack up the path to reflect the changes made by the module load. That is a bit of hack but would work for this case. That said I think you make a good point about the environment file, I will modify the new <env> tags to allow a file attribute and just source that file. This will give you your per destination environment files - hopefully this is satisfactory.
-John
On Fri, Apr 25, 2014 at 8:22 AM, Ido Tamir tamir@imp.ac.at wrote:
Dear Björn,
maybe I could change the tool, which is not mine, and which I don't want to maintain to use a specific R-version that is already available on our cluster and which I can put into my path with "module load R/3.1.0-devel" ( http://modules.sourceforge.net/)
there even is a requirements in the wrapper which is not fulfilled (these versions are not available at the moment) and still it installed without problems.
<requirements> <requirement type="R-module" version="3.5.27">edgeR</requirement> <requirement type="R-module" version="3.18.13">limma</requirement> </requirements>
This requirements tags looks also rather inflexible to me. With an additional level of user configurable indirection it would be possible to make the tools fit to different infrastructures without having to use binaries provided by somebody else, taking up space for just one tool etc...
Currently for jobs that are not run locally there is a general environment_setup_file. There could be one optional environment_setup_file for every job (or destination).
In the end I created yet another wrapper.
best, ido
On Apr 25, 2014, at 2:34 PM, Björn Grüning bjoern.gruening@gmail.com wrote:
Hi Ido,
I do not get your question in all detail, but it is possible to define a tool_dependencies.xml with a specific R version, + R libraries and use only that specific version from your tool with <requirement> tags.
For an example please see various R tools from:
https://github.com/bgruening/galaxytools
Cheers, Bjoern
Am 25.04.2014 14:25, schrieb Ido Tamir:
Hi, has anything changed in galaxy in this regard? Any way to modify an environment before a tools is run?
I now have a tool relying on R-devel and bioconductor devel, both of which I can load in a module. The tool comes from the toolshed with xml like:
<command interpreter="Rscript"> …
I don't want to hack around in the tool itself, but simply load the necessary R-version.
thank you very much, ido
On Sep 13, 2013, at 3:23 AM, "Guest, Simon" Simon.Guest@agresearch.co.nz wrote:
Just been reading a bit more about the Galaxy packaging system. Here's a slight modification to what I was suggesting that might fit in a bit better. Apologies for not being more familiar with the existing system before proposing extensions.
Recall that my goal is to support using a system-installed (native) package, at a defined version, which I aim to achieve by loading the appropriate environment module before running a tool.
We still have tool_dependencies.xml defining a package at a particular version, but rather than download and build the source code, there's just a directive that says how to pick up the correct program version at runtime, e.g. which environment module to load.
So instead of the tool_dependencies.xml fragment:
<tool_dependency> <package name="bwa" version="0.6.2"> <install version="1.0"> <actions> <action type="download_by_url">http://downloads.sourceforge.net/project/bio-bwa/bwa-0.6.2.tar.bz2</action> <action type="shell_command">make</action> <action type="move_file"> <source>bwa</source> <destination>$INSTALL_DIR/bin</destination> </action> <action type="set_environment"> <environment_variable name="PATH" action="prepend_to">$INSTALL_DIR/bin</environment_variable> </action> </actions> </install> </package> </tool_dependency>
We have something like this (NB: element and attribute names are for illustrative purposes only):
<tool_dependency> <package name="bwa" version="0.6.2"> <use_native> <actions> <action type="module_load">bwa/0.6.2</action> </actions> </use_native> </package> </tool_dependency>
This causes the right thing (module load bwa/0.6.2) to be stuck into the dependencies env.sh file when this package is installed from the toolshed. We could call this toolshed tool native_package_bwa_0_6_2, to avoid confusion with the existing download-and-make one.
We might want a bit of flexibility on what actions are supported (in case we want to support Software Collections, for example).
What do you think?
cheers, Simon
PS: In case it wasn't already clear, solving this problem well is quite important to us here at AgResearch. ;-)
======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Ida,
now I got it. I may have an idea, not tested, but you could try the following.
- Add a file under $galaxy_tools/R_env/3.1.0/env.sh - Create a symlink from $galaxy_tools/R_env/default/ to $galaxy_tools/R_env/3.1.0/ - change your requirement tag to <requirement type="package">R_env</requirement>
Now you can put in everything you want have in the env.sh file. That file will be sourced as soon as your tool will be executed. For example you can load your module.
Hope that works! Thanks, Bjoern
Dear Björn,
maybe I could change the tool, which is not mine, and which I don't want to maintain to use a specific R-version that is already available on our cluster and which I can put into my path with "module load R/3.1.0-devel" ( http://modules.sourceforge.net/)
there even is a requirements in the wrapper which is not fulfilled (these versions are not available at the moment) and still it installed without problems.
<requirements> <requirement type="R-module" version="3.5.27">edgeR</requirement> <requirement type="R-module" version="3.18.13">limma</requirement> </requirements>
This requirements tags looks also rather inflexible to me. With an additional level of user configurable indirection it would be possible to make the tools fit to different infrastructures without having to use binaries provided by somebody else, taking up space for just one tool etc...
Currently for jobs that are not run locally there is a general environment_setup_file. There could be one optional environment_setup_file for every job (or destination).
In the end I created yet another wrapper.
best, ido
On Apr 25, 2014, at 2:34 PM, Björn Grüning bjoern.gruening@gmail.com wrote:
Hi Ido,
I do not get your question in all detail, but it is possible to define a tool_dependencies.xml with a specific R version, + R libraries and use only that specific version from your tool with <requirement> tags.
For an example please see various R tools from:
https://github.com/bgruening/galaxytools
Cheers, Bjoern
Am 25.04.2014 14:25, schrieb Ido Tamir:
Hi, has anything changed in galaxy in this regard? Any way to modify an environment before a tools is run?
I now have a tool relying on R-devel and bioconductor devel, both of which I can load in a module. The tool comes from the toolshed with xml like:
<command interpreter="Rscript"> …
I don't want to hack around in the tool itself, but simply load the necessary R-version.
thank you very much, ido
On Sep 13, 2013, at 3:23 AM, "Guest, Simon" Simon.Guest@agresearch.co.nz wrote:
Just been reading a bit more about the Galaxy packaging system. Here's a slight modification to what I was suggesting that might fit in a bit better. Apologies for not being more familiar with the existing system before proposing extensions.
Recall that my goal is to support using a system-installed (native) package, at a defined version, which I aim to achieve by loading the appropriate environment module before running a tool.
We still have tool_dependencies.xml defining a package at a particular version, but rather than download and build the source code, there's just a directive that says how to pick up the correct program version at runtime, e.g. which environment module to load.
So instead of the tool_dependencies.xml fragment:
<tool_dependency> <package name="bwa" version="0.6.2"> <install version="1.0"> <actions> <action type="download_by_url">http://downloads.sourceforge.net/project/bio-bwa/bwa-0.6.2.tar.bz2</action> <action type="shell_command">make</action> <action type="move_file"> <source>bwa</source> <destination>$INSTALL_DIR/bin</destination> </action> <action type="set_environment"> <environment_variable name="PATH" action="prepend_to">$INSTALL_DIR/bin</environment_variable> </action> </actions> </install> </package> </tool_dependency>
We have something like this (NB: element and attribute names are for illustrative purposes only):
<tool_dependency> <package name="bwa" version="0.6.2"> <use_native> <actions> <action type="module_load">bwa/0.6.2</action> </actions> </use_native> </package> </tool_dependency>
This causes the right thing (module load bwa/0.6.2) to be stuck into the dependencies env.sh file when this package is installed from the toolshed. We could call this toolshed tool native_package_bwa_0_6_2, to avoid confusion with the existing download-and-make one.
We might want a bit of flexibility on what actions are supported (in case we want to support Software Collections, for example).
What do you think?
cheers, Simon
PS: In case it wasn't already clear, solving this problem well is quite important to us here at AgResearch. ;-)
======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
On Apr 25, 2014, at 3:55 PM, John Chilton jmchilton@gmail.com wrote:
Additionally (perhaps more pertinently), I have an open pull request (created just this week) to allow setting environment variables on a per destination basis - this is sort of what you are getting at and might be a better way to go - https://bitbucket.org/galaxy/galaxy-central/pull-request/378/allow-specifica.... With this modification you could hack up the path to reflect the changes made by the module load. That is a bit of hack but would work for this case. That said I think you make a good point about the environment file, I will modify the new <env> tags to allow a file attribute and just source that file. This will give you your per destination environment files - hopefully this is satisfactory.
That sounds great!
best, ido
On Fri, Apr 25, 2014 at 8:22 AM, Ido Tamir tamir@imp.ac.at wrote:
Dear Björn,
maybe I could change the tool, which is not mine, and which I don't want to maintain to use a specific R-version that is already available on our cluster and which I can put into my path with "module load R/3.1.0-devel" ( http://modules.sourceforge.net/)
there even is a requirements in the wrapper which is not fulfilled (these versions are not available at the moment) and still it installed without problems.
<requirements> <requirement type="R-module" version="3.5.27">edgeR</requirement> <requirement type="R-module" version="3.18.13">limma</requirement> </requirements>
This requirements tags looks also rather inflexible to me. With an additional level of user configurable indirection it would be possible to make the tools fit to different infrastructures without having to use binaries provided by somebody else, taking up space for just one tool etc...
Currently for jobs that are not run locally there is a general environment_setup_file. There could be one optional environment_setup_file for every job (or destination).
In the end I created yet another wrapper.
best, ido
On Apr 25, 2014, at 2:34 PM, Björn Grüning bjoern.gruening@gmail.com wrote:
Hi Ido,
I do not get your question in all detail, but it is possible to define a tool_dependencies.xml with a specific R version, + R libraries and use only that specific version from your tool with <requirement> tags.
For an example please see various R tools from:
https://github.com/bgruening/galaxytools
Cheers, Bjoern
Am 25.04.2014 14:25, schrieb Ido Tamir:
Hi, has anything changed in galaxy in this regard? Any way to modify an environment before a tools is run?
I now have a tool relying on R-devel and bioconductor devel, both of which I can load in a module. The tool comes from the toolshed with xml like:
<command interpreter="Rscript"> …
I don't want to hack around in the tool itself, but simply load the necessary R-version.
thank you very much, ido
On Sep 13, 2013, at 3:23 AM, "Guest, Simon" Simon.Guest@agresearch.co.nz wrote:
Just been reading a bit more about the Galaxy packaging system. Here's a slight modification to what I was suggesting that might fit in a bit better. Apologies for not being more familiar with the existing system before proposing extensions.
Recall that my goal is to support using a system-installed (native) package, at a defined version, which I aim to achieve by loading the appropriate environment module before running a tool.
We still have tool_dependencies.xml defining a package at a particular version, but rather than download and build the source code, there's just a directive that says how to pick up the correct program version at runtime, e.g. which environment module to load.
So instead of the tool_dependencies.xml fragment:
<tool_dependency> <package name="bwa" version="0.6.2"> <install version="1.0"> <actions> <action type="download_by_url">http://downloads.sourceforge.net/project/bio-bwa/bwa-0.6.2.tar.bz2</action> <action type="shell_command">make</action> <action type="move_file"> <source>bwa</source> <destination>$INSTALL_DIR/bin</destination> </action> <action type="set_environment"> <environment_variable name="PATH" action="prepend_to">$INSTALL_DIR/bin</environment_variable> </action> </actions> </install> </package> </tool_dependency>
We have something like this (NB: element and attribute names are for illustrative purposes only):
<tool_dependency> <package name="bwa" version="0.6.2"> <use_native> <actions> <action type="module_load">bwa/0.6.2</action> </actions> </use_native> </package> </tool_dependency>
This causes the right thing (module load bwa/0.6.2) to be stuck into the dependencies env.sh file when this package is installed from the toolshed. We could call this toolshed tool native_package_bwa_0_6_2, to avoid confusion with the existing download-and-make one.
We might want a bit of flexibility on what actions are supported (in case we want to support Software Collections, for example).
What do you think?
cheers, Simon
PS: In case it wasn't already clear, solving this problem well is quite important to us here at AgResearch. ;-)
======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Great - I have added the option to the pull request https://bitbucket.org/jmchilton/galaxy-central-fork-1/commits/d72967b2b7b25b... as well as an option to supply a command instead of a file so you can specify the module load directly in job_conf.xml without having to create a file. Barring objections, I will merge this pull request later today or early next week and it should be available in the June release.
All of that said - I agree with Bjoern approach completely and would go one step further - if working with the packager to modify the tool to add a package tag that you can create a standard env.sh file for that would be the 'recommend' solution :).
-John
On Fri, Apr 25, 2014 at 9:06 AM, Björn Grüning bjoern.gruening@gmail.com wrote:
Hi Ida,
now I got it. I may have an idea, not tested, but you could try the following.
- Add a file under $galaxy_tools/R_env/3.1.0/env.sh
- Create a symlink from $galaxy_tools/R_env/default/ to
$galaxy_tools/R_env/3.1.0/
- change your requirement tag to
<requirement type="package">R_env</requirement>
Now you can put in everything you want have in the env.sh file. That file will be sourced as soon as your tool will be executed. For example you can load your module.
Hope that works! Thanks, Bjoern
Dear Björn,
maybe I could change the tool, which is not mine, and which I don't want to maintain to use a specific R-version that is already available on our cluster and which I can put into my path with "module load R/3.1.0-devel" ( http://modules.sourceforge.net/)
there even is a requirements in the wrapper which is not fulfilled (these versions are not available at the moment) and still it installed without problems.
<requirements> <requirement type="R-module" version="3.5.27">edgeR</requirement> <requirement type="R-module" version="3.18.13">limma</requirement> </requirements>
This requirements tags looks also rather inflexible to me. With an additional level of user configurable indirection it would be possible to make the tools fit to different infrastructures without having to use binaries provided by somebody else, taking up space for just one tool etc...
Currently for jobs that are not run locally there is a general environment_setup_file. There could be one optional environment_setup_file for every job (or destination).
In the end I created yet another wrapper.
best, ido
On Apr 25, 2014, at 2:34 PM, Björn Grüning bjoern.gruening@gmail.com wrote:
Hi Ido,
I do not get your question in all detail, but it is possible to define a tool_dependencies.xml with a specific R version, + R libraries and use only that specific version from your tool with <requirement> tags.
For an example please see various R tools from:
https://github.com/bgruening/galaxytools
Cheers, Bjoern
Am 25.04.2014 14:25, schrieb Ido Tamir:
Hi, has anything changed in galaxy in this regard? Any way to modify an environment before a tools is run?
I now have a tool relying on R-devel and bioconductor devel, both of which I can load in a module. The tool comes from the toolshed with xml like:
<command interpreter="Rscript"> …
I don't want to hack around in the tool itself, but simply load the necessary R-version.
thank you very much, ido
On Sep 13, 2013, at 3:23 AM, "Guest, Simon" Simon.Guest@agresearch.co.nz wrote:
Just been reading a bit more about the Galaxy packaging system. Here's a slight modification to what I was suggesting that might fit in a bit better. Apologies for not being more familiar with the existing system before proposing extensions.
Recall that my goal is to support using a system-installed (native) package, at a defined version, which I aim to achieve by loading the appropriate environment module before running a tool.
We still have tool_dependencies.xml defining a package at a particular version, but rather than download and build the source code, there's just a directive that says how to pick up the correct program version at runtime, e.g. which environment module to load.
So instead of the tool_dependencies.xml fragment:
<tool_dependency> <package name="bwa" version="0.6.2"> <install version="1.0"> <actions> <action type="download_by_url">http://downloads.sourceforge.net/project/bio-bwa/bwa-0.6.2.tar.bz2</action> <action type="shell_command">make</action> <action type="move_file"> <source>bwa</source> <destination>$INSTALL_DIR/bin</destination> </action> <action type="set_environment"> <environment_variable name="PATH" action="prepend_to">$INSTALL_DIR/bin</environment_variable> </action> </actions> </install> </package> </tool_dependency>
We have something like this (NB: element and attribute names are for illustrative purposes only):
<tool_dependency> <package name="bwa" version="0.6.2"> <use_native> <actions> <action type="module_load">bwa/0.6.2</action> </actions> </use_native> </package> </tool_dependency>
This causes the right thing (module load bwa/0.6.2) to be stuck into the dependencies env.sh file when this package is installed from the toolshed. We could call this toolshed tool native_package_bwa_0_6_2, to avoid confusion with the existing download-and-make one.
We might want a bit of flexibility on what actions are supported (in case we want to support Software Collections, for example).
What do you think?
cheers, Simon
PS: In case it wasn't already clear, solving this problem well is quite important to us here at AgResearch. ;-)
======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
galaxy-dev@lists.galaxyproject.org
-
Björn Grüning -
Guest, Simon -
Ido Tamir -
John Chilton