Hi All,
I'm currently caught between the requests of our Galaxy users to install things from main tool-shed, and our information security dept's concerns r.e. the automated installation of tool-deps on our systems. Due to restrictive web access policies for servers here our galaxy server can't access SourceForge, where many tool-dep downloads are. A request to unblock this for a particular tool-dep package led to our infosec justifiably raising concerns r.e. a tool-dep package that is quite out of date (details sent off list previously). We're now currently unable to install tool-shed tools that users have requested.
The current proposal from our infosec dept is to get all our deps from system repos etc. However the way I'm aware of implementing this for tool-shed tools, which need to run across our cluster, would be something pretty arduous like:
* Clone the tool from the upstream toolshed repo * Edit the tool code to remove the package requirements * Identify and install all the requirements on the cluster as system pkgs / environment modules - with attention to versions so things work as expected * Edit the tool code so it knows to load the right environment modules / set right PATH when it runs * Install the tool into our galaxy 'tools' dir , not the 'shed_tools' * Manually add the tool to galaxy's tool_conf.xml.main * Schedule downtime to restart galaxy * Test things out
.... or we have to host our own tool shed, import tools we want from upstream, edit out the package requirements, provide the deps ourselves. These have all the headaches of merging things in when upstream shed-tools change.
Just wondering if I'm missing anything? I know you can turn off 'handle repository dependencies' when installing a tool, but the tool still defines 'requirements' in its XML file and shows 'Missing repository/tool dependencies' in the Admin. Has anyone had any experience of dealing with this kind of situation?
Many thanks!
-- David Trudgian Ph.D. Computational Scientist, BioHPC Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833
________________________________
UT Southwestern
Medical Center
The future of medicine, today.
Hi David,
which sites are blocked? Can you access tools that take it's binaries from https://github.com/galaxyproject/cargo-port?
If so Eric has ported all URLs over to cargo-port sometime ago. So we just need to update the packages for you. This only holds true for IUC packages and we probably need to update the toolshed repos for a few packages.
An other solution would be to get a similar setup as your cluster running elsewhere install all tools on this computer and just move your installations over. This should work as well.
Cheers, Bjoern
On 12.04.2016 23:42, David Trudgian wrote:
Hi All,
I'm currently caught between the requests of our Galaxy users to install things from main tool-shed, and our information security dept's concerns r.e. the automated installation of tool-deps on our systems. Due to restrictive web access policies for servers here our galaxy server can't access SourceForge, where many tool-dep downloads are. A request to unblock this for a particular tool-dep package led to our infosec justifiably raising concerns r.e. a tool-dep package that is quite out of date (details sent off list previously). We're now currently unable to install tool-shed tools that users have requested.
The current proposal from our infosec dept is to get all our deps from system repos etc. However the way I'm aware of implementing this for tool-shed tools, which need to run across our cluster, would be something pretty arduous like:
- Clone the tool from the upstream toolshed repo
- Edit the tool code to remove the package requirements
- Identify and install all the requirements on the cluster as system pkgs / environment modules - with attention to versions so things work as expected
- Edit the tool code so it knows to load the right environment modules / set right PATH when it runs
- Install the tool into our galaxy 'tools' dir , not the 'shed_tools'
- Manually add the tool to galaxy's tool_conf.xml.main
- Schedule downtime to restart galaxy
- Test things out
.... or we have to host our own tool shed, import tools we want from upstream, edit out the package requirements, provide the deps ourselves. These have all the headaches of merging things in when upstream shed-tools change.
Just wondering if I'm missing anything? I know you can turn off 'handle repository dependencies' when installing a tool, but the tool still defines 'requirements' in its XML file and shows 'Missing repository/tool dependencies' in the Admin. Has anyone had any experience of dealing with this kind of situation?
Many thanks!
-- David Trudgian Ph.D. Computational Scientist, BioHPC Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833
UT Southwestern
Medical Center
The future of medicine, today.
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi David,
one thing that you might be missing is that the wrapper for the infosec-approved version might not exist yet (you would have to adapt the older shed-wrapped version if there were any relevant changes). But this largely depends on what software you want to use.
Besides that your flow seems quite accurate. I would probably argue against own tool shed, as the maintenance overhead will probably not be worth it for you.
Let us know if we can help in any way,
thank you for using Galaxy.
Martin
On Tue, Apr 12, 2016 at 5:42 PM David Trudgian < David.Trudgian@utsouthwestern.edu> wrote:
Hi All,
I’m currently caught between the requests of our Galaxy users to install things from main tool-shed, and our information security dept’s concerns r.e. the automated installation of tool-deps on our systems. Due to restrictive web access policies for servers here our galaxy server can’t access SourceForge, where many tool-dep downloads are. A request to unblock this for a particular tool-dep package led to our infosec justifiably raising concerns r.e. a tool-dep package that is quite out of date (details sent off list previously). We’re now currently unable to install tool-shed tools that users have requested.
The current proposal from our infosec dept is to get all our deps from system repos etc. However the way I’m aware of implementing this for tool-shed tools, which need to run across our cluster, would be something pretty arduous like:
Clone the tool from the upstream toolshed repo
Edit the tool code to remove the package requirements
Identify and install all the requirements on the cluster as system pkgs
/ environment modules – with attention to versions so things work as expected
- Edit the tool code so it knows to load the right environment modules /
set right PATH when it runs
Install the tool into our galaxy ‘tools’ dir , not the ‘shed_tools’
Manually add the tool to galaxy’s tool_conf.xml.main
Schedule downtime to restart galaxy
Test things out
…. or we have to host our own tool shed, import tools we want from upstream, edit out the package requirements, provide the deps ourselves. These have all the headaches of merging things in when upstream shed-tools change.
Just wondering if I’m missing anything? I know you can turn off ‘handle repository dependencies’ when installing a tool, but the tool still defines ‘requirements’ in its XML file and shows ‘Missing repository/tool dependencies’ in the Admin. Has anyone had any experience of dealing with this kind of situation?
Many thanks!
-- David Trudgian Ph.D. Computational Scientist, BioHPC
Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833
UT Southwestern
Medical Center
The future of medicine, today. ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Bjoern,
Yes, we can reach stuff in the cargo-port. Unfortunately doing that isn’t a solution as the package that concerns our infosec is the same older version there in cargo-port as it has been trying to pull from SourceForge. Galaxy might be able to get it then, but I’m not permitted to install that package.
Thanks,
DT
-- David Trudgian Ph.D. Computational Scientist, BioHPC Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833
From: Bjoern Gruening [mailto:bjoern.gruening@gmail.com] Sent: Wednesday, April 13, 2016 8:46 AM To: David Trudgian David.Trudgian@UTSouthwestern.edu; galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Tool shed tools, manual dep installation
Hi David,
which sites are blocked? Can you access tools that take it's binaries from https://github.com/galaxyproject/cargo-port?
If so Eric has ported all URLs over to cargo-port sometime ago. So we just need to update the packages for you. This only holds true for IUC packages and we probably need to update the toolshed repos for a few packages.
An other solution would be to get a similar setup as your cluster running elsewhere install all tools on this computer and just move your installations over. This should work as well.
Cheers, Bjoern On 12.04.2016 23:42, David Trudgian wrote:
Hi All,
I'm currently caught between the requests of our Galaxy users to install things from main tool-shed, and our information security dept's concerns r.e. the automated installation of tool-deps on our systems. Due to restrictive web access policies for servers here our galaxy server can't access SourceForge, where many tool-dep downloads are. A request to unblock this for a particular tool-dep package led to our infosec justifiably raising concerns r.e. a tool-dep package that is quite out of date (details sent off list previously). We're now currently unable to install tool-shed tools that users have requested.
The current proposal from our infosec dept is to get all our deps from system repos etc. However the way I'm aware of implementing this for tool-shed tools, which need to run across our cluster, would be something pretty arduous like:
* Clone the tool from the upstream toolshed repo
* Edit the tool code to remove the package requirements
* Identify and install all the requirements on the cluster as system pkgs / environment modules - with attention to versions so things work as expected
* Edit the tool code so it knows to load the right environment modules / set right PATH when it runs
* Install the tool into our galaxy 'tools' dir , not the 'shed_tools'
* Manually add the tool to galaxy's tool_conf.xml.main
* Schedule downtime to restart galaxy
* Test things out
.... or we have to host our own tool shed, import tools we want from upstream, edit out the package requirements, provide the deps ourselves. These have all the headaches of merging things in when upstream shed-tools change.
Just wondering if I'm missing anything? I know you can turn off 'handle repository dependencies' when installing a tool, but the tool still defines 'requirements' in its XML file and shows 'Missing repository/tool dependencies' in the Admin. Has anyone had any experience of dealing with this kind of situation?
Many thanks!
--
David Trudgian Ph.D.
Computational Scientist, BioHPC
Lyda Hill Department of Bioinformatics
UT Southwestern Medical Center
Dallas, TX 75390-9039
Tel: (214) 648-4833
________________________________
UT Southwestern
Medical Center
The future of medicine, today.
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at:
Hi Martin,
Thanks for the input - yes there is going to be maintenance headache either way, private tool-shed or not. If we don't go with our own tool-shed though, am I right in thinking we lose versioning for tools? I.E. installing manually into 'tools' and tool_conf.xml I can't use <version> tags like the tool shed stuff does in shed_tool.conf. The infosec concern is mainly with libraries which are provided by tool-shed dependency package installs, not the main software package the tool wrapper is calling. With a tool shed it's feasible then we could keep older versions of tools/wrappers for reproducibility, rebuilding them against updated (but compatible) libraries when there's a security issue, perhaps?
Glad to know that my idea of the workflow isn't totally wrong though.
Thanks,
DT
-- David Trudgian Ph.D. Computational Scientist, BioHPC Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833
From: Martin Čech [mailto:marten@bx.psu.edu] Sent: Wednesday, April 13, 2016 8:47 AM To: David Trudgian David.Trudgian@UTSouthwestern.edu; galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Tool shed tools, manual dep installation
Hi David,
one thing that you might be missing is that the wrapper for the infosec-approved version might not exist yet (you would have to adapt the older shed-wrapped version if there were any relevant changes). But this largely depends on what software you want to use.
Besides that your flow seems quite accurate. I would probably argue against own tool shed, as the maintenance overhead will probably not be worth it for you.
Let us know if we can help in any way,
thank you for using Galaxy.
Martin
On Tue, Apr 12, 2016 at 5:42 PM David Trudgian David.Trudgian@utsouthwestern.edu wrote: Hi All, I’m currently caught between the requests of our Galaxy users to install things from main tool-shed, and our information security dept’s concerns r.e. the automated installation of tool-deps on our systems. Due to restrictive web access policies for servers here our galaxy server can’t access SourceForge, where many tool-dep downloads are. A request to unblock this for a particular tool-dep package led to our infosec justifiably raising concerns r.e. a tool-dep package that is quite out of date (details sent off list previously). We’re now currently unable to install tool-shed tools that users have requested. The current proposal from our infosec dept is to get all our deps from system repos etc. However the way I’m aware of implementing this for tool-shed tools, which need to run across our cluster, would be something pretty arduous like: * Clone the tool from the upstream toolshed repo * Edit the tool code to remove the package requirements * Identify and install all the requirements on the cluster as system pkgs / environment modules – with attention to versions so things work as expected * Edit the tool code so it knows to load the right environment modules / set right PATH when it runs * Install the tool into our galaxy ‘tools’ dir , not the ‘shed_tools’ * Manually add the tool to galaxy’s tool_conf.xml.main * Schedule downtime to restart galaxy * Test things out …. or we have to host our own tool shed, import tools we want from upstream, edit out the package requirements, provide the deps ourselves. These have all the headaches of merging things in when upstream shed-tools change. Just wondering if I’m missing anything? I know you can turn off ‘handle repository dependencies’ when installing a tool, but the tool still defines ‘requirements’ in its XML file and shows ‘Missing repository/tool dependencies’ in the Admin. Has anyone had any experience of dealing with this kind of situation? Many thanks! -- David Trudgian Ph.D. Computational Scientist, BioHPC Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833
________________________________________ UT Southwestern Medical Center
The future of medicine, today. ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
David,
You can have multiple versions of tool 'installed' manually (i.e. without shed).
That said, there is an option of working with the IUC to make the wanted packages to be up-to-date and secure. i am pretty sure any PRs this direction will get merged.
M.
On Wed, Apr 13, 2016 at 11:39 AM David Trudgian < David.Trudgian@utsouthwestern.edu> wrote:
Hi Martin,
Thanks for the input - yes there is going to be maintenance headache either way, private tool-shed or not. If we don't go with our own tool-shed though, am I right in thinking we lose versioning for tools? I.E. installing manually into 'tools' and tool_conf.xml I can't use <version> tags like the tool shed stuff does in shed_tool.conf. The infosec concern is mainly with libraries which are provided by tool-shed dependency package installs, not the main software package the tool wrapper is calling. With a tool shed it's feasible then we could keep older versions of tools/wrappers for reproducibility, rebuilding them against updated (but compatible) libraries when there's a security issue, perhaps?
Glad to know that my idea of the workflow isn't totally wrong though.
Thanks,
DT
-- David Trudgian Ph.D. Computational Scientist, BioHPC Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833
From: Martin Čech [mailto:marten@bx.psu.edu] Sent: Wednesday, April 13, 2016 8:47 AM To: David Trudgian David.Trudgian@UTSouthwestern.edu; galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Tool shed tools, manual dep installation
Hi David,
one thing that you might be missing is that the wrapper for the infosec-approved version might not exist yet (you would have to adapt the older shed-wrapped version if there were any relevant changes). But this largely depends on what software you want to use.
Besides that your flow seems quite accurate. I would probably argue against own tool shed, as the maintenance overhead will probably not be worth it for you.
Let us know if we can help in any way,
thank you for using Galaxy.
Martin
On Tue, Apr 12, 2016 at 5:42 PM David Trudgian < David.Trudgian@utsouthwestern.edu> wrote: Hi All,
I’m currently caught between the requests of our Galaxy users to install things from main tool-shed, and our information security dept’s concerns r.e. the automated installation of tool-deps on our systems. Due to restrictive web access policies for servers here our galaxy server can’t access SourceForge, where many tool-dep downloads are. A request to unblock this for a particular tool-dep package led to our infosec justifiably raising concerns r.e. a tool-dep package that is quite out of date (details sent off list previously). We’re now currently unable to install tool-shed tools that users have requested.
The current proposal from our infosec dept is to get all our deps from system repos etc. However the way I’m aware of implementing this for tool-shed tools, which need to run across our cluster, would be something pretty arduous like:
- Clone the tool from the upstream toolshed repo
- Edit the tool code to remove the package requirements
- Identify and install all the requirements on the cluster as system pkgs
/ environment modules – with attention to versions so things work as expected
- Edit the tool code so it knows to load the right environment modules /
set right PATH when it runs
- Install the tool into our galaxy ‘tools’ dir , not the ‘shed_tools’
- Manually add the tool to galaxy’s tool_conf.xml.main
- Schedule downtime to restart galaxy
- Test things out
…. or we have to host our own tool shed, import tools we want from upstream, edit out the package requirements, provide the deps ourselves. These have all the headaches of merging things in when upstream shed-tools change.
Just wondering if I’m missing anything? I know you can turn off ‘handle repository dependencies’ when installing a tool, but the tool still defines ‘requirements’ in its XML file and shows ‘Missing repository/tool dependencies’ in the Admin. Has anyone had any experience of dealing with this kind of situation?
Many thanks!
-- David Trudgian Ph.D. Computational Scientist, BioHPC Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833
UT Southwestern Medical Center
The future of medicine, today. ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi David, in the server I am administering I don't use the Tool Shed dependency resolver and is not much work if you are used to environment modules. For more information, see:
https://docs.galaxyproject.org/en/master/admin/dependency_resolvers.html
Cheers, Nicola
On 13/04/16 16:42, Martin Čech wrote:
David,
You can have multiple versions of tool 'installed' manually (i.e. without shed).
That said, there is an option of working with the IUC to make the wanted packages to be up-to-date and secure. i am pretty sure any PRs this direction will get merged.
M.
On Wed, Apr 13, 2016 at 11:39 AM David Trudgian <David.Trudgian@utsouthwestern.edu mailto:David.Trudgian@utsouthwestern.edu> wrote:
Hi Martin, Thanks for the input - yes there is going to be maintenance headache either way, private tool-shed or not. If we don't go with our own tool-shed though, am I right in thinking we lose versioning for tools? I.E. installing manually into 'tools' and tool_conf.xml I can't use <version> tags like the tool shed stuff does in shed_tool.conf. The infosec concern is mainly with libraries which are provided by tool-shed dependency package installs, not the main software package the tool wrapper is calling. With a tool shed it's feasible then we could keep older versions of tools/wrappers for reproducibility, rebuilding them against updated (but compatible) libraries when there's a security issue, perhaps? Glad to know that my idea of the workflow isn't totally wrong though. Thanks, DT -- David Trudgian Ph.D. Computational Scientist, BioHPC Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833 From: Martin Čech [mailto:marten@bx.psu.edu <mailto:marten@bx.psu.edu>] Sent: Wednesday, April 13, 2016 8:47 AM To: David Trudgian <David.Trudgian@UTSouthwestern.edu>; galaxy-dev@lists.galaxyproject.org <mailto:galaxy-dev@lists.galaxyproject.org> Subject: Re: [galaxy-dev] Tool shed tools, manual dep installation Hi David, one thing that you might be missing is that the wrapper for the infosec-approved version might not exist yet (you would have to adapt the older shed-wrapped version if there were any relevant changes). But this largely depends on what software you want to use. Besides that your flow seems quite accurate. I would probably argue against own tool shed, as the maintenance overhead will probably not be worth it for you. Let us know if we can help in any way, thank you for using Galaxy. Martin On Tue, Apr 12, 2016 at 5:42 PM David Trudgian <David.Trudgian@utsouthwestern.edu <mailto:David.Trudgian@utsouthwestern.edu>> wrote: Hi All, I’m currently caught between the requests of our Galaxy users to install things from main tool-shed, and our information security dept’s concerns r.e. the automated installation of tool-deps on our systems. Due to restrictive web access policies for servers here our galaxy server can’t access SourceForge, where many tool-dep downloads are. A request to unblock this for a particular tool-dep package led to our infosec justifiably raising concerns r.e. a tool-dep package that is quite out of date (details sent off list previously). We’re now currently unable to install tool-shed tools that users have requested. The current proposal from our infosec dept is to get all our deps from system repos etc. However the way I’m aware of implementing this for tool-shed tools, which need to run across our cluster, would be something pretty arduous like: * Clone the tool from the upstream toolshed repo * Edit the tool code to remove the package requirements * Identify and install all the requirements on the cluster as system pkgs / environment modules – with attention to versions so things work as expected * Edit the tool code so it knows to load the right environment modules / set right PATH when it runs * Install the tool into our galaxy ‘tools’ dir , not the ‘shed_tools’ * Manually add the tool to galaxy’s tool_conf.xml.main * Schedule downtime to restart galaxy * Test things out …. or we have to host our own tool shed, import tools we want from upstream, edit out the package requirements, provide the deps ourselves. These have all the headaches of merging things in when upstream shed-tools change. Just wondering if I’m missing anything? I know you can turn off ‘handle repository dependencies’ when installing a tool, but the tool still defines ‘requirements’ in its XML file and shows ‘Missing repository/tool dependencies’ in the Admin. Has anyone had any experience of dealing with this kind of situation? Many thanks! -- David Trudgian Ph.D. Computational Scientist, BioHPC Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833 ________________________________________ UT Southwestern Medical Center The future of medicine, today. ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Nicola,
Thanks! I completely missed that. That’s very useful to know.
Martin – yes a PR is on the to-do list, but unfortunately it’s a low-ish priority as the wish is to be using system / module deps. Have an impending vacation, so am having to get through a lot of stuff right now. Hopefully eventually we can get back to tool shed deps.
DT
-- David Trudgian Ph.D. Computational Scientist, BioHPC Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833
From: Nicola Soranzo [mailto:nicola.soranzo@gmail.com] On Behalf Of Nicola Soranzo Sent: Wednesday, April 13, 2016 11:03 AM To: Martin Čech marten@bx.psu.edu; David Trudgian David.Trudgian@UTSouthwestern.edu; galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Tool shed tools, manual dep installation
Hi David, in the server I am administering I don't use the Tool Shed dependency resolver and is not much work if you are used to environment modules. For more information, see:
https://docs.galaxyproject.org/en/master/admin/dependency_resolvers.html
Cheers, Nicola On 13/04/16 16:42, Martin Čech wrote: David,
You can have multiple versions of tool 'installed' manually (i.e. without shed).
That said, there is an option of working with the IUC to make the wanted packages to be up-to-date and secure. i am pretty sure any PRs this direction will get merged.
M.
On Wed, Apr 13, 2016 at 11:39 AM David Trudgian <David.Trudgian@utsouthwestern.edumailto:David.Trudgian@utsouthwestern.edu> wrote: Hi Martin,
Thanks for the input - yes there is going to be maintenance headache either way, private tool-shed or not. If we don't go with our own tool-shed though, am I right in thinking we lose versioning for tools? I.E. installing manually into 'tools' and tool_conf.xml I can't use <version> tags like the tool shed stuff does in shed_tool.conf. The infosec concern is mainly with libraries which are provided by tool-shed dependency package installs, not the main software package the tool wrapper is calling. With a tool shed it's feasible then we could keep older versions of tools/wrappers for reproducibility, rebuilding them against updated (but compatible) libraries when there's a security issue, perhaps?
Glad to know that my idea of the workflow isn't totally wrong though.
Thanks,
DT
-- David Trudgian Ph.D. Computational Scientist, BioHPC Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833
From: Martin Čech [mailto:marten@bx.psu.edumailto:marten@bx.psu.edu] Sent: Wednesday, April 13, 2016 8:47 AM To: David Trudgian David.Trudgian@UTSouthwestern.edumailto:David.Trudgian@UTSouthwestern.edu; galaxy-dev@lists.galaxyproject.orgmailto:galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Tool shed tools, manual dep installation
Hi David,
one thing that you might be missing is that the wrapper for the infosec-approved version might not exist yet (you would have to adapt the older shed-wrapped version if there were any relevant changes). But this largely depends on what software you want to use.
Besides that your flow seems quite accurate. I would probably argue against own tool shed, as the maintenance overhead will probably not be worth it for you.
Let us know if we can help in any way,
thank you for using Galaxy.
Martin
On Tue, Apr 12, 2016 at 5:42 PM David Trudgian <David.Trudgian@utsouthwestern.edumailto:David.Trudgian@utsouthwestern.edu> wrote: Hi All,
I’m currently caught between the requests of our Galaxy users to install things from main tool-shed, and our information security dept’s concerns r.e. the automated installation of tool-deps on our systems. Due to restrictive web access policies for servers here our galaxy server can’t access SourceForge, where many tool-dep downloads are. A request to unblock this for a particular tool-dep package led to our infosec justifiably raising concerns r.e. a tool-dep package that is quite out of date (details sent off list previously). We’re now currently unable to install tool-shed tools that users have requested.
The current proposal from our infosec dept is to get all our deps from system repos etc. However the way I’m aware of implementing this for tool-shed tools, which need to run across our cluster, would be something pretty arduous like:
* Clone the tool from the upstream toolshed repo * Edit the tool code to remove the package requirements * Identify and install all the requirements on the cluster as system pkgs / environment modules – with attention to versions so things work as expected * Edit the tool code so it knows to load the right environment modules / set right PATH when it runs * Install the tool into our galaxy ‘tools’ dir , not the ‘shed_tools’ * Manually add the tool to galaxy’s tool_conf.xml.main * Schedule downtime to restart galaxy * Test things out
…. or we have to host our own tool shed, import tools we want from upstream, edit out the package requirements, provide the deps ourselves. These have all the headaches of merging things in when upstream shed-tools change.
Just wondering if I’m missing anything? I know you can turn off ‘handle repository dependencies’ when installing a tool, but the tool still defines ‘requirements’ in its XML file and shows ‘Missing repository/tool dependencies’ in the Admin. Has anyone had any experience of dealing with this kind of situation?
Many thanks!
-- David Trudgian Ph.D. Computational Scientist, BioHPC Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833
________________________________________ UT Southwestern Medical Center
The future of medicine, today. ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at:
Nicola,
Just a note that the resolver setup worked nicely to have manual tool-deps for those that concern infosec – and can now use the tools we need.
Thanks again for the pointer to that!
DT
-- David Trudgian Ph.D. Computational Scientist, BioHPC Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833
From: David Trudgian Sent: Wednesday, April 13, 2016 11:12 AM To: 'Nicola Soranzo' nsoranzo@tiscali.it; Martin Čech marten@bx.psu.edu; galaxy-dev@lists.galaxyproject.org Subject: RE: [galaxy-dev] Tool shed tools, manual dep installation
Nicola,
Thanks! I completely missed that. That’s very useful to know.
Martin – yes a PR is on the to-do list, but unfortunately it’s a low-ish priority as the wish is to be using system / module deps. Have an impending vacation, so am having to get through a lot of stuff right now. Hopefully eventually we can get back to tool shed deps.
DT
-- David Trudgian Ph.D. Computational Scientist, BioHPC Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833
From: Nicola Soranzo [mailto:nicola.soranzo@gmail.com] On Behalf Of Nicola Soranzo Sent: Wednesday, April 13, 2016 11:03 AM To: Martin Čech <marten@bx.psu.edumailto:marten@bx.psu.edu>; David Trudgian <David.Trudgian@UTSouthwestern.edumailto:David.Trudgian@UTSouthwestern.edu>; galaxy-dev@lists.galaxyproject.orgmailto:galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Tool shed tools, manual dep installation
Hi David, in the server I am administering I don't use the Tool Shed dependency resolver and is not much work if you are used to environment modules. For more information, see:
https://docs.galaxyproject.org/en/master/admin/dependency_resolvers.html
Cheers, Nicola On 13/04/16 16:42, Martin Čech wrote: David,
You can have multiple versions of tool 'installed' manually (i.e. without shed).
That said, there is an option of working with the IUC to make the wanted packages to be up-to-date and secure. i am pretty sure any PRs this direction will get merged.
M.
On Wed, Apr 13, 2016 at 11:39 AM David Trudgian <David.Trudgian@utsouthwestern.edumailto:David.Trudgian@utsouthwestern.edu> wrote: Hi Martin,
Thanks for the input - yes there is going to be maintenance headache either way, private tool-shed or not. If we don't go with our own tool-shed though, am I right in thinking we lose versioning for tools? I.E. installing manually into 'tools' and tool_conf.xml I can't use <version> tags like the tool shed stuff does in shed_tool.conf. The infosec concern is mainly with libraries which are provided by tool-shed dependency package installs, not the main software package the tool wrapper is calling. With a tool shed it's feasible then we could keep older versions of tools/wrappers for reproducibility, rebuilding them against updated (but compatible) libraries when there's a security issue, perhaps?
Glad to know that my idea of the workflow isn't totally wrong though.
Thanks,
DT
-- David Trudgian Ph.D. Computational Scientist, BioHPC Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833
From: Martin Čech [mailto:marten@bx.psu.edumailto:marten@bx.psu.edu] Sent: Wednesday, April 13, 2016 8:47 AM To: David Trudgian David.Trudgian@UTSouthwestern.edumailto:David.Trudgian@UTSouthwestern.edu; galaxy-dev@lists.galaxyproject.orgmailto:galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Tool shed tools, manual dep installation
Hi David,
one thing that you might be missing is that the wrapper for the infosec-approved version might not exist yet (you would have to adapt the older shed-wrapped version if there were any relevant changes). But this largely depends on what software you want to use.
Besides that your flow seems quite accurate. I would probably argue against own tool shed, as the maintenance overhead will probably not be worth it for you.
Let us know if we can help in any way,
thank you for using Galaxy.
Martin
On Tue, Apr 12, 2016 at 5:42 PM David Trudgian <David.Trudgian@utsouthwestern.edumailto:David.Trudgian@utsouthwestern.edu> wrote: Hi All,
I’m currently caught between the requests of our Galaxy users to install things from main tool-shed, and our information security dept’s concerns r.e. the automated installation of tool-deps on our systems. Due to restrictive web access policies for servers here our galaxy server can’t access SourceForge, where many tool-dep downloads are. A request to unblock this for a particular tool-dep package led to our infosec justifiably raising concerns r.e. a tool-dep package that is quite out of date (details sent off list previously). We’re now currently unable to install tool-shed tools that users have requested.
The current proposal from our infosec dept is to get all our deps from system repos etc. However the way I’m aware of implementing this for tool-shed tools, which need to run across our cluster, would be something pretty arduous like:
* Clone the tool from the upstream toolshed repo * Edit the tool code to remove the package requirements * Identify and install all the requirements on the cluster as system pkgs / environment modules – with attention to versions so things work as expected * Edit the tool code so it knows to load the right environment modules / set right PATH when it runs * Install the tool into our galaxy ‘tools’ dir , not the ‘shed_tools’ * Manually add the tool to galaxy’s tool_conf.xml.main * Schedule downtime to restart galaxy * Test things out
…. or we have to host our own tool shed, import tools we want from upstream, edit out the package requirements, provide the deps ourselves. These have all the headaches of merging things in when upstream shed-tools change.
Just wondering if I’m missing anything? I know you can turn off ‘handle repository dependencies’ when installing a tool, but the tool still defines ‘requirements’ in its XML file and shows ‘Missing repository/tool dependencies’ in the Admin. Has anyone had any experience of dealing with this kind of situation?
Many thanks!
-- David Trudgian Ph.D. Computational Scientist, BioHPC Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833
________________________________________ UT Southwestern Medical Center
The future of medicine, today. ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at:
Anytime!
Thank you for the feedback!
Cheers, Nicola
---- David Trudgian ha scritto ----
<!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} @font-face {font-family:Consolas; panose-1:2 11 6 9 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:12.0pt; font-family:"Times New Roman",serif; color:black;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:purple; text-decoration:underline;} pre {mso-style-priority:99; mso-style-link:"HTML Preformatted Char"; margin:0in; margin-bottom:.0001pt; font-size:10.0pt; font-family:"Courier New"; color:black;} span.HTMLPreformattedChar {mso-style-name:"HTML Preformatted Char"; mso-style-priority:99; mso-style-link:"HTML Preformatted"; font-family:Consolas; color:black;} span.EmailStyle19 {mso-style-type:personal; font-family:"Calibri",sans-serif; color:#1F497D;} span.EmailStyle20 {mso-style-type:personal-reply; font-family:"Calibri",sans-serif; color:#1F497D;} .MsoChpDefault {mso-style-type:export-only; font-size:10.0pt;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} -->
Nicola,
Just a note that the resolver setup worked nicely to have manual tool-deps for those that concern infosec – and can now use the tools we need.
Thanks again for the pointer to that!
DT
-- David Trudgian Ph.D. Computational Scientist, BioHPC
Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833
From: David Trudgian Sent: Wednesday, April 13, 2016 11:12 AM To: 'Nicola Soranzo' nsoranzo@tiscali.it; Martin Čech marten@bx.psu.edu; galaxy-dev@lists.galaxyproject.org Subject: RE: [galaxy-dev] Tool shed tools, manual dep installation
Nicola,
Thanks! I completely missed that. That’s very useful to know.
Martin – yes a PR is on the to-do list, but unfortunately it’s a low-ish priority as the wish is to be using system / module deps. Have an impending vacation, so am having to get through a lot of stuff right now. Hopefully eventually we can get back to tool shed deps.
DT
-- David Trudgian Ph.D. Computational Scientist, BioHPC
Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833
From: Nicola Soranzo [mailto:nicola.soranzo@gmail.com] On Behalf Of Nicola Soranzo Sent: Wednesday, April 13, 2016 11:03 AM To: Martin Čech marten@bx.psu.edu; David Trudgian David.Trudgian@UTSouthwestern.edu; galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Tool shed tools, manual dep installation
Hi David, in the server I am administering I don't use the Tool Shed dependency resolver and is not much work if you are used to environment modules. For more information, see:
https://docs.galaxyproject.org/en/master/admin/dependency_resolvers.html
Cheers, Nicola
On 13/04/16 16:42, Martin Čech wrote:
David,
You can have multiple versions of tool 'installed' manually (i.e. without shed).
That said, there is an option of working with the IUC to make the wanted packages to be up-to-date and secure. i am pretty sure any PRs this direction will get merged.
M.
On Wed, Apr 13, 2016 at 11:39 AM David Trudgian David.Trudgian@utsouthwestern.edu wrote:
Hi Martin,
Thanks for the input - yes there is going to be maintenance headache either way, private tool-shed or not. If we don't go with our own tool-shed though, am I right in thinking we lose versioning for tools? I.E. installing manually into 'tools' and tool_conf.xml I can't use <version> tags like the tool shed stuff does in shed_tool.conf. The infosec concern is mainly with libraries which are provided by tool-shed dependency package installs, not the main software package the tool wrapper is calling. With a tool shed it's feasible then we could keep older versions of tools/wrappers for reproducibility, rebuilding them against updated (but compatible) libraries when there's a security issue, perhaps?
Glad to know that my idea of the workflow isn't totally wrong though.
Thanks,
DT
-- David Trudgian Ph.D. Computational Scientist, BioHPC Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833
From: Martin Čech [mailto:marten@bx.psu.edu] Sent: Wednesday, April 13, 2016 8:47 AM To: David Trudgian David.Trudgian@UTSouthwestern.edu; galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Tool shed tools, manual dep installation
Hi David,
one thing that you might be missing is that the wrapper for the infosec-approved version might not exist yet (you would have to adapt the older shed-wrapped version if there were any relevant changes). But this largely depends on what software you want to use.
Besides that your flow seems quite accurate. I would probably argue against own tool shed, as the maintenance overhead will probably not be worth it for you.
Let us know if we can help in any way,
thank you for using Galaxy.
Martin
On Tue, Apr 12, 2016 at 5:42 PM David Trudgian David.Trudgian@utsouthwestern.edu wrote: Hi All, I’m currently caught between the requests of our Galaxy users to install things from main tool-shed, and our information security dept’s concerns r.e. the automated installation of tool-deps on our systems. Due to restrictive web access policies for servers here our galaxy server can’t access SourceForge, where many tool-dep downloads are. A request to unblock this for a particular tool-dep package led to our infosec justifiably raising concerns r.e. a tool-dep package that is quite out of date (details sent off list previously). We’re now currently unable to install tool-shed tools that users have requested. The current proposal from our infosec dept is to get all our deps from system repos etc. However the way I’m aware of implementing this for tool-shed tools, which need to run across our cluster, would be something pretty arduous like:
- Clone the tool from the upstream toolshed repo
- Edit the tool code to remove the package requirements
- Identify and install all the requirements on the cluster as system pkgs / environment modules – with attention to versions so things work as expected
- Edit the tool code so it knows to load the right environment modules / set right PATH when it runs
- Install the tool into our galaxy ‘tools’ dir , not the ‘shed_tools’
- Manually add the tool to galaxy’s tool_conf.xml.main
- Schedule downtime to restart galaxy
- Test things out
…. or we have to host our own tool shed, import tools we want from upstream, edit out the package requirements, provide the deps ourselves. These have all the headaches of merging things in when upstream shed-tools change. Just wondering if I’m missing anything? I know you can turn off ‘handle repository dependencies’ when installing a tool, but the tool still defines ‘requirements’ in its XML file and shows ‘Missing repository/tool dependencies’ in the Admin. Has anyone had any experience of dealing with this kind of situation? Many thanks! -- David Trudgian Ph.D. Computational Scientist, BioHPC Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833
UT Southwestern Medical Center
The future of medicine, today. ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at:
galaxy-dev@lists.galaxyproject.org
-
Bjoern Gruening -
David Trudgian -
Martin Čech -
Nicola Soranzo