Hi,
I was wondering whether it might be possible/safe to install tools and deps from the toolshed on a VM galaxy installation, and then rsync the shed_tools and tool dependencies directories to a production galaxy server?
Unfortunately, our institution has web access only through a proxy and blocks 'personal web browsing' from servers. This works on a white list - so everything is blocked unless there was considered to be a valid need when the list was originally setup, or an exception request is made and approved. We have an exception in place for the main toolshed, but various dependency downloads will fail, blocked by the proxy. Getting exceptions for each URL we find not to work is painful and slow.
The suggestion I've been given is to download things on a workstation (not subject to the blocking) and transfer them manually to the server, hence the question about whether tool directories (and XML conf files) can be synced between installs? Is there anything in the SQL DB that would prevent this working, or being safe?
Otherwise I'm installing a lot of deps manually :(
Thanks,
Dave Trudgian
________________________________
UT Southwestern
Medical Center
The future of medicine, today.
Dave,
There are a number of things to consider when doing this, but it should be possible. Things to keep in mind are, in no particular order:
The tool dependency path on the workstation should match what would be the path on the server. A substantial number of packages link their libraries with absolute paths, so any change in paths would of course break the package at runtime.
The operating system, hardware, and installed libraries on the server should at worst be a superset of those on the workstation, to avoid compiling packages on the workstation that have dependencies not present on the server.
I would recommend setting the install_database_connection option in your config/galaxy.ini file, so that you can export the database of installed repositories and tool dependencies from the workstation, and import this to the server without overwriting users, jobs, and so on.
If those hurdles are cleared, it's likely that installing and copying dependencies from one system to another might work.
Another solution, more complicated to set up but easier to use, would be to run a single install database that both systems access, and install the dependencies on the workstation, into a network filesystem that is mounted in the same path on both systems. You would still need to sync the shed_tool_conf.xml across machines, but a restart of the server would then show all the tools and their dependencies with whatever status they should have.
--Dave B.
On 03/27/2015 11:36 AM, David Trudgian wrote:
Hi,
I was wondering whether it might be possible/safe to install tools and deps from the toolshed on a VM galaxy installation, and then rsync the shed_tools and tool dependencies directories to a production galaxy server?
Unfortunately, our institution has web access only through a proxy and blocks ‘personal web browsing’ from servers. This works on a white list – so everything is blocked unless there was considered to be a valid need when the list was originally setup, or an exception request is made and approved. We have an exception in place for the main toolshed, but various dependency downloads will fail, blocked by the proxy. Getting exceptions for each URL we find not to work is painful and slow.
The suggestion I’ve been given is to download things on a workstation (not subject to the blocking) and transfer them manually to the server, hence the question about whether tool directories (and XML conf files) can be synced between installs? Is there anything in the SQL DB that would prevent this working, or being safe?
Otherwise I’m installing a lot of deps manually L
Thanks,
Dave Trudgian
UTSouthwestern
Medical Center
The future of medicine, today.
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Dave,
Many thanks for your detailed reply. I like the sound of the second solution which should be no problem to setup on our systems.
Still hoping it'll be possible to get a proxy exception for our system and avoid this altogether longer term, but good to know it should be possible for now.
Thanks again,
DT
-----Original Message----- From: Dave Bouvier [mailto:dave@bx.psu.edu] Sent: Monday, March 30, 2015 8:03 AM To: David Trudgian; galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Safe / possible to sync toolshed tools and deps from another machine?
Dave,
There are a number of things to consider when doing this, but it should be possible. Things to keep in mind are, in no particular order:
The tool dependency path on the workstation should match what would be the path on the server. A substantial number of packages link their libraries with absolute paths, so any change in paths would of course break the package at runtime.
The operating system, hardware, and installed libraries on the server should at worst be a superset of those on the workstation, to avoid compiling packages on the workstation that have dependencies not present on the server.
I would recommend setting the install_database_connection option in your config/galaxy.ini file, so that you can export the database of installed repositories and tool dependencies from the workstation, and import this to the server without overwriting users, jobs, and so on.
If those hurdles are cleared, it's likely that installing and copying dependencies from one system to another might work.
Another solution, more complicated to set up but easier to use, would be to run a single install database that both systems access, and install the dependencies on the workstation, into a network filesystem that is mounted in the same path on both systems. You would still need to sync the shed_tool_conf.xml across machines, but a restart of the server would then show all the tools and their dependencies with whatever status they should have.
--Dave B.
On 03/27/2015 11:36 AM, David Trudgian wrote:
Hi,
I was wondering whether it might be possible/safe to install tools and deps from the toolshed on a VM galaxy installation, and then rsync the shed_tools and tool dependencies directories to a production galaxy server?
Unfortunately, our institution has web access only through a proxy and blocks ‘personal web browsing’ from servers. This works on a white list – so everything is blocked unless there was considered to be a valid need when the list was originally setup, or an exception request is made and approved. We have an exception in place for the main toolshed, but various dependency downloads will fail, blocked by the proxy. Getting exceptions for each URL we find not to work is painful and slow.
The suggestion I’ve been given is to download things on a workstation (not subject to the blocking) and transfer them manually to the server, hence the question about whether tool directories (and XML conf files) can be synced between installs? Is there anything in the SQL DB that would prevent this working, or being safe?
Otherwise I’m installing a lot of deps manually L
Thanks,
Dave Trudgian
--
UTSouthwestern
Medical Center
The future of medicine, today.
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
galaxy-dev@lists.galaxyproject.org
-
Dave Bouvier -
David Trudgian