Is there a good method for moving a fully functional galaxy install from one machine to another in a different directory?
Hi Evan
Well, in theory, this is a simple task, since Galaxy mostly works with relative paths. Of course, it all depends on how sophisticated your installation is and how different the old and the new 'machine' is (eg same OS?)
So my advice is: just do it or rather try it.
- make a copy of your installation on the new hardware (at first it is sufficient to copy only the latest sub directories of database/files/)
- make a copy of your PostgreSQL database
- connect your copied installation to the PostgreSQL database copy and test, and test, and test
- pay special attention to tools you have installed via the toolshed. You might need to change some absolute paths in the 'env.sh' files located in the tool_dependencies/ directory.
Once your happy with all the tests, do an rsync on the database/ directory and connect to the original PostgreSQL database.
Regards, Hans-Rudolf
On 11/22/2016 07:52 PM, evan clark wrote:
Is there a good method for moving a fully functional galaxy install from one machine to another in a different directory? ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hello Evan, Hans-Rudolf
I'm just in the middle of doing a similar migration for our local production server, and Hans-Rudolf's advice seems sound to me. Definitely moving the core Galaxy server has been relatively straightforward.
However: for the installed tools, I'm not sure that changing the paths in the env.sh files is sufficient - in our installation the absolute paths seemed to be baked into a lot of other files under the 'tool_dependencies' directory - including things like compiled files (e.g. static and shared libraries). So for many of the tools I wouldn't feel confident that they would still work after the move.
My plan has been to reinstall each of the tools from the toolshed (i.e. uninstall via the admin interface then reinstall the same tool repository revision(s) using the API), but I don't feel able to recommend this approach either as in my testing this has also had problems - ranging from some tool revisions no longer being available, through to more serious issues (such as tool dependencies which used to work but since become broken). I figured I'd just have to knuckle down and work through each problem as I encountered it.
If anyone else has experiences with these kinds of migrations then I'm also very interested to know what worked (and what didn't)!
Best wishes
Peter
On 23/11/16 08:20, Hans-Rudolf Hotz wrote:
Hi Evan
Well, in theory, this is a simple task, since Galaxy mostly works with relative paths. Of course, it all depends on how sophisticated your installation is and how different the old and the new 'machine' is (eg same OS?)
So my advice is: just do it or rather try it.
- make a copy of your installation on the new hardware (at first it is
sufficient to copy only the latest sub directories of database/files/)
make a copy of your PostgreSQL database
connect your copied installation to the PostgreSQL database copy and
test, and test, and test
- pay special attention to tools you have installed via the toolshed.
You might need to change some absolute paths in the 'env.sh' files located in the tool_dependencies/ directory.
Once your happy with all the tests, do an rsync on the database/ directory and connect to the original PostgreSQL database.
Regards, Hans-Rudolf
On 11/22/2016 07:52 PM, evan clark wrote:
Is there a good method for moving a fully functional galaxy install from one machine to another in a different directory? ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
On Wed, Nov 23, 2016 at 10:49 AM, Peter Briggs peter.briggs@manchester.ac.uk wrote:
Hello Evan, Hans-Rudolf
I'm just in the middle of doing a similar migration for our local production server, and Hans-Rudolf's advice seems sound to me. Definitely moving the core Galaxy server has been relatively straightforward.
However: for the installed tools, I'm not sure that changing the paths in the env.sh files is sufficient - in our installation the absolute paths seemed to be baked into a lot of other files under the 'tool_dependencies' directory - including things like compiled files (e.g. static and shared libraries). So for many of the tools I wouldn't feel confident that they would still work after the move.
Yes, I've seen that with various third party tool installations (out side of Galaxy), so sadly you cannot in general move the files to a different path :(
I don't know if we can do anything about this directly in Galaxy, or even in BioConda?
My plan has been to reinstall each of the tools from the toolshed (i.e. uninstall via the admin interface then reinstall the same tool repository revision(s) using the API), but I don't feel able to recommend this approach either as in my testing this has also had problems - ranging from some tool revisions no longer being available, through to more serious issues (such as tool dependencies which used to work but since become broken). I figured I'd just have to knuckle down and work through each problem as I encountered it.
One simple reason for this is stale URLs, where Galaxy's cache can help but is another step for tool wrapper authors to do when setting up a new Galaxy dependency: https://github.com/galaxyproject/cargo-port
However, even well intentioned Tool Shed updates could also break things :(
If anyone else has experiences with these kinds of migrations then I'm also very interested to know what worked (and what didn't)!
Sorry - the closest I've been is setting up a new Galaxy server in parallel with our old server, and manually installing "missing" tools via the Tool Shed.
Peter
Hi Peter
Thanks for your comments - at least it doesn't sound like I'm missing the "perfect" way to migrate the installed tools. In the longer term moving towards bioconda dependency resolution will probably be more stable.
Also re breaking tool repos: even repos that aren't updated on the toolshed can break over time, for example if the URL for an executable moves, or if an implicit Python dependency is updated and breaks an install (these are both things that I'm seeing in my tests).
Thanks again
Best wishes
Peter
On 23/11/16 11:40, Peter Cock wrote:
On Wed, Nov 23, 2016 at 10:49 AM, Peter Briggs peter.briggs@manchester.ac.uk wrote:
Hello Evan, Hans-Rudolf
I'm just in the middle of doing a similar migration for our local production server, and Hans-Rudolf's advice seems sound to me. Definitely moving the core Galaxy server has been relatively straightforward.
However: for the installed tools, I'm not sure that changing the paths in the env.sh files is sufficient - in our installation the absolute paths seemed to be baked into a lot of other files under the 'tool_dependencies' directory - including things like compiled files (e.g. static and shared libraries). So for many of the tools I wouldn't feel confident that they would still work after the move.
Yes, I've seen that with various third party tool installations (out side of Galaxy), so sadly you cannot in general move the files to a different path :(
I don't know if we can do anything about this directly in Galaxy, or even in BioConda?
My plan has been to reinstall each of the tools from the toolshed (i.e. uninstall via the admin interface then reinstall the same tool repository revision(s) using the API), but I don't feel able to recommend this approach either as in my testing this has also had problems - ranging from some tool revisions no longer being available, through to more serious issues (such as tool dependencies which used to work but since become broken). I figured I'd just have to knuckle down and work through each problem as I encountered it.
One simple reason for this is stale URLs, where Galaxy's cache can help but is another step for tool wrapper authors to do when setting up a new Galaxy dependency: https://github.com/galaxyproject/cargo-port
However, even well intentioned Tool Shed updates could also break things :(
If anyone else has experiences with these kinds of migrations then I'm also very interested to know what worked (and what didn't)!
Sorry - the closest I've been is setting up a new Galaxy server in parallel with our old server, and manually installing "missing" tools via the Tool Shed.
Peter
I wrote something in Python a while back (which I've attached) to parse a shed_tool_conf.xml file and generate a shell script to install those tools on a different server. The shell script uses Galaxy's scripts/api/install_tool_shed_repositories.py to install the tools and ends up looking something like this:
#!/bin/sh set -eux
python ./scripts/api/install_tool_shed_repositories.py --local $GALAXY_INSTALL_URL --api $GALAXY_INSTALL_KEY --tool-deps --repository-deps --url https://toolshed.g2.bx.psu.edu --owner devteam --name fasta_to_tabular --revision 9d189d08f2ad --panel-section-id convert python ./scripts/api/install_tool_shed_repositories.py --local $GALAXY_INSTALL_URL --api $GALAXY_INSTALL_KEY --tool-deps --repository-deps --url https://toolshed.g2.bx.psu.edu --owner devteam --name tabular_to_fasta --revision 0b4e36026794 --panel-section-id convert ... etc
All you have to do is set those two environment variables appropriately and run the shell script from your Galaxy root directory. I did end up having to modify our nginx config to increase the proxy read timeout for the /api location to a much larger value (I used 3600s) to give the tool dependencies enough time to build, though.
Hope you find it useful.
Cheers,
Brian
On Wed, Nov 23, 2016 at 7:32 AM, Peter Briggs <peter.briggs@manchester.ac.uk
wrote:
Hi Peter
Thanks for your comments - at least it doesn't sound like I'm missing the "perfect" way to migrate the installed tools. In the longer term moving towards bioconda dependency resolution will probably be more stable.
Also re breaking tool repos: even repos that aren't updated on the toolshed can break over time, for example if the URL for an executable moves, or if an implicit Python dependency is updated and breaks an install (these are both things that I'm seeing in my tests).
Thanks again
Best wishes
Peter
On 23/11/16 11:40, Peter Cock wrote:
On Wed, Nov 23, 2016 at 10:49 AM, Peter Briggs peter.briggs@manchester.ac.uk wrote:
Hello Evan, Hans-Rudolf
I'm just in the middle of doing a similar migration for our local production server, and Hans-Rudolf's advice seems sound to me. Definitely moving the core Galaxy server has been relatively straightforward.
However: for the installed tools, I'm not sure that changing the paths in the env.sh files is sufficient - in our installation the absolute paths seemed to be baked into a lot of other files under the 'tool_dependencies' directory - including things like compiled files (e.g. static and shared libraries). So for many of the tools I wouldn't feel confident that they would still work after the move.
Yes, I've seen that with various third party tool installations (out side of Galaxy), so sadly you cannot in general move the files to a different path :(
I don't know if we can do anything about this directly in Galaxy, or even in BioConda?
My plan has been to reinstall each of the tools from the toolshed (i.e.
uninstall via the admin interface then reinstall the same tool repository revision(s) using the API), but I don't feel able to recommend this approach either as in my testing this has also had problems - ranging from some tool revisions no longer being available, through to more serious issues (such as tool dependencies which used to work but since become broken). I figured I'd just have to knuckle down and work through each problem as I encountered it.
One simple reason for this is stale URLs, where Galaxy's cache can help but is another step for tool wrapper authors to do when setting up a new Galaxy dependency: https://github.com/galaxyproject/cargo-port
However, even well intentioned Tool Shed updates could also break things :(
If anyone else has experiences with these kinds of migrations then I'm
also very interested to know what worked (and what didn't)!
Sorry - the closest I've been is setting up a new Galaxy server in parallel with our old server, and manually installing "missing" tools via the Tool Shed.
Peter
-- Peter Briggs peter.briggs@manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482 ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
galaxy-dev@lists.galaxyproject.org
-
Brian Claywell -
evan clark -
Hans-Rudolf Hotz -
Peter Briggs -
Peter Cock