Instructions for migrating users, configuration and data to other Galaxy machine
Hi all, I am looking for instructions to migrate users, configuration and data to other Galaxy machine. I have a production Galaxy, with users, histories, pages, data libraries, configured tool panel, running on a postgresdb. I want to move this information to a fresh Galaxy install (same version as production) on another machine. Is this feasible to do without much hassle? I am thinking about a merge/sync of configuration files and dumping and creating the postgresdb,... Has anyone experience with this and a check list for this? The idea is that the user should have no clue that Galaxy is on a new machine: all data, tools, histories, etc... are there. Thanks for your consideration. Kind regards, Joachim -- Joachim Jacob, PhD Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib
Hi Joachim This is roughly what we did ~15 months ago: 1. make a copy of the MySQL DB (postgresdb in your case) 2. copy the complete galaxy directory to the new server (make sure you keep the path) 3. point the new galaxy server to the DB copy and start it (different port number) -> due to the higher Python version, news eggs were downloaded -> all python code was re-compiled 4. test the new server (while the old one is still in use) 5. stop the old server 6. rsync ~/galaxy_dist/database/files/ 7. point the new galaxy server to the 'live' DB and re-start it Obviously, this is won't be a 'fresh' Galaxy install. If you want to start from a new download of the galaxy distribution, the amount of work (eg: merging of configuration files) depends on the existing modifications you have made. If you have only a few, the copy/merge of the: universe_wsgi.ini and tool_conf.xml files plus a copy of the database/ and tools/ directory might be sufficient...or you need to look into the tool-data/ directory, the datatypes_conf.xml file, etc, etc....but you can test all this while the new server is already running in parallel to the current server. Regards, Hans On 06/12/2012 01:14 PM, Joachim Jacob wrote:
Hi all,
I am looking for instructions to migrate users, configuration and data to other Galaxy machine. I have a production Galaxy, with users, histories, pages, data libraries, configured tool panel, running on a postgresdb. I want to move this information to a fresh Galaxy install (same version as production) on another machine.
Is this feasible to do without much hassle? I am thinking about a merge/sync of configuration files and dumping and creating the postgresdb,... Has anyone experience with this and a check list for this?
The idea is that the user should have no clue that Galaxy is on a new machine: all data, tools, histories, etc... are there. Thanks for your consideration.
Kind regards, Joachim
Hi Hans-Rudolf, Thank you for sharing your experience! To summarize: 1. user information is stored in the database (histories, pages, workflows). This info is linked to the datasets, 2. all datasets are in the '../database/files' (or the location specified in universe_wsgi.ini) directory, and 3. all configuration files are located in the root of Galaxy. So would it be feasible to run the postgresdb not locally, but on a separate machine, to which I connect the freshly installed Galaxy? Then I can also keep the datasets directory on a network share, which is mounted to the fresh install. [perhaps other directories can also be accessed over the network, such as the toolshed tools directory]. The configuration files is perhaps the most difficult part. these have to be merged. But perhaps I could use mercurial for this? Seems a powerful way to do this. Joachim. Joachim Jacob, PhD Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib On 06/12/2012 02:29 PM, Hans-Rudolf Hotz wrote:
Hi Joachim
This is roughly what we did ~15 months ago:
1. make a copy of the MySQL DB (postgresdb in your case)
2. copy the complete galaxy directory to the new server (make sure you keep the path)
3. point the new galaxy server to the DB copy and start it (different port number) -> due to the higher Python version, news eggs were downloaded -> all python code was re-compiled
4. test the new server (while the old one is still in use)
5. stop the old server
6. rsync ~/galaxy_dist/database/files/
7. point the new galaxy server to the 'live' DB and re-start it
Obviously, this is won't be a 'fresh' Galaxy install.
If you want to start from a new download of the galaxy distribution, the amount of work (eg: merging of configuration files) depends on the existing modifications you have made. If you have only a few, the copy/merge of the: universe_wsgi.ini and tool_conf.xml files plus a copy of the database/ and tools/ directory might be sufficient...or you need to look into the tool-data/ directory, the datatypes_conf.xml file, etc, etc....but you can test all this while the new server is already running in parallel to the current server.
Regards, Hans
On 06/12/2012 01:14 PM, Joachim Jacob wrote:
Hi all,
I am looking for instructions to migrate users, configuration and data to other Galaxy machine. I have a production Galaxy, with users, histories, pages, data libraries, configured tool panel, running on a postgresdb. I want to move this information to a fresh Galaxy install (same version as production) on another machine.
Is this feasible to do without much hassle? I am thinking about a merge/sync of configuration files and dumping and creating the postgresdb,... Has anyone experience with this and a check list for this?
The idea is that the user should have no clue that Galaxy is on a new machine: all data, tools, histories, etc... are there. Thanks for your consideration.
Kind regards, Joachim
On 06/12/2012 04:04 PM, Joachim Jacob wrote:
Hi Hans-Rudolf,
Thank you for sharing your experience!
To summarize: 1. user information is stored in the database (histories, pages, workflows). This info is linked to the datasets, 2. all datasets are in the '../database/files' (or the location specified in universe_wsgi.ini) directory, and 3. all configuration files are located in the root of Galaxy.
So would it be feasible to run the postgresdb not locally, but on a separate machine, to which I connect the freshly installed Galaxy? Then I can also keep the datasets directory on a network share, which is mounted to the fresh install. [perhaps other directories can also be accessed over the network, such as the toolshed tools directory].
No, don't share the postgresdb and the datasets. Two different galaxy installations accessing the same postgresdb and the datasets will cause troubles. Hence, I have step 6 and 7 in my procedure, to make sure all the data accumulated during the test phase gets 'replaced' with all the data the users continue to produce on the old production server.
The configuration files is perhaps the most difficult part. these have to be merged. But perhaps I could use mercurial for this? Seems a powerful way to do this.
Yes, to some extend, see the ".hgignore" file Regards, Hans
Joachim.
Joachim Jacob, PhD
Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib
On 06/12/2012 02:29 PM, Hans-Rudolf Hotz wrote:
Hi Joachim
This is roughly what we did ~15 months ago:
1. make a copy of the MySQL DB (postgresdb in your case)
2. copy the complete galaxy directory to the new server (make sure you keep the path)
3. point the new galaxy server to the DB copy and start it (different port number) -> due to the higher Python version, news eggs were downloaded -> all python code was re-compiled
4. test the new server (while the old one is still in use)
5. stop the old server
6. rsync ~/galaxy_dist/database/files/
7. point the new galaxy server to the 'live' DB and re-start it
Obviously, this is won't be a 'fresh' Galaxy install.
If you want to start from a new download of the galaxy distribution, the amount of work (eg: merging of configuration files) depends on the existing modifications you have made. If you have only a few, the copy/merge of the: universe_wsgi.ini and tool_conf.xml files plus a copy of the database/ and tools/ directory might be sufficient...or you need to look into the tool-data/ directory, the datatypes_conf.xml file, etc, etc....but you can test all this while the new server is already running in parallel to the current server.
Regards, Hans
On 06/12/2012 01:14 PM, Joachim Jacob wrote:
Hi all,
I am looking for instructions to migrate users, configuration and data to other Galaxy machine. I have a production Galaxy, with users, histories, pages, data libraries, configured tool panel, running on a postgresdb. I want to move this information to a fresh Galaxy install (same version as production) on another machine.
Is this feasible to do without much hassle? I am thinking about a merge/sync of configuration files and dumping and creating the postgresdb,... Has anyone experience with this and a check list for this?
The idea is that the user should have no clue that Galaxy is on a new machine: all data, tools, histories, etc... are there. Thanks for your consideration.
Kind regards, Joachim
participants (2)
-
Hans-Rudolf Hotz
-
Joachim Jacob