Dear John,
thank you very much your help and the warnings!
I'm testing it with a second database and so far it seems to work :-).
I was also suspecting the encoding to be the issue, that's why I did it through pg_dump and specified the encoding of the new server for the dump. The error stayed the same and so far I have just seen it in the context of toolshed operations. I don't know what else I could do to ensure proper encoding.
Best regards, Sarah
----- Original Message ----- From: "John Chilton" jmchilton@gmail.com To: "Sarah Diehl" diehl@ie-freiburg.mpg.de Cc: "galaxy-dev@lists.bx.psu.edu List" galaxy-dev@lists.bx.psu.edu Sent: Thursday, October 16, 2014 3:17:39 PM Subject: Re: [galaxy-dev] Help with Galaxy server migration
Hmm.... hopefully someone more knowledgeable about the tool shed than me responds also but I had a couple quick thoughts.
The first is a warning - workflows may break. At very least workflows that depend on the previous instance having gone through tool migrations instead of tool install. My understanding is tool migrations cause links to the older versions of the tools to be created so workflows continue to operate - just installing the same version of the tool again doesn't set this up. Also - in theory if you resinstall the same versions of tools tool re-runs and stuff should still work - but I am not confident and I have not tested it myself - so if that functionality is important to you, you test it out before throwing out the old configuration.
Next - I am a little bit concerned about this error. It would seem to point to some sort of encoding problem with the way the database was transferred - different table encodings or something? I don't really know anything about encoding and postgres though.
That said I am not surprised there are problems with the tool shed. There are all sorts of implicit dependencies between what is on disk and what is in the database. If you really do want to start fresh I would clear out all of the tool shed tables before continuing. These tables including - tool_shed_repository,repository_repository_dependency_association, repository_dependency, tool_dependency, tool_version, tool_version_association, migrate_tools.
Rather than truncating the existing tables - you could also just have Galaxy target a completely separate database for tool shed install stuff by creating a new postgres database, setting install_database_connection in your galaxy ini file (either universe_wsgi.ini or galaxy.ini depending on what version of Bjoern's docker image you are targetting) - or in the newest version of that Dockerfile you can just make sure your docker run includes a -e "GALAXY_CONFIG_INSTALL_DATABASE_CONNECTION=<postgresconnectionstring>".
You will also want to make sure that database gets populated - I think either of the following would work ./create_db.sh install ./migrate_db.sh install upgrade
Sorry I have not definitive answers - but hopefully this is helpful. Good luck with the migration. You should let the list know how the Docker container thing goes for a production server.
-John
On Thu, Oct 16, 2014 at 8:36 AM, Sarah Diehl diehl@ie-freiburg.mpg.de wrote:
Dear all,
I'm trying to migrate our local Galaxy server from a standalone server running on CentOS 5 to a setup that runs Galaxy in a Docker container (bgruening/galaxy-stable). Due to Docker and the setup of the container, almost all data, tools, configs etc. have to move to a different place. Since our old server is pretty cluttered up anyway, I would like to move just user data (logins, histories, datasets ...), but none of the tool installations, tool data or configs.
My plan was to start with a fresh and clean new Galaxy installation, transfer the datasets and the database and then reinstall all tools through the toolshed (of course there are many more small odds and ends that need fixing).
The problem is that after transfering the database, I always get an error when going to "Manage installed tool shed repositories" (see end of the mail). I tried just overwriting the pg_data directory (PostgreSQL versions are identical) and also exporting to sql (pg_dump) and importing in the new DB. Both times I made sure to run manage_db.sh upgrade afterwards. The error stays the same.
I can't make much sense of the error message, my only guess is that it looks for something that's not there. However, I really don't want to transfer the tool directories and tool configurations, because that will get messy and difficult, since many paths need to be adjusted. There's a high chance for error and this way I also carry over the clutter.
So is there a way to remove all tool installation information from the database, before I transfer it? Can this even be disentangled from the histories and datasets? Any help or other ideas on how to do this migration are highly appreciated!
Thanks, Sarah
10.1.5.190 - - [16/Oct/2014:09:54:35 +0000] "GET /admin_toolshed/browse_repositories HTTP/1.1" 500 - "http://deep2:8080/admin/index" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:32.0) Gecko/20100101 Firefox/32.0" Error - <type 'exceptions.UnicodeDecodeError'>: 'ascii' codec can't decode byte 0xe2 in position 393: ordinal not in range(128) URL: http://deep2:8080/admin_toolshed/browse_repositories File 'lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/galaxy-central/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/galaxy-central/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File 'lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File 'lib/galaxy/web/framework/base.py', line 190 in handle_request body = method( trans, **kwargs ) File 'lib/galaxy/web/framework/decorators.py', line 87 in decorator return func( self, trans, *args, **kwargs ) File 'lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py', line 167 in browse_repositories return self.installed_repository_grid( trans, **kwd ) File 'lib/galaxy/web/framework/helpers/grids.py', line 306 in __call__ kwargs=kwargs ) File 'lib/galaxy/web/framework/webapp.py', line 771 in fill_template return self.fill_template_mako( filename, **kwargs ) File 'lib/galaxy/web/framework/webapp.py', line 786 in fill_template_mako return template.render( **data ) File '/galaxy-central/eggs/Mako-0.4.1-py2.7.egg/mako/template.py', line 296 in render return runtime._render(self, self.callable_, args, data) File '/galaxy-central/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line 660 in _render **_kwargs_for_callable(callable_, data)) File '/galaxy-central/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line 692 in _render_context _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) File '/galaxy-central/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line 718 in _exec_template callable_(context, *args, **kwargs) File '/galaxy-central/database/compiled_templates/base.mako.py', line 66 in render_body __M_writer(unicode(next.body())) File '/galaxy-central/database/compiled_templates/grid_base.mako.py', line 91 in render_body __M_writer(unicode(self.load())) File '/galaxy-central/database/compiled_templates/grid_base.mako.py', line 120 in render_load __M_writer(unicode( h.dumps( self.get_grid_config( embedded=embedded, insert=insert ) ) )) File '/galaxy-central/database/compiled_templates/grid_base.mako.py', line 303 in render_get_grid_config value = column.get_value( trans, grid, item ) File 'lib/tool_shed/galaxy_install/grids/admin_toolshed_grids.py', line 106 in get_value return suc.get_tool_shed_repository_status_label( trans.app, tool_shed_repository ) File 'lib/tool_shed/util/shed_util_common.py', line 829 in get_tool_shed_repository_status_label elif tool_shed_repository.tool_dependencies_being_installed: File 'lib/galaxy/model/tool_shed_install/__init__.py', line 408 in tool_dependencies_being_installed for tool_dependency in self.tool_dependencies: File '/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/attributes.py', line 168 in __get__ File '/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/attributes.py', line 453 in get File '/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/strategies.py', line 508 in _load_for_state File '/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/strategies.py', line 574 in _emit_lazyload File '/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py', line 2115 in all File '/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py', line 2341 in instances File '/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py', line 3204 in fetchall File '/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py', line 3171 in _fetchall_impl UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 393: ordinal not in range(128) ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/