Hello all,

We just ran into a Tool Shed installation problem under a test installation of Galaxy v16.01, while trying to install the meme_fimo tool.

Clicking through the GUI takes me to the failed install of the perl_xml_parser_expat version 2.41 package (under package_perl_xml_parser_expat_2_41),

https://toolshed.g2.bx.psu.edu/view/iuc/package_perl_xml_parser_expat_2_41vi... https://github.com/galaxyproject/tools-iuc/blob/master/packages/package_perl...

The Admin GUI shows this long stack trace error:

File "/mnt/qnap/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py", line 123, in install_and_build_package_via_fabric tool_dependency = self.install_and_build_package( install_environment, tool_dependency, actions_dict ) File "/mnt/qnap/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py", line 81, in install_and_build_package initial_download=True ) File "/mnt/qnap/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/recipe/recipe_manager.py", line 31, in execute_step initial_download=initial_download ) File "/mnt/qnap/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py", line 1253, in execute_step job_name=package_name) File "/mnt/qnap/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/recipe/install_environment.py", line 140, in handle_command error_message=error_message) File "/mnt/qnap/galaxy/galaxy-dist/lib/tool_shed/util/tool_dependency_util.py", line 390, in set_tool_dependency_attributes sa_session.flush() File "/mnt/shared/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/sqlalchemy/orm/scoping.py", line 150, in do return getattr(self.registry(), name)(*args, **kwargs) ... File "/mnt/shared/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/MySQLdb/cursors.py", line 187, in execute query = query % tuple([db.literal(item) for item in args]) File "/mnt/shared/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/MySQLdb/connections.py", line 278, in literal return self.escape(o, self.encoders) File "/mnt/shared/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/MySQLdb/connections.py", line 208, in unicode_literal return db.literal(u.encode(unicode_literal.charset))

'latin-1' codec can't encode character u'\u2018' in position 1070: ordinal not in range(256)

This is the left single quotation mark (U+2018), which I think comes from the failed installation log:

$ tail /mnt/shared/galaxy/tool_dependency_dir/perl_xml_parser_expat/2.41/iuc/package_perl_xml_parser_expat_2_41/1ce3818a8eb9/INSTALLATION.log Expat.xs:2179: error: ‘parser’ undeclared (first use in this function) Expat.xs:2180: error: ‘CallbackVector’ has no member named ‘st_serial’ Expat.xs:2182: error: ‘CallbackVector’ has no member named ‘skip_until’ Expat.c: In function ‘XS_XML__Parser__Expat_Do_External_Parse’: Expat.c:2904: error: ‘XML_Parser’ undeclared (first use in this function) Expat.c:2904: error: expected ‘;’ before ‘parser’ Expat.xs:2194: error: ‘parser’ undeclared (first use in this function) make[1]: *** [Expat.o] Error 1 make: *** [subdirs] Error 2 #############################################

Note those fancy quote characters? Yep:

$ hexdump -C /mnt/shared/galaxy/tool_dependency_dir/perl_xml_parser_expat/2.41/iuc/package_perl_xml_parser_expat_2_41/1ce3818a8eb9/INSTALLATION.log | tail -n 15 00022e90 6f 72 3a 20 65 78 70 65 63 74 65 64 20 e2 80 98 |or: expected ...| 00022ea0 3b e2 80 99 20 62 65 66 6f 72 65 20 e2 80 98 70 |;... before ...p| 00022eb0 61 72 73 65 72 e2 80 99 0a 45 78 70 61 74 2e 78 |arser....Expat.x| 00022ec0 73 3a 32 31 39 34 3a 20 65 72 72 6f 72 3a 20 e2 |s:2194: error: .| 00022ed0 80 98 70 61 72 73 65 72 e2 80 99 20 75 6e 64 65 |..parser... unde| 00022ee0 63 6c 61 72 65 64 20 28 66 69 72 73 74 20 75 73 |clared (first us| 00022ef0 65 20 69 6e 20 74 68 69 73 20 66 75 6e 63 74 69 |e in this functi| 00022f00 6f 6e 29 0a 6d 61 6b 65 5b 31 5d 3a 20 2a 2a 2a |on).make[1]: ***| 00022f10 20 5b 45 78 70 61 74 2e 6f 5d 20 45 72 72 6f 72 | [Expat.o] Error| 00022f20 20 31 0a 6d 61 6b 65 3a 20 2a 2a 2a 20 5b 73 75 | 1.make: *** [su| 00022f30 62 64 69 72 73 5d 20 45 72 72 6f 72 20 32 0a 23 |bdirs] Error 2.#| 00022f40 23 23 23 23 23 23 23 23 23 23 23 23 23 23 23 23 |################| * 00022f60 23 23 23 23 23 23 23 23 23 23 23 23 0a |############.| 00022f6d

Notice the sequence in hex of e2 80 98, that's the UTF-8 (hex) for the problem character. We also have e2 80 99 which is the matching right single quote.

While I can go over the full INSTALLATION.log to work out what might be wrong, there is clearly a Galaxy problem with the UTF-8 encoded character. Going over the Galaxy mailing list there have been similar problems (although not ToolShed related) reported in the past (2013) with MySQL and the table encodings:

http://dev.list.galaxyproject.org/error-with-unicode-output-td4662783.html

Are there still known issues with encodings and the MySQL galaxy schema?

Are we likely to run into other corner cases with MySQL where we would be better off switching to PostgreSQL?

Regards,

Peter