I'm getting an exception when trying to look at the admin job management screen. It looks like SQLAlchemy doesn't like non-ascii characters. Any ideas about what to do? Error - <type 'exceptions.UnicodeDecodeError'>: 'ascii' codec can't decode byte 0x8b in position 404: ordinal not in range(128) URL: http://pk.kilokluster.ucsc.edu:8079/admin/jobs File '/inside/depot4/galaxy/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/inside/depot4/galaxy/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/inside/depot4/galaxy/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/inside/depot4/galaxy/lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File '/inside/depot4/galaxy/lib/galaxy/web/framework/base.py', line 190 in handle_request body = method( trans, **kwargs ) File '/inside/depot4/galaxy/lib/galaxy/web/framework/__init__.py', line 221 in decorator return func( self, trans, *args, **kwargs ) File '/inside/depot4/galaxy/lib/galaxy/web/base/controllers/admin.py', line 1053 in jobs for job in jobs: File '/inside/depot4/galaxy/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/query.py', line 2341 in instances File '/inside/depot4/galaxy/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py', line 3204 in fetchall File '/inside/depot4/galaxy/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py', line 3171 in _fetchall_impl UnicodeDecodeError: 'ascii' codec can't decode byte 0x8b in position 404: ordinal not in range(128)
This also took out my job handlers (exception below). So the introduction of non-ascii characters into the table (via the job stdout/stderr capture) can make a galaxy instance pretty useless. I was able to find the offending records using "SELECT count(*) FROM job WHERE stderr similar to '%\x8b%';" Turns out it was the byproduct of doing a path paste of some gzipped fastq files (so they where never decompressed, just passed along), and then fastq_groomer.py complains about the file having an invalid header (line 22?) reports back an invalid header and prints it out, thus the non-ascii characters in the stderr. I 'cleaned' the database with 'update job set stderr = regexp_replace(stderr, '\x8b', '\x5f');'. But there should probably be some safe guards put in place to stop this from happening. Kyle galaxy.jobs.handler INFO 2013-07-06 12:35:32,033 job handler stop queue started Traceback (most recent call last): File "/inside/depot4/galaxy/lib/galaxy/webapps/galaxy/buildapp.py", line 35, in app_factory app = UniverseApplication( global_conf = global_conf, **kwargs ) File "/inside/depot4/galaxy/lib/galaxy/app.py", line 164, in __init__ self.job_manager = manager.JobManager( self ) File "/inside/depot4/galaxy/lib/galaxy/jobs/manager.py", line 36, in __init__ self.job_handler.start() File "/inside/depot4/galaxy/lib/galaxy/jobs/handler.py", line 34, in start self.job_queue.start() File "/inside/depot4/galaxy/lib/galaxy/jobs/handler.py", line 77, in start self.__check_jobs_at_startup() File "/inside/depot4/galaxy/lib/galaxy/jobs/handler.py", line 92, in __check_jobs_at_startup & ( model.Job.handler == self.app.config.server_name ) ): File "/inside/depot4/galaxy/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/query.py", line 2341, in instances fetch = cursor.fetchall() File "/inside/depot4/galaxy/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py", line 3204, in fetchall l = self.process_rows(self._fetchall_impl()) File "/inside/depot4/galaxy/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py", line 3171, in _fetchall_impl return self.cursor.fetchall() UnicodeDecodeError: 'ascii' codec can't decode byte 0x8b in position 404: ordinal not in range(128) On Fri, Jul 5, 2013 at 11:43 PM, Kyle Ellrott <kellrott@soe.ucsc.edu> wrote:
I'm getting an exception when trying to look at the admin job management screen. It looks like SQLAlchemy doesn't like non-ascii characters. Any ideas about what to do?
Error - <type 'exceptions.UnicodeDecodeError'>: 'ascii' codec can't decode byte 0x8b in position 404: ordinal not in range(128) URL: http://pk.kilokluster.ucsc.edu:8079/admin/jobs File '/inside/depot4/galaxy/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/inside/depot4/galaxy/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/inside/depot4/galaxy/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/inside/depot4/galaxy/lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File '/inside/depot4/galaxy/lib/galaxy/web/framework/base.py', line 190 in handle_request body = method( trans, **kwargs ) File '/inside/depot4/galaxy/lib/galaxy/web/framework/__init__.py', line 221 in decorator return func( self, trans, *args, **kwargs ) File '/inside/depot4/galaxy/lib/galaxy/web/base/controllers/admin.py', line 1053 in jobs for job in jobs: File '/inside/depot4/galaxy/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/query.py', line 2341 in instances File '/inside/depot4/galaxy/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py', line 3204 in fetchall File '/inside/depot4/galaxy/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py', line 3171 in _fetchall_impl UnicodeDecodeError: 'ascii' codec can't decode byte 0x8b in position 404: ordinal not in range(128)
That's a great point. I've made a bug card for it here: https://trello.com/c/LsZoPtwv/1053-jobs-better-unicode-handling-in-stderr Thanks for reporting this. Carl On Sat, Jul 6, 2013 at 11:33 PM, Kyle Ellrott <kellrott@soe.ucsc.edu> wrote:
This also took out my job handlers (exception below). So the introduction of non-ascii characters into the table (via the job stdout/stderr capture) can make a galaxy instance pretty useless.
I was able to find the offending records using "SELECT count(*) FROM job WHERE stderr similar to '%\x8b%';" Turns out it was the byproduct of doing a path paste of some gzipped fastq files (so they where never decompressed, just passed along), and then fastq_groomer.py complains about the file having an invalid header (line 22?) reports back an invalid header and prints it out, thus the non-ascii characters in the stderr.
I 'cleaned' the database with 'update job set stderr = regexp_replace(stderr, '\x8b', '\x5f');'. But there should probably be some safe guards put in place to stop this from happening.
Kyle
galaxy.jobs.handler INFO 2013-07-06 12:35:32,033 job handler stop queue started Traceback (most recent call last): File "/inside/depot4/galaxy/lib/galaxy/webapps/galaxy/buildapp.py", line 35, in app_factory app = UniverseApplication( global_conf = global_conf, **kwargs ) File "/inside/depot4/galaxy/lib/galaxy/app.py", line 164, in __init__ self.job_manager = manager.JobManager( self ) File "/inside/depot4/galaxy/lib/galaxy/jobs/manager.py", line 36, in __init__ self.job_handler.start() File "/inside/depot4/galaxy/lib/galaxy/jobs/handler.py", line 34, in start self.job_queue.start() File "/inside/depot4/galaxy/lib/galaxy/jobs/handler.py", line 77, in start self.__check_jobs_at_startup() File "/inside/depot4/galaxy/lib/galaxy/jobs/handler.py", line 92, in __check_jobs_at_startup & ( model.Job.handler == self.app.config.server_name ) ): File "/inside/depot4/galaxy/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/query.py", line 2341, in instances fetch = cursor.fetchall() File "/inside/depot4/galaxy/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py", line 3204, in fetchall l = self.process_rows(self._fetchall_impl()) File "/inside/depot4/galaxy/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py", line 3171, in _fetchall_impl return self.cursor.fetchall() UnicodeDecodeError: 'ascii' codec can't decode byte 0x8b in position 404: ordinal not in range(128)
On Fri, Jul 5, 2013 at 11:43 PM, Kyle Ellrott <kellrott@soe.ucsc.edu>wrote:
I'm getting an exception when trying to look at the admin job management screen. It looks like SQLAlchemy doesn't like non-ascii characters. Any ideas about what to do?
Error - <type 'exceptions.UnicodeDecodeError'>: 'ascii' codec can't decode byte 0x8b in position 404: ordinal not in range(128) URL: http://pk.kilokluster.ucsc.edu:8079/admin/jobs File '/inside/depot4/galaxy/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/inside/depot4/galaxy/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/inside/depot4/galaxy/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/inside/depot4/galaxy/lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File '/inside/depot4/galaxy/lib/galaxy/web/framework/base.py', line 190 in handle_request body = method( trans, **kwargs ) File '/inside/depot4/galaxy/lib/galaxy/web/framework/__init__.py', line 221 in decorator return func( self, trans, *args, **kwargs ) File '/inside/depot4/galaxy/lib/galaxy/web/base/controllers/admin.py', line 1053 in jobs for job in jobs: File '/inside/depot4/galaxy/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/query.py', line 2341 in instances File '/inside/depot4/galaxy/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py', line 3204 in fetchall File '/inside/depot4/galaxy/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py', line 3171 in _fetchall_impl UnicodeDecodeError: 'ascii' codec can't decode byte 0x8b in position 404: ordinal not in range(128)
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
participants (2)
-
Carl Eberhard -
Kyle Ellrott