Unicode in tool stderr crashing galaxy?
My optimism was short lived. I think I've hit a galaxy bug in Unicode processing. Having managed to get the deseq tool not to crash galaxy in the sqlalchemy layer, I now get a crash when clicking on the info->stderr for the tool output dataset, probably because it contains Unicode. Here's the traceback from paster.log: ---------------------------------------- Exception happened during processing of request from ('127.0.0.1', 47124) Traceback (most recent call last): File "/home/galaxy-dev/galaxy/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 1068, in process_request_in_thread self.finish_request(request, client_address) File "/usr/lib64/python2.6/SocketServer.py", line 322, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib64/python2.6/SocketServer.py", line 617, in __init__ self.handle() File "/home/galaxy-dev/galaxy/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 442, in handle BaseHTTPRequestHandler.handle(self) File "/usr/lib64/python2.6/BaseHTTPServer.py", line 329, in handle self.handle_one_request() File "/home/galaxy-dev/galaxy/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 437, in handle_one_request self.wsgi_execute() File "/home/galaxy-dev/galaxy/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 290, in wsgi_execute self.wsgi_write_chunk(chunk) File "/home/galaxy-dev/galaxy/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 150, in wsgi_write_chunk self.wfile.write(chunk) File "/usr/lib64/python2.6/socket.py", line 316, in write data = str(data) # XXX Should really reject non-string non-buffers UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position 169: ordinal not in range(128) ---------------------------------------- I don't know where to look to find the actual stderr output. Any pointers? Is this actually a galaxy bug now? cheers, Simon ======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================
Hi Simon, it's just a guess but can you try the attached patch if you have a reproducible case. It is not tested, but at least it should point you in the right direction. If its working, I will create a pull request. Cheers, Bjoern
My optimism was short lived. I think I've hit a galaxy bug in Unicode processing. Having managed to get the deseq tool not to crash galaxy in the sqlalchemy layer, I now get a crash when clicking on the info->stderr for the tool output dataset, probably because it contains Unicode. Here's the traceback from paster.log:
---------------------------------------- Exception happened during processing of request from ('127.0.0.1', 47124) Traceback (most recent call last): File "/home/galaxy-dev/galaxy/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 1068, in process_request_in_thread self.finish_request(request, client_address) File "/usr/lib64/python2.6/SocketServer.py", line 322, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib64/python2.6/SocketServer.py", line 617, in __init__ self.handle() File "/home/galaxy-dev/galaxy/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 442, in handle BaseHTTPRequestHandler.handle(self) File "/usr/lib64/python2.6/BaseHTTPServer.py", line 329, in handle self.handle_one_request() File "/home/galaxy-dev/galaxy/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 437, in handle_one_request self.wsgi_execute() File "/home/galaxy-dev/galaxy/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 290, in wsgi_execute self.wsgi_write_chunk(chunk) File "/home/galaxy-dev/galaxy/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 150, in wsgi_write_chunk self.wfile.write(chunk) File "/usr/lib64/python2.6/socket.py", line 316, in write data = str(data) # XXX Should really reject non-string non-buffers UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position 169: ordinal not in range(128) ----------------------------------------
I don't know where to look to find the actual stderr output. Any pointers?
Is this actually a galaxy bug now?
cheers, Simon
======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
At Thu, 19 Sep 2013 11:44:35 +0200, Bjoern Gruening wrote:
Hi Simon,
it's just a guess but can you try the attached patch if you have a reproducible case. It is not tested, but at least it should point you in the right direction. If its working, I will create a pull request.
Hi Bjoern, Thanks for your reply and your patch. (Sorry for the delay in responding, I have been busy with other stuff.) While your patch didn't solve my problem, it certainly pointed me in the right direction. I made a new patch based on your idea, which I attach. The idea is, stdout and stderr of each tool run is converted from possible UTF-8 to ASCII, using a new function asciify which I added to Galaxy util. I made a repository on the Galaxy test toolshed called unicode_tester, to test this. This has 2 tools which generates some Unicode output, either on stdout or stderr accordingly. Log output shows that unicode characters are being removed. What else should I do to contribute this patch into Galaxy core? cheers, Simon
My optimism was short lived. I think I've hit a galaxy bug in Unicode processing. Having managed to get the deseq tool not to crash galaxy in the sqlalchemy layer, I now get a crash when clicking on the info->stderr for the tool output dataset, probably because it contains Unicode. Here's the traceback from paster.log:
---------------------------------------- Exception happened during processing of request from ('127.0.0.1', 47124) Traceback (most recent call last): File "/home/galaxy-dev/galaxy/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 1068, in process_request_in_thread self.finish_request(request, client_address) File "/usr/lib64/python2.6/SocketServer.py", line 322, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib64/python2.6/SocketServer.py", line 617, in __init__ self.handle() File "/home/galaxy-dev/galaxy/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 442, in handle BaseHTTPRequestHandler.handle(self) File "/usr/lib64/python2.6/BaseHTTPServer.py", line 329, in handle self.handle_one_request() File "/home/galaxy-dev/galaxy/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 437, in handle_one_request self.wsgi_execute() File "/home/galaxy-dev/galaxy/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 290, in wsgi_execute self.wsgi_write_chunk(chunk) File "/home/galaxy-dev/galaxy/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 150, in wsgi_write_chunk self.wfile.write(chunk) File "/usr/lib64/python2.6/socket.py", line 316, in write data = str(data) # XXX Should really reject non-string non-buffers UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position 169: ordinal not in range(128) ----------------------------------------
I don't know where to look to find the actual stderr output. Any pointers?
Is this actually a galaxy bug now?
cheers, Simon
======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================
Simon, using your test tools I was able to reproduce your errors. Pull request 233 is my attempt to fix them with a lighter touch than just immediately converting outputs to ascii. Do these patches work on your end? Bjoern, do these patches fix your problem as well? If I don't hear anything from anyone, I will merge these changes in a few days. Thanks guys, -John On Tue, Sep 24, 2013 at 8:29 PM, Guest, Simon <Simon.Guest@agresearch.co.nz> wrote:
At Thu, 19 Sep 2013 11:44:35 +0200, Bjoern Gruening wrote:
Hi Simon,
it's just a guess but can you try the attached patch if you have a reproducible case. It is not tested, but at least it should point you in the right direction. If its working, I will create a pull request.
Hi Bjoern,
Thanks for your reply and your patch. (Sorry for the delay in responding, I have been busy with other stuff.)
While your patch didn't solve my problem, it certainly pointed me in the right direction.
I made a new patch based on your idea, which I attach. The idea is, stdout and stderr of each tool run is converted from possible UTF-8 to ASCII, using a new function asciify which I added to Galaxy util.
I made a repository on the Galaxy test toolshed called unicode_tester, to test this. This has 2 tools which generates some Unicode output, either on stdout or stderr accordingly. Log output shows that unicode characters are being removed.
What else should I do to contribute this patch into Galaxy core?
cheers, Simon
My optimism was short lived. I think I've hit a galaxy bug in Unicode processing. Having managed to get the deseq tool not to crash galaxy in the sqlalchemy layer, I now get a crash when clicking on the info->stderr for the tool output dataset, probably because it contains Unicode. Here's the traceback from paster.log:
---------------------------------------- Exception happened during processing of request from ('127.0.0.1', 47124) Traceback (most recent call last): File "/home/galaxy-dev/galaxy/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 1068, in process_request_in_thread self.finish_request(request, client_address) File "/usr/lib64/python2.6/SocketServer.py", line 322, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib64/python2.6/SocketServer.py", line 617, in __init__ self.handle() File "/home/galaxy-dev/galaxy/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 442, in handle BaseHTTPRequestHandler.handle(self) File "/usr/lib64/python2.6/BaseHTTPServer.py", line 329, in handle self.handle_one_request() File "/home/galaxy-dev/galaxy/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 437, in handle_one_request self.wsgi_execute() File "/home/galaxy-dev/galaxy/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 290, in wsgi_execute self.wsgi_write_chunk(chunk) File "/home/galaxy-dev/galaxy/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 150, in wsgi_write_chunk self.wfile.write(chunk) File "/usr/lib64/python2.6/socket.py", line 316, in write data = str(data) # XXX Should really reject non-string non-buffers UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position 169: ordinal not in range(128) ----------------------------------------
I don't know where to look to find the actual stderr output. Any pointers?
Is this actually a galaxy bug now?
cheers, Simon
======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
At Thu, 10 Oct 2013 00:31:56 -0500, John Chilton wrote:
Simon, using your test tools I was able to reproduce your errors. Pull request 233 is my attempt to fix them with a lighter touch than just immediately converting outputs to ascii. Do these patches work on your end?
Bjoern, do these patches fix your problem as well?
If I don't hear anything from anyone, I will merge these changes in a few days.
Hi John, That works nicely, and is better than what I did, thanks! cheers, Simon
participants (3)
-
Bjoern Gruening -
Guest, Simon -
John Chilton