Re: [galaxy-dev] Bug report : JSONDecodeError

13 Nov 2012

      Hi again, 

Sorry for the mistake in the request, but I think that now we spotted the problem more precisely. 

When I install 'mothur_toolsuite', the 'metadata' field is set with a long JSON value (length > 68,000). We noticed that when suppressing the tool, this value is truncated around a length of 65,500. Only this value was altered in the row. This field's type is BLOB. 

Maybe this BLOB type is too short for that value but I don't understand why there is no problem when the tool is installed (and the row inserted). 

Cheers, 
Mathieu 

----- Mail original -----
...
De: "Greg Von Kuster" <greg@bx.psu.edu>
À: "Mathieu Bahin" <mathieu.bahin@irisa.fr>
Cc: "Galaxy Dev" <galaxy-dev@bx.psu.edu>
Envoyé: Mardi 13 Novembre 2012 14:29:39
Objet: Re: [galaxy-dev] Bug report : JSONDecodeError
...
Hi Mathieu,
...
As I mentioned in my previous email, here is the select that is
needed. I need to see the values of all columns for the
mothur_toolsuite row.
...
select * from tool_shed_repository where name = ' mothur_toolsuite ';
...
On Nov 13, 2012, at 7:33 AM, Mathieu Bahin wrote:
...
...
Hi Greg !

...
...
Running the sql command, I got that (actually, it's
'mothur_toolsuite' so I change a bit the request but it is not
deleted):

...
...
mysql> select name from tool_shed_repository;

...
...
+--------------------+
...
| name |
...
+--------------------+
...
| mothur_toolsuite |
...
| protk |
...
| roche454_toolsuite |
...
| toolfactory |
...
+--------------------+
...
4 rows in set (0.00 sec)

...
...
The mothur entries in 'shed_tool_conf.xml' file and the mothur
repository are still existing.

...
...
Cheers,
...
Mathieu Bahin

...
...
----- Mail original -----

...
...
...
De: "Greg Von Kuster" < greg@bx.psu.edu >
...
...
À: "Mathieu Bahin" < mathieu.bahin@irisa.fr >
...
...
Cc: "Galaxy Dev" < galaxy-dev@bx.psu.edu >
...
...
Envoyé: Lundi 12 Novembre 2012 22:18:05
...
...
Objet: Re: [galaxy-dev] Bug report : JSONDecodeError

...
...
...
Hi Mathieu,

...
...
...
It's pretty difficult to tell you where to start looking within
your
environment for the cause of the problem.

...
...
...
You mentioned that you attempted to uninstall the repository name
mothur from you local Galaxy instance, and that action produced
this
problem. What is the result of running the following sql command
in
your db?

...
...
...
select * from tool_shed_repository where name = 'mothur';

...
...
...
Also, have the entries for the mothur tools been eliminated from
your
shed_tool_conf.xml file?

...
...
...
How about the install directory for the mothur repository? Has is
been removed from disk?

...
...
...
Greg Von Kuster

...
...
...
On Nov 12, 2012, at 10:57 AM, Mathieu Bahin wrote:

...
...
...
...
Hi Greg,

...
...
...
...
I kind of forgot a maybe interesting information. We found this
message at the end of the paster.log :

...
...
...
...
Exception happened during processing of request from
('127.0.0.1',
55428)
...
...
...
Traceback (most recent call last):
...
...
...
File
"/opt/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/httpserver.py",
line 1053, in process_request_in_thread
...
...
...
self.finish_request(request, client_address)
...
...
...
File "/local/python/2.7-bis/lib/python2.7/SocketServer.py",
line
323,
in finish_request
...
...
...
self.RequestHandlerClass(request, client_address, self)
...
...
...
File "/local/python/2.7-bis/lib/python2.7/SocketServer.py",
line
641,
in __init__
...
...
...
self.finish()
...
...
...
File "/local/python/2.7-bis/lib/python2.7/SocketServer.py",
line
694,
in finish
...
...
...
self.wfile.flush()
...
...
...
File "/local/python/2.7-bis/lib/python2.7/socket.py", line 301,
in
flush
...
...
...
self._sock.sendall(view[write_offset:write_offset+buffer_size])
...
...
...
error: [Errno 32] Broken pipe

...
...
...
...
Cheers,
...
...
...
Mathieu

...
...
...
...
----- Mail original -----

...
...
...
...
...
De: "Mathieu Bahin" < mathieu.bahin@irisa.fr >
...
...
...
...
À: "Greg Von Kuster" < greg@bx.psu.edu >
...
...
...
...
Cc: galaxy-dev@bx.psu.edu
...
...
...
...
Envoyé: Lundi 12 Novembre 2012 15:54:35
...
...
...
...
Objet: Re: [galaxy-dev] Bug report : JSONDecodeError

...
...
...
...
...
Hi again,

...
...
...
...
...
I got this with "hg heads" :

...
...
...
...
...
changeset: 7986:12fcd068b12e
...
...
...
...
tag: tip
...
...
...
...
user: Daniel Blankenberg < dan@bx.psu.edu >
...
...
...
...
date: Thu Oct 18 11:22:12 2012 -0400
...
...
...
...
summary: Do not hide failed datasets with HideDatasetAction
post
job
action.

...
...
...
...
...
changeset: 7152:ffc20445b35e
...
...
...
...
branch: split_blast2
...
...
...
...
parent: 7058:22f545c0d412
...
...
...
...
user: Dannon Baker < dannonbaker@me.com >
...
...
...
...
date: Wed Apr 18 07:59:38 2012 -0400
...
...
...
...
summary: Branch close.

...
...
...
...
...
changeset: 7151:c94957ac5454
...
...
...
...
branch: workflow_import_menu_api
...
...
...
...
parent: 6834:1a3d01b09f7a
...
...
...
...
user: Dannon Baker < dannonbaker@me.com >
...
...
...
...
date: Wed Mar 21 09:50:44 2012 -0400
...
...
...
...
summary: Branch close for pull request.

...
...
...
...
...
changeset: 7150:314a10ae112d
...
...
...
...
branch: split_blast
...
...
...
...
parent: 6757:ebf65c0b1e26
...
...
...
...
user: Dannon Baker < dannonbaker@me.com >
...
...
...
...
date: Tue Feb 28 12:32:54 2012 -0500
...
...
...
...
summary: Close split_blast branch

...
...
...
...
...
changeset: 7149:1327f2219423
...
...
...
...
branch: meme_requirement_on_fimo
...
...
...
...
parent: 6662:8ba74c17e809
...
...
...
...
user: Daniel Blankenberg < dan@bx.psu.edu >
...
...
...
...
date: Fri Feb 03 11:15:29 2012 -0500
...
...
...
...
summary: close branch
...
...
...
...
--------------------------------------------------------

...
...
...
...
...
And I got this with the debug at :

...
...
...
...
...
JSONDecodeError: Unterminated string starting at: line 1
column
65522
(char 65522)

...
...
...
...
...
URL:
http://dev-galaxy.genouest.org/admin_toolshed/browse_repositories
...
...
...
...
File
'/opt/galaxy-dist/eggs/WebError-0.8a-py2.7.egg/weberror/evalexception/middleware.py',
line 364 in respond
...
...
...
...
app_iter = self.application(environ, detect_start_response)
...
...
...
...
File
'/opt/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/debug/prints.py',
line 98 in __call__
...
...
...
...
environ, self.app)
...
...
...
...
File
'/opt/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/wsgilib.py',
line 539 in intercept_output
...
...
...
...
app_iter = application(environ, replacement_start_response)
...
...
...
...
File
'/opt/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/recursive.py',
line 80 in __call__
...
...
...
...
return self.application(environ, start_response)
...
...
...
...
File
'/opt/galaxy-dist/lib/galaxy/web/framework/middleware/remoteuser.py',
line 91 in __call__
...
...
...
...
return self.app( environ, start_response )
...
...
...
...
File
'/opt/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/httpexceptions.py',
line 632 in __call__
...
...
...
...
return self.application(environ, start_response)
...
...
...
...
File '/opt/galaxy-dist/lib/galaxy/web/framework/base.py',
line
160
in
__call__
...
...
...
...
body = method( trans, **kwargs )
...
...
...
...
File '/opt/galaxy-dist/lib/galaxy/web/framework/__init__.py',
line
203 in decorator
...
...
...
...
return func( self, trans, *args, **kwargs )
...
...
...
...
File
'/opt/galaxy-dist/lib/galaxy/web/controllers/admin_toolshed.py',
line 381 in browse_repositories
...
...
...
...
return self.installed_repository_grid( trans, **kwd )
...
...
...
...
File
'/opt/galaxy-dist/lib/galaxy/web/framework/helpers/grids.py',
line 277 in __call__
...
...
...
...
kwargs=kwargs )
...
...
...
...
File '/opt/galaxy-dist/lib/galaxy/web/framework/__init__.py',
line
765 in fill_template
...
...
...
...
return self.fill_template_mako( filename, **kwargs )
...
...
...
...
File '/opt/galaxy-dist/lib/galaxy/web/framework/__init__.py',
line
776 in fill_template_mako
...
...
...
...
return template.render( **data )
...
...
...
...
File
'/opt/galaxy-dist/eggs/Mako-0.4.1-py2.7.egg/mako/template.py',
line 296 in render
...
...
...
...
return runtime._render(self, self.callable_, args, data)
...
...
...
...
File
'/opt/galaxy-dist/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py',
line 660 in _render
...
...
...
...
**_kwargs_for_callable(callable_, data))
...
...
...
...
File
'/opt/galaxy-dist/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py',
line 692 in _render_context
...
...
...
...
_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
...
...
...
...
File
'/opt/galaxy-dist/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py',
line 718 in _exec_template
...
...
...
...
callable_(context, *args, **kwargs)
...
...
...
...
File
'/opt/galaxy-dist/database/compiled_templates/base.mako.py',
line 42 in render_body
...
...
...
...
__M_writer(unicode(next.body()))
...
...
...
...
File
'/opt/galaxy-dist/database/compiled_templates/grid_base.mako.py',
line 119 in render_body
...
...
...
...
__M_writer(unicode(self.grid_body( grid )))
...
...
...
...
File
'/opt/galaxy-dist/database/compiled_templates/grid_base.mako.py',
line 784 in render_grid_body
...
...
...
...
__M_writer(unicode(self.make_grid( grid )))
...
...
...
...
File
'/opt/galaxy-dist/database/compiled_templates/grid_base.mako.py',
line 813 in render_make_grid
...
...
...
...
__M_writer(unicode(self.render_grid_table( grid,
show_item_checkboxes
)))
...
...
...
...
File
'/opt/galaxy-dist/database/compiled_templates/grid_base.mako.py',
line 762 in render_render_grid_table
...
...
...
...
__M_writer(unicode(render_grid_table_body_contents( grid,
show_item_checkboxes )))
...
...
...
...
File
'/opt/galaxy-dist/database/compiled_templates/grid_base.mako.py',
line 658 in render_grid_table_body_contents
...
...
...
...
return
render_render_grid_table_body_contents(context,grid,show_item_checkboxes)
...
...
...
...
File
'/opt/galaxy-dist/database/compiled_templates/grid_base.mako.py',
line 281 in render_render_grid_table_body_contents
...
...
...
...
for i, item in enumerate( query ):
...
...
...
...
File
'/opt/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/query.py',
line 1422 in instances
...
...
...
...
rows = [process[0](context, row) for row in fetch]
...
...
...
...
File
'/opt/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/query.py',
line 2032 in main
...
...
...
...
return _instance(row, None)
...
...
...
...
File
'/opt/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/mapper.py',
line 1729 in _instance
...
...
...
...
populate_state(state, dict_, row, isnew, only_load_props)
...
...
...
...
File
'/opt/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/mapper.py',
line 1614 in populate_state
...
...
...
...
populator(state, dict_, row, isnew=isnew, **flags)
...
...
...
...
File
'/opt/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/strategies.py',
line 120 in new_execute
...
...
...
...
dict_[key] = row[col]
...
...
...
...
File
'/opt/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/engine/base.py',
line 1348 in __getitem__
...
...
...
...
return self.__parent._get_col(self.__row, key)
...
...
...
...
File
'/opt/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/engine/base.py',
line 1620 in _get_col
...
...
...
...
return processor(row[index])
...
...
...
...
File
'/opt/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/types.py',
line 284 in process
...
...
...
...
return self.process_result_value(impl_processor(value),
dialect)
...
...
...
...
File '/opt/galaxy-dist/lib/galaxy/model/custom_types.py',
line
50
in
process_result_value
...
...
...
...
return json_decoder.decode( str( _sniffnfix_pg9_hex(value) )
)
...
...
...
...
File
'/opt/galaxy-dist/eggs/simplejson-2.1.1-py2.7-linux-x86_64-ucs2.egg/simplejson/decoder.py',
line 402 in decode
...
...
...
...
File
'/opt/galaxy-dist/eggs/simplejson-2.1.1-py2.7-linux-x86_64-ucs2.egg/simplejson/decoder.py',
line 418 in raw_decode
...
...
...
...
JSONDecodeError: Unterminated string starting at: line 1
column
65522
(char 65522)
...
...
...
...
----------------------------------------------

...
...
...
...
...
I hope this can help you find the problem !

...
...
...
...
...
Thanks,
...
...
...
...
Mathieu

...
...
...
...
...
----- Mail original -----

...
...
...
...
...
...
De: "Greg Von Kuster" < greg@bx.psu.edu >
...
...
...
...
...
À: "Mathieu Bahin" < mathieu.bahin@irisa.fr >
...
...
...
...
...
Cc: galaxy-dev@bx.psu.edu
...
...
...
...
...
Envoyé: Lundi 12 Novembre 2012 12:33:10
...
...
...
...
...
Objet: Re: [galaxy-dev] Bug report : JSONDecodeError

...
...
...
...
...
...
Hi Matthiew,

...
...
...
...
...
...
Can you clarify the version of Galaxy you are using? What
is
the
result of typing "hg heads" in your Galaxy installation
directory?

...
...
...
...
...
...
ALso, I noticed the following in your paster log:

...
...
...
...
...
...
----------------------------------------
...
...
...
...
...
131.254.16.26 - - [12/Nov/2012:10:52:26 +0200] "GET
/admin_toolshed/browse_repositories HTTP/1.1" 500 - "
http://dev-galaxy.genouest.org/admin " "Mozilla/5.0 (X11;
Linux
x86_64; rv:12.0) Gecko/20100101 Firefox/12.0"
...
...
...
...
...
Debug at:
http://dev-galaxy.genouest.org/_debug/view/1352713905

...
...
...
...
...
...
Directing your browser to the URL in the "Debug at" address
will
provide a stack trace of the exception thrown due to the
problem.
What is this this complete stack trace?

...
...
...
...
...
...
Thanks,

...
...
...
...
...
...
Greg Von Kuster

...
...
...
...
...
...
On Nov 12, 2012, at 4:56 AM, Mathieu Bahin wrote:

...
...
...
...
...
...
> And here is the paster...

...
...
...
...
...
...
> ----- Mail original -----

...
...
...
...
...
...
> > De: "Mathieu Bahin" < mathieu.bahin@irisa.fr >
>
...
...
...
...
...
> > À: "Greg Von Kuster" < greg@bx.psu.edu >
>
...
...
...
...
...
> > Cc: galaxy-dev@bx.psu.edu
>
...
...
...
...
...
> > Envoyé: Lundi 12 Novembre 2012 10:54:21
>
...
...
...
...
...
> > Objet: Re: [galaxy-dev] Bug report : JSONDecodeError
>

...
...
...
...
...
...
> > Hi Greg !
>

...
...
...
...
...
...
> > Thanks for the quick answer.
>

...
...
...
...
...
...
> > I am using the 7720 parent version but I keep it
> > up-to-date
> > without
> > any problems.
>
...
...
...
...
...
> > Enclosed you'll find the 'paster.log' file generated
> > when
> > I
> > start
> > Galaxy and try to access the manage installed tool shed
> > repositories
> > page.
>

...
...
...
...
...
...
> > Cheers,
>
...
...
...
...
...
> > Mathieu Bahin
>

...
...
...
...
...
...
> > ----- Mail original -----
>

...
...
...
...
...
...
> > > De: "Greg Von Kuster" < greg@bx.psu.edu >
> > 
>
...
...
...
...
...
> > > À: "Mathieu Bahin" < mathieu.bahin@irisa.fr >
> > 
>
...
...
...
...
...
> > > Cc: galaxy-dev@bx.psu.edu
> > 
>
...
...
...
...
...
> > > Envoyé: Vendredi 9 Novembre 2012 19:28:09
> > 
>
...
...
...
...
...
> > > Objet: Re: [galaxy-dev] Bug report : JSONDecodeError
> > 
>

...
...
...
...
...
...
> > > Hello Mathieu,
> > 
>

...
...
...
...
...
...
> > > What version of Galaxy are you running? Also, I
> > > cannot
> > > read
> > > the
> > > stack
> > > trace from your pasted image, so can you send a text
> > > version
> > > of
> > > the
> > > complete stack trace from your paster log?
> > 
>

...
...
...
...
...
...
> > > Thanks,
> > 
>

...
...
...
...
...
...
> > > Greg Von Kuster
> > 
>

...
...
...
...
...
...
> > > On Nov 9, 2012, at 11:26 AM, Mathieu Bahin wrote:
> > 
>

...
...
...
...
...
...
> > > > Hi,
> > > 
> > 
>

...
...
...
...
...
...
> > > > I am using my own Galaxy instance and I had a
> > > > problem
> > > > removing
> > > > a
> > > > tool
> > > > from the tool shed : mothur.
> > > 
> > 
>
...
...
...
...
...
> > > > The installation was ok and I the mothur tool
> > > > running
> > > > without
> > > > any
> > > > problem. But when I tried to uninstall the tool, I
> > > > get
> > > > an
> > > > error
> > > > (screenshot enclosed). And I get the same every
> > > > time
> > > > I
> > > > try
> > > > to
> > > > go
> > > > on
> > > > the 'Manage tool shed repositories' in the Admin
> > > > panel.
> > > 
> > 
>

...
...
...
...
...
...
> > > > I have tried to rebuilt the eggs and to restart
> > > > Galaxy
> > > > basically.
> > > > Actually, I don't really understand the error
> > > > message.
> > > 
> > 
>

...
...
...
...
...
...
> > > > Cheers,
> > > 
> > 
>
...
...
...
...
...
> > > > Mathieu Bahin
> > > 
> > 
>
...
...
...
...
...
> > > > <07.png>
> > > > ___________________________________________________________
> > > 
> > 
>
...
...
...
...
...
> > > > Please keep all replies on the list by using "reply
> > > > all"
> > > 
> > 
>
...
...
...
...
...
> > > > in your mail client. To manage your subscriptions
> > > > to
> > > > this
> > > 
> > 
>
...
...
...
...
...
> > > > and other Galaxy lists, please use the interface
> > > > at:
> > > 
> > 
>

...
...
...
...
...
...
> > > > http://lists.bx.psu.edu/
> > > 
> > 
>
...
...
...
...
...
> <paster.log>