Blank history panel / Error in history API at listing contents
Hello all, I've noticed on my test setup (tracking galaxy-central) a problem with some histories not displaying. If I click on saved histories, I am show a list. Some will load (slower than I recall), but others result in a blank right hand pane - with the following error from run.sh galaxy.webapps.galaxy.api.history_contents ERROR 2013-01-16 15:24:22,726 Error in history API at listing contents: <type 'exceptions.TypeError'>, cannot concatenate 'str' and 'NoneType' objects The same error and blank pane happens if I click on the reload history icon. A screenshot is attached. This error is coming from HistoryContentsController in file lib/galaxy/webapps/galaxy/api/history_contents.py - specifically in the call to get_hda_dict. I tried modifying this for loop so that any individual transaction with an error is ignored (but a debug message logged), and I could then load the rest of the problem histories. The error from get_hda_dict is from the first line, which takes the galaxy.model.HistoryDatasetAssociation object and asks: hda_dict = hda.get_api_value( view='element' ) That takes us to lib/galaxy/model/__init__.py and the root of the problem: I am seeing entries where both name and val are None, meaning this line fails with the TypeError above: rval['metadata_' + name] = val Clearly some of my histories now have some 'bad' data in them, and currently Galaxy is failing to cope. Based on the ID numbers this has been happening for a while, so not a recent regression. Here is a patch which seems to help. Peter $ hg diff lib/galaxy/model/__init__.py diff -r 81c3b8a6a621 lib/galaxy/model/__init__.py --- a/lib/galaxy/model/__init__.py Wed Jan 16 15:14:04 2013 +0000 +++ b/lib/galaxy/model/__init__.py Wed Jan 16 16:26:28 2013 +0000 @@ -1530,7 +1530,11 @@ val = val.file_name elif isinstance( val, list ): val = ', '.join( [str(v) for v in val] ) - rval['metadata_' + name] = val + if name is None: + log.debug("get_api_value(view=%r) : name %r, val %r" % (view, name, val)) + assert val is None, "get_api_value(view=%r) : name %r, val %r" % (view, name, val) + else: + rval['metadata_' + name] = val return rval class HistoryDatasetAssociationDisplayAtAuthorization( object ):
Hello, Peter The blank panel should definitely be handled more gracefully in this situation - I'll work on that. Have you noticed though, since your patch, any particular pattern to which metadata names are turning out to equal None (some obviously missing metadata field)? Is there a particular datatype? Have you seen the assertion fail? C On Wed, Jan 16, 2013 at 11:31 AM, Peter Cock <p.j.a.cock@googlemail.com>wrote:
Hello all,
I've noticed on my test setup (tracking galaxy-central) a problem with some histories not displaying. If I click on saved histories, I am show a list. Some will load (slower than I recall), but others result in a blank right hand pane - with the following error from run.sh
galaxy.webapps.galaxy.api.history_contents ERROR 2013-01-16 15:24:22,726 Error in history API at listing contents: <type 'exceptions.TypeError'>, cannot concatenate 'str' and 'NoneType' objects
The same error and blank pane happens if I click on the reload history icon. A screenshot is attached.
This error is coming from HistoryContentsController in file lib/galaxy/webapps/galaxy/api/history_contents.py - specifically in the call to get_hda_dict. I tried modifying this for loop so that any individual transaction with an error is ignored (but a debug message logged), and I could then load the rest of the problem histories.
The error from get_hda_dict is from the first line, which takes the galaxy.model.HistoryDatasetAssociation object and asks:
hda_dict = hda.get_api_value( view='element' )
That takes us to lib/galaxy/model/__init__.py and the root of the problem: I am seeing entries where both name and val are None, meaning this line fails with the TypeError above:
rval['metadata_' + name] = val
Clearly some of my histories now have some 'bad' data in them, and currently Galaxy is failing to cope. Based on the ID numbers this has been happening for a while, so not a recent regression.
Here is a patch which seems to help.
Peter
$ hg diff lib/galaxy/model/__init__.py diff -r 81c3b8a6a621 lib/galaxy/model/__init__.py --- a/lib/galaxy/model/__init__.py Wed Jan 16 15:14:04 2013 +0000 +++ b/lib/galaxy/model/__init__.py Wed Jan 16 16:26:28 2013 +0000 @@ -1530,7 +1530,11 @@ val = val.file_name elif isinstance( val, list ): val = ', '.join( [str(v) for v in val] ) - rval['metadata_' + name] = val + if name is None: + log.debug("get_api_value(view=%r) : name %r, val %r" % (view, name, val)) + assert val is None, "get_api_value(view=%r) : name %r, val %r" % (view, name, val) + else: + rval['metadata_' + name] = val return rval
class HistoryDatasetAssociationDisplayAtAuthorization( object ):
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
On Thursday, January 17, 2013, Carl Eberhard wrote:
Hello, Peter
The blank panel should definitely be handled more gracefully in this situation - I'll work on that.
Great :)
Have you noticed though, since your patch, any particular pattern to which metadata names are turning out to equal None (some obviously missing metadata field)? Is there a particular datatype?
No, and thus far I've only had it on my development Galaxy install which has (compared to a production system) been exposed to plenty of cluster oddities and other corner cases. It is also running on SQLite (easy to reset and it is just me running jobs so contention is not normally an issue). Note that without adding more debugging or looking directly in the database there is no easy way to tell what the datasets causing this problem were, or what file type.
Have you seen the assertion fail?
C
Not yet, no. The fact the two fields were both None suggests to me sometimes an entry was not recorded properly... Regards, Peter
Hello, Peter In revision 8646:50c65739cd1a, I've changed the error handling in history_contents.py:index and the client-side code that fetches dataset info. Long form: Now when an error (like yours) occurs when fetching dataset information for one or more datasets, the method will record the error for that dataset as part of the returned list and continue trying to fetch the other requested datasets. In other words, one specific dataset in error will not cause the entire API request to fail (and break the history panel). Client-side code has been added to handle these types of errors better through the API and the initial mako page building. Now a dataset's server error will be shown on the client side as dataset in the 'error' state. Short form: Both the API and client side should handle single datasets error-ing more gracefully than they did and the history panel should be more resilient and useful during and after a server error (at least of this kind). Please let me know if you see more problems, C On Thu, Jan 17, 2013 at 5:54 PM, Peter Cock <p.j.a.cock@googlemail.com>wrote:
On Thursday, January 17, 2013, Carl Eberhard wrote:
Hello, Peter
The blank panel should definitely be handled more gracefully in this situation - I'll work on that.
Great :)
Have you noticed though, since your patch, any particular pattern to which metadata names are turning out to equal None (some obviously missing metadata field)? Is there a particular datatype?
No, and thus far I've only had it on my development Galaxy install which has (compared to a production system) been exposed to plenty of cluster oddities and other corner cases. It is also running on SQLite (easy to reset and it is just me running jobs so contention is not normally an issue).
Note that without adding more debugging or looking directly in the database there is no easy way to tell what the datasets causing this problem were, or what file type.
Have you seen the assertion fail?
C
Not yet, no. The fact the two fields were both None suggests to me sometimes an entry was not recorded properly...
Regards,
Peter
On Tue, Jan 22, 2013 at 4:43 PM, Carl Eberhard <carlfeberhard@gmail.com> wrote:
Hello, Peter
In revision 8646:50c65739cd1a, I've changed the error handling in history_contents.py:index and the client-side code that fetches dataset info.
Long form: Now when an error (like yours) occurs when fetching dataset information for one or more datasets, the method will record the error for that dataset as part of the returned list and continue trying to fetch the other requested datasets. In other words, one specific dataset in error will not cause the entire API request to fail (and break the history panel).
Client-side code has been added to handle these types of errors better through the API and the initial mako page building. Now a dataset's server error will be shown on the client side as dataset in the 'error' state.
Short form: Both the API and client side should handle single datasets error-ing more gracefully than they did and the history panel should be more resilient and useful during and after a server error (at least of this kind).
Please let me know if you see more problems, C
Will do, thanks, Peter
On Tue, Jan 22, 2013 at 5:06 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Tue, Jan 22, 2013 at 4:43 PM, Carl Eberhard <carlfeberhard@gmail.com> wrote:
Hello, Peter
In revision 8646:50c65739cd1a, I've changed the error handling in history_contents.py:index and the client-side code that fetches dataset info.
Long form: Now when an error (like yours) occurs when fetching dataset information for one or more datasets, the method will record the error for that dataset as part of the returned list and continue trying to fetch the other requested datasets. In other words, one specific dataset in error will not cause the entire API request to fail (and break the history panel).
Client-side code has been added to handle these types of errors better through the API and the initial mako page building. Now a dataset's server error will be shown on the client side as dataset in the 'error' state.
Short form: Both the API and client side should handle single datasets error-ing more gracefully than they did and the history panel should be more resilient and useful during and after a server error (at least of this kind).
Please let me know if you see more problems, C
Will do, thanks,
Peter
OK, having tried the new code (merged into my tools branch), the histories with these problem entries now show up in the right hand pane - the only hint of a problem is in the logs, e.g. galaxy.webapps.galaxy.api.history_contents ERROR 2013-01-23 16:38:52,802 Error in history API at listing contents with history ebfb8f50c6abde6d, hda 8c59bbd49fe7aae1: cannot concatenate 'str' and 'NoneType' objects This is consistent with the original exception (which my patch caught and logged) where both the name and val where None. So we don't know how these None values got into the database, but the user interface handles this gracefully now :) Thanks, Peter
On Tue, Jan 22, 2013 at 4:43 PM, Carl Eberhard <carlfeberhard@gmail.com> wrote:
Short form: Both the API and client side should handle single datasets error-ing more gracefully than they did and the history panel should be more resilient and useful during and after a server error (at least of this kind).
Please let me know if you see more problems, C
I'm seeing some (apparently harmless) errors in the output during the 'upload' step when running Galaxy unit tests - in this case for one of my tools: functional_tests.py INFO 2013-01-25 10:37:19,810 Functional tests will be run against localhost:9500 nose.plugins.manager DEBUG 2013-01-25 10:37:19,865 DefaultPluginManager load plugin sqlalchemy = sqlalchemy.test.noseplugin:NoseSQLAlchemy nose.plugins.manager DEBUG 2013-01-25 10:37:19,986 DefaultPluginManager load plugin nosetestdiff = nosetestdiff.plugin:NoseTestDiff nose.plugins.manager DEBUG 2013-01-25 10:37:19,989 DefaultPluginManager load plugin nosehtml = nosehtml.plugin:NoseHTML TMHMM 2.0 ( tmhmm2 ) > Test-1 ... galaxy.web.framework DEBUG 2013-01-25 10:37:20,366 Error: this request returned None from get_history(): http://localhost:9500/ galaxy.web.framework DEBUG 2013-01-25 10:37:20,425 Error: this request returned None from get_history(): http://localhost:9500/ galaxy.web.framework DEBUG 2013-01-25 10:37:20,676 Error: this request returned None from get_history(): http://localhost:9500/user/logout galaxy.web.framework DEBUG 2013-01-25 10:37:20,731 Error: this request returned None from get_history(): http://localhost:9500/ galaxy.tools.actions.upload_common INFO 2013-01-25 10:37:24,312 tool upload1 created job id 1 galaxy.jobs.manager DEBUG 2013-01-25 10:37:27,460 (1) Job assigned to handler 'main' galaxy.jobs DEBUG 2013-01-25 10:37:32,755 (1) Working directory for job is: /mnt/galaxy/galaxy-central/database/job_working_directory/000/1 galaxy.jobs.handler DEBUG 2013-01-25 10:37:32,755 dispatching job 1 to local runner galaxy.jobs.handler INFO 2013-01-25 10:37:33,041 (1) Job dispatched galaxy.jobs.runners.local DEBUG 2013-01-25 10:37:33,153 Local runner: starting job 1 galaxy.jobs.runners.local DEBUG 2013-01-25 10:37:33,712 executing: ... Are these lines just false positives? Error: this request returned None from get_history(): http://localhost:9500/ Error: this request returned None from get_history(): http://localhost:9500/user/logout Error: this request returned None from get_history(): http://localhost:9500/ Thanks, Peter
They likely shouldn't say 'Error' and do increase the noise to signal in the logs. The web transactions have a convenience function for getting the most current history for that transaction's user (get_history). If I understand correctly, these messages occur the transaction can't get or create a history ( when no user is currently logged in - or other situations where such as web crawlers). You can also see these messages in the day-to-day logs of your server, local instance, or the Galaxy main or test servers. As far as I know, they're harmless. C On Fri, Jan 25, 2013 at 5:41 AM, Peter Cock <p.j.a.cock@googlemail.com>wrote:
On Tue, Jan 22, 2013 at 4:43 PM, Carl Eberhard <carlfeberhard@gmail.com> wrote:
Short form: Both the API and client side should handle single datasets error-ing more gracefully than they did and the history panel should be more resilient and useful during and after a server error (at least of this kind).
Please let me know if you see more problems, C
I'm seeing some (apparently harmless) errors in the output during the 'upload' step when running Galaxy unit tests - in this case for one of my tools:
functional_tests.py INFO 2013-01-25 10:37:19,810 Functional tests will be run against localhost:9500 nose.plugins.manager DEBUG 2013-01-25 10:37:19,865 DefaultPluginManager load plugin sqlalchemy = sqlalchemy.test.noseplugin:NoseSQLAlchemy nose.plugins.manager DEBUG 2013-01-25 10:37:19,986 DefaultPluginManager load plugin nosetestdiff = nosetestdiff.plugin:NoseTestDiff nose.plugins.manager DEBUG 2013-01-25 10:37:19,989 DefaultPluginManager load plugin nosehtml = nosehtml.plugin:NoseHTML TMHMM 2.0 ( tmhmm2 ) > Test-1 ... galaxy.web.framework DEBUG 2013-01-25 10:37:20,366 Error: this request returned None from get_history(): http://localhost:9500/ galaxy.web.framework DEBUG 2013-01-25 10:37:20,425 Error: this request returned None from get_history(): http://localhost:9500/ galaxy.web.framework DEBUG 2013-01-25 10:37:20,676 Error: this request returned None from get_history(): http://localhost:9500/user/logout galaxy.web.framework DEBUG 2013-01-25 10:37:20,731 Error: this request returned None from get_history(): http://localhost:9500/ galaxy.tools.actions.upload_common INFO 2013-01-25 10:37:24,312 tool upload1 created job id 1 galaxy.jobs.manager DEBUG 2013-01-25 10:37:27,460 (1) Job assigned to handler 'main' galaxy.jobs DEBUG 2013-01-25 10:37:32,755 (1) Working directory for job is: /mnt/galaxy/galaxy-central/database/job_working_directory/000/1 galaxy.jobs.handler DEBUG 2013-01-25 10:37:32,755 dispatching job 1 to local runner galaxy.jobs.handler INFO 2013-01-25 10:37:33,041 (1) Job dispatched galaxy.jobs.runners.local DEBUG 2013-01-25 10:37:33,153 Local runner: starting job 1 galaxy.jobs.runners.local DEBUG 2013-01-25 10:37:33,712 executing: ...
Are these lines just false positives? Error: this request returned None from get_history(): http://localhost:9500/ Error: this request returned None from get_history(): http://localhost:9500/user/logout Error: this request returned None from get_history(): http://localhost:9500/
Thanks,
Peter
participants (2)
-
Carl Eberhard -
Peter Cock