Documentation of Galaxy API metadata objects
Hi all, I'm wondering if there is any documentation of the JSON structures returned by particular Galaxy REST API calls. Some aspects of the Galaxy API are effectively documented via bioblend's structure, and others are documented at http://galaxy.readthedocs.org/en/master/api_doc.html . However I don't know where to look to find out the actual JSON structure that should be returned by an API call. Does this exist? For instance, if I call (using bioblend) gi.histories.show_history(..., contents=True, details="all"), I will get back a list of dicts where each dict contains dataset metadata, e.g.: { u'api_type': u'file', u'create_time': u'2015-09-28T06:00:19.850421', u'data_type': u'galaxy.datatypes.sequence.FastqSanger', u'file_name': u'/mnt/galaxy/files/000/dataset_2.dat', .... I'd like to know how to find out what dictionary keys to expect for a particular API call in a particular Galaxy version. In this example, IIRC at some point the key 'file_name' changed to 'file_path', but I don't know how to determine which version of Galaxy uses which convention. I don't think this change would necessarily be reflected in bioblend, because bioblend just returns the entire JSON structure as a Python object without caring much about its contents. I also don't know how to derive this information from the Galaxy source code itself (even though I know where the API code is under webapp), so alternatively any guidance on that would be helpful! Thanks, Clare
Hi Clare, I am sorry but you seem to have found the best resources already. Namely 'what dictionary keys to expect...in a particular Galaxy version' is a hard problem with the current docs. :/ We could definitely do a better job providing example calls and responses in the docs with the code, so they are versioned together. I believe we are trying pretty hard not to break backward compatibility on both Galaxy side and BioBlend side so the change of file_name to file_path attribute you described sounds alarming. I will look into it. What I usualy do when I need to have a sample response is to actually call the endpoint using Postman Chrome extension ( https://chrome.google.com/webstore/detail/postman/fhbjgbiflinjbdggehcddcbncd...). However if you need this for multiple Galaxy versions it might be tedious to set them up. Sorry I couldn't be of more help. Thank you for using Galaxy! Martin On Wed, Sep 30, 2015 at 6:58 AM Clare Sloggett <sloc@unimelb.edu.au> wrote:
Hi all,
I'm wondering if there is any documentation of the JSON structures returned by particular Galaxy REST API calls.
Some aspects of the Galaxy API are effectively documented via bioblend's structure, and others are documented at http://galaxy.readthedocs.org/en/master/api_doc.html . However I don't know where to look to find out the actual JSON structure that should be returned by an API call. Does this exist?
For instance, if I call (using bioblend) gi.histories.show_history(..., contents=True, details="all"), I will get back a list of dicts where each dict contains dataset metadata, e.g.: { u'api_type': u'file', u'create_time': u'2015-09-28T06:00:19.850421', u'data_type': u'galaxy.datatypes.sequence.FastqSanger', u'file_name': u'/mnt/galaxy/files/000/dataset_2.dat', ....
I'd like to know how to find out what dictionary keys to expect for a particular API call in a particular Galaxy version. In this example, IIRC at some point the key 'file_name' changed to 'file_path', but I don't know how to determine which version of Galaxy uses which convention. I don't think this change would necessarily be reflected in bioblend, because bioblend just returns the entire JSON structure as a Python object without caring much about its contents.
I also don't know how to derive this information from the Galaxy source code itself (even though I know where the API code is under webapp), so alternatively any guidance on that would be helpful!
Thanks, Clare ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Thanks Martin! I could have the details wrong for the API change example I gave (in fact, checking this was part of my motivation for asking about documentation, I wasn't sure whether our local setup was normal, and I would like our scripts to be reusable on other systems). But I'm fairly certain I have successfully (and exclusively) used file_path vs file_name on different servers set up at different times. It's not a big deal, as I can make the code handle both cases. But just in the interests of knowing how to look into this stuff I thought I'd see if it's feasible to examine the Galaxy source code itself to spot these changes. I was thinking it might be relatively simple to do (at least if I'm looking for a specific attribute and just need to look for diffs in a specific file). Anyway after looking - I don't think it's relatively simple, but I think I have traced the relevant code :) Following code from lib/galaxy/webapps/galaxy/api , I think the code I'm interested in may actually be handled by various classes under galaxy.managers; in the case of histories and datasets, the HDASerialiser class in lib/galaxy/managers/hdas.py. Specifically I suspect what might be relevant here are the lines class HDASerializer( # datasets._UnflattenedMetadataDatasetAssociationSerializer, def __init__( self, app ): .... self.add_view( 'detailed', [ .... 'file_name', and class HDASerializer( # datasets._UnflattenedMetadataDatasetAssociationSerializer, def add_serializers( self ): .... self.serializers.update({ .... 'file_path' : self._remap_from( 'file_name' ), But I'm not sure what the purpose of this remapping is - possibly a deliberate API update? On 30 September 2015 at 23:23, Martin Čech <marten@bx.psu.edu> wrote:
Hi Clare,
I am sorry but you seem to have found the best resources already. Namely 'what dictionary keys to expect...in a particular Galaxy version' is a hard problem with the current docs. :/ We could definitely do a better job providing example calls and responses in the docs with the code, so they are versioned together.
I believe we are trying pretty hard not to break backward compatibility on both Galaxy side and BioBlend side so the change of file_name to file_path attribute you described sounds alarming. I will look into it.
What I usualy do when I need to have a sample response is to actually call the endpoint using Postman Chrome extension (https://chrome.google.com/webstore/detail/postman/fhbjgbiflinjbdggehcddcbncd...). However if you need this for multiple Galaxy versions it might be tedious to set them up.
Sorry I couldn't be of more help.
Thank you for using Galaxy!
Martin
On Wed, Sep 30, 2015 at 6:58 AM Clare Sloggett <sloc@unimelb.edu.au> wrote:
Hi all,
I'm wondering if there is any documentation of the JSON structures returned by particular Galaxy REST API calls.
Some aspects of the Galaxy API are effectively documented via bioblend's structure, and others are documented at http://galaxy.readthedocs.org/en/master/api_doc.html . However I don't know where to look to find out the actual JSON structure that should be returned by an API call. Does this exist?
For instance, if I call (using bioblend) gi.histories.show_history(..., contents=True, details="all"), I will get back a list of dicts where each dict contains dataset metadata, e.g.: { u'api_type': u'file', u'create_time': u'2015-09-28T06:00:19.850421', u'data_type': u'galaxy.datatypes.sequence.FastqSanger', u'file_name': u'/mnt/galaxy/files/000/dataset_2.dat', ....
I'd like to know how to find out what dictionary keys to expect for a particular API call in a particular Galaxy version. In this example, IIRC at some point the key 'file_name' changed to 'file_path', but I don't know how to determine which version of Galaxy uses which convention. I don't think this change would necessarily be reflected in bioblend, because bioblend just returns the entire JSON structure as a Python object without caring much about its contents.
I also don't know how to derive this information from the Galaxy source code itself (even though I know where the API code is under webapp), so alternatively any guidance on that would be helpful!
Thanks, Clare ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
-- Clare Sloggett Research Fellow / Bioinformatician Life Sciences Computation Centre Victorian Life Sciences Computation Initiative University of Melbourne, Parkville Campus LAB-14, 700 Swanston St, Carlton VIC 3053, Australia Ph: 03 903 53357 M: 0414 854 759
Hi, Clare When working on the serializers and managers, I incorrectly moved the file_name attribute over as file_path. In a later commit, I re-added file_name and made file_path still available. Sorry for the confusion. They are the same variable under two different keys (the 'path' _remap uses the 'name' serializer). The file_name key was restored as the default; the file_path variable is still available to clients/consumers that want/need it by with the 'keys' argument to the api endpoint. There's quite a bit of cruft in the serializers at this point making things a bit difficult to find let alone justify. Last I heard, the plan to version the API didn't have enough interested parties for the work to be done. In this case, the remapping is fairly superfluous and arguably unneeded since it was an error. It had been in the wild for a few releases and I wanted to make it available at least for a few more before removal. We'd still like to use http options for describing parameters and return values if we can do that without interfering with any features that may need it. On Thu, Oct 1, 2015 at 4:12 AM, Clare Sloggett <sloc@unimelb.edu.au> wrote:
Thanks Martin!
I could have the details wrong for the API change example I gave (in fact, checking this was part of my motivation for asking about documentation, I wasn't sure whether our local setup was normal, and I would like our scripts to be reusable on other systems). But I'm fairly certain I have successfully (and exclusively) used file_path vs file_name on different servers set up at different times.
It's not a big deal, as I can make the code handle both cases. But just in the interests of knowing how to look into this stuff I thought I'd see if it's feasible to examine the Galaxy source code itself to spot these changes. I was thinking it might be relatively simple to do (at least if I'm looking for a specific attribute and just need to look for diffs in a specific file). Anyway after looking - I don't think it's relatively simple, but I think I have traced the relevant code :)
Following code from lib/galaxy/webapps/galaxy/api , I think the code I'm interested in may actually be handled by various classes under galaxy.managers; in the case of histories and datasets, the HDASerialiser class in lib/galaxy/managers/hdas.py. Specifically I suspect what might be relevant here are the lines
class HDASerializer( # datasets._UnflattenedMetadataDatasetAssociationSerializer, def __init__( self, app ): .... self.add_view( 'detailed', [ .... 'file_name',
and
class HDASerializer( # datasets._UnflattenedMetadataDatasetAssociationSerializer, def add_serializers( self ): .... self.serializers.update({ .... 'file_path' : self._remap_from( 'file_name' ),
But I'm not sure what the purpose of this remapping is - possibly a deliberate API update?
Hi Clare,
I am sorry but you seem to have found the best resources already. Namely 'what dictionary keys to expect...in a particular Galaxy version' is a hard problem with the current docs. :/ We could definitely do a better job providing example calls and responses in the docs with the code, so they are versioned together.
I believe we are trying pretty hard not to break backward compatibility on both Galaxy side and BioBlend side so the change of file_name to file_path attribute you described sounds alarming. I will look into it.
What I usualy do when I need to have a sample response is to actually call the endpoint using Postman Chrome extension ( https://chrome.google.com/webstore/detail/postman/fhbjgbiflinjbdggehcddcbncd... ). However if you need this for multiple Galaxy versions it might be tedious to set them up.
Sorry I couldn't be of more help.
Thank you for using Galaxy!
Martin
On Wed, Sep 30, 2015 at 6:58 AM Clare Sloggett <sloc@unimelb.edu.au> wrote:
Hi all,
I'm wondering if there is any documentation of the JSON structures returned by particular Galaxy REST API calls.
Some aspects of the Galaxy API are effectively documented via bioblend's structure, and others are documented at http://galaxy.readthedocs.org/en/master/api_doc.html . However I don't know where to look to find out the actual JSON structure that should be returned by an API call. Does this exist?
For instance, if I call (using bioblend) gi.histories.show_history(..., contents=True, details="all"), I will get back a list of dicts where each dict contains dataset
On 30 September 2015 at 23:23, Martin Čech <marten@bx.psu.edu> wrote: metadata,
e.g.: { u'api_type': u'file', u'create_time': u'2015-09-28T06:00:19.850421', u'data_type': u'galaxy.datatypes.sequence.FastqSanger', u'file_name': u'/mnt/galaxy/files/000/dataset_2.dat', ....
I'd like to know how to find out what dictionary keys to expect for a particular API call in a particular Galaxy version. In this example, IIRC at some point the key 'file_name' changed to 'file_path', but I don't know how to determine which version of Galaxy uses which convention. I don't think this change would necessarily be reflected in bioblend, because bioblend just returns the entire JSON structure as a Python object without caring much about its contents.
I also don't know how to derive this information from the Galaxy source code itself (even though I know where the API code is under webapp), so alternatively any guidance on that would be helpful!
Thanks, Clare ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
--
Clare Sloggett Research Fellow / Bioinformatician Life Sciences Computation Centre Victorian Life Sciences Computation Initiative University of Melbourne, Parkville Campus LAB-14, 700 Swanston St, Carlton VIC 3053, Australia Ph: 03 903 53357 M: 0414 854 759 ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
participants (3)
-
Carl Eberhard -
Clare Sloggett -
Martin Čech