I'm looking at extending the metadata fields for one of the supported file types. The files can get VERY large, and since I'm creating those files, I'd like to save as metadata some of the information I have on the contents.
Specifically, I'd like to tag the files with information about the sample as well as the byte offsets in the file for every millionth record (to facilitate fast file splitting). I saw the sample script for uploading a file to a library via the API. It looks to me like the most direct approach would be to modify LibrariesController.create() to handle arbitrary key-value pairs to be set on the data file.
However, that might not support things like having metadata values be arrays.
So, does my understanding of the problem and the areas involved seem reasonable? Is there something already out there that does what I need? I saw the set_metadata.py script, but that looks like it is meant to operate directly on the files, which might not make a running instance of Galaxy too happy. Is there an API version of that (or something in one of the web controllers I can repurpose to my needs)?
Thanks!
John Duddy Illumina, Inc.
Duddy, John wrote:
I'm looking at extending the metadata fields for one of the supported file types. The files can get VERY large, and since I'm creating those files, I'd like to save as metadata some of the information I have on the contents.
Specifically, I'd like to tag the files with information about the sample as well as the byte offsets in the file for every millionth record (to facilitate fast file splitting). I saw the sample script for uploading a file to a library via the API. It looks to me like the most direct approach would be to modify LibrariesController.create() to handle arbitrary key-value pairs to be set on the data file.
However, that might not support things like having metadata values be arrays.
So, does my understanding of the problem and the areas involved seem reasonable? Is there something already out there that does what I need? I saw the set_metadata.py script, but that looks like it is meant to operate directly on the files, which might not make a running instance of Galaxy too happy. Is there an API version of that (or something in one of the web controllers I can repurpose to my needs)?
Hi John,
Galaxy will still set its own required metadata elements (determined by the file type) on upload, but if you create additional optional metadata elements and don't set them in your datatype's set_meta(), you could then set them after the job runs with a call to the API. Editing metadata via the API is not yet implemented, but shouldn't be too difficult to add.
This does mean you'd have to get the newly created dataset ID and make second call to the API. If that's not feasible it could probably be incorporated at creation time, but would be trickier since the upload code doesn't account for the possibility that metadata exists prior to running the upload tool itself.
--nate
Thanks!
John Duddy Illumina, Inc.
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
So, to summarize: 1 - I can upload files via the API now, which returns the id 2 - If I don't upload via directory paths, the file upload will not be instant, so I'll have to use a poller to find out when the file upload is done 3 - I can write a new API to set metadata on a dataset and call that when the file is done. If I don't, the only metadata file file will have is the stuff created during set_meta().
Right?
John Duddy Sr. Staff Software Engineer Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858-736-3584 E-mail: jduddy@illumina.com
-----Original Message----- From: Nate Coraor [mailto:nate@bx.psu.edu] Sent: Friday, May 06, 2011 8:12 AM To: Duddy, John Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Question on setting metadata on upload via API
Duddy, John wrote:
I'm looking at extending the metadata fields for one of the supported file types. The files can get VERY large, and since I'm creating those files, I'd like to save as metadata some of the information I have on the contents.
Specifically, I'd like to tag the files with information about the sample as well as the byte offsets in the file for every millionth record (to facilitate fast file splitting). I saw the sample script for uploading a file to a library via the API. It looks to me like the most direct approach would be to modify LibrariesController.create() to handle arbitrary key-value pairs to be set on the data file.
However, that might not support things like having metadata values be arrays.
So, does my understanding of the problem and the areas involved seem reasonable? Is there something already out there that does what I need? I saw the set_metadata.py script, but that looks like it is meant to operate directly on the files, which might not make a running instance of Galaxy too happy. Is there an API version of that (or something in one of the web controllers I can repurpose to my needs)?
Hi John,
Galaxy will still set its own required metadata elements (determined by the file type) on upload, but if you create additional optional metadata elements and don't set them in your datatype's set_meta(), you could then set them after the job runs with a call to the API. Editing metadata via the API is not yet implemented, but shouldn't be too difficult to add.
This does mean you'd have to get the newly created dataset ID and make second call to the API. If that's not feasible it could probably be incorporated at creation time, but would be trickier since the upload code doesn't account for the possibility that metadata exists prior to running the upload tool itself.
--nate
Thanks!
John Duddy Illumina, Inc.
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Duddy, John wrote:
So, to summarize: 1 - I can upload files via the API now, which returns the id 2 - If I don't upload via directory paths, the file upload will not be instant, so I'll have to use a poller to find out when the file upload is done 3 - I can write a new API to set metadata on a dataset and call that when the file is done. If I don't, the only metadata file file will have is the stuff created during set_meta().
Right?
I haven't worked on the API since it was originally written, so I am not sure whether direct upload via the API was ever written/tested. You can upload via filesystem paths or the server directory methods, though.
None of the methods will be instant - once the upload request is made, a dataset is created, but the upload will still need to run through the upload tool and set required metadata.
3 is right.
--nate
John Duddy Sr. Staff Software Engineer Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858-736-3584 E-mail: jduddy@illumina.com
-----Original Message----- From: Nate Coraor [mailto:nate@bx.psu.edu] Sent: Friday, May 06, 2011 8:12 AM To: Duddy, John Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Question on setting metadata on upload via API
Duddy, John wrote:
I'm looking at extending the metadata fields for one of the supported file types. The files can get VERY large, and since I'm creating those files, I'd like to save as metadata some of the information I have on the contents.
Specifically, I'd like to tag the files with information about the sample as well as the byte offsets in the file for every millionth record (to facilitate fast file splitting). I saw the sample script for uploading a file to a library via the API. It looks to me like the most direct approach would be to modify LibrariesController.create() to handle arbitrary key-value pairs to be set on the data file.
However, that might not support things like having metadata values be arrays.
So, does my understanding of the problem and the areas involved seem reasonable? Is there something already out there that does what I need? I saw the set_metadata.py script, but that looks like it is meant to operate directly on the files, which might not make a running instance of Galaxy too happy. Is there an API version of that (or something in one of the web controllers I can repurpose to my needs)?
Hi John,
Galaxy will still set its own required metadata elements (determined by the file type) on upload, but if you create additional optional metadata elements and don't set them in your datatype's set_meta(), you could then set them after the job runs with a call to the API. Editing metadata via the API is not yet implemented, but shouldn't be too difficult to add.
This does mean you'd have to get the newly created dataset ID and make second call to the API. If that's not feasible it could probably be incorporated at creation time, but would be trickier since the upload code doesn't account for the possibility that metadata exists prior to running the upload tool itself.
--nate
Thanks!
John Duddy Illumina, Inc.
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
galaxy-dev@lists.galaxyproject.org
-
Duddy, John -
Nate Coraor