binary data types get corrupted on upload
Hi, I am trying to create a sqlite datatype. But whenever I upload the database its corrupted. With the latest galaxy-dist checkout changeset: 8153:5dcbbdfe1087 tag: tip user: Daniel Blankenberg <dan@bx.psu.edu> date: Tue Nov 06 12:57:23 2012 -0500 summary: Allow rerun to access hidden datasets. from http://wiki.galaxyproject.org/Admin/Datatypes/Adding%20Datatypes I thought that adding subclass="True" in datatypes_conf.xml would be enough to register my new type, but it seems like I stil have to add an entry in binary.py for it to be recognized. But its still corrupted. thank you very much, ido ------ Tool: Upload File Name: hg19.sqlite Created: Dec 07, 2012 Filesize: 11.1 MB Dbkey: hg19 Format: sqlite Tool Version: Tool Standard Output: stdout Tool Standard Error: stderr Tool Exit Code: 0 ------ Input Parameter Value File Format sqlite Genome File Format sqlite Inheritance Chain hg19.sqlite ------ I tried combinations of: datatypes_conf.xml <datatype extension="sqlite" type="galaxy.datatypes.binary:SQLite" mimetype="application/octet-stream" display_in_upload="true" subclass="True"/> with/without: in lib/galaxy/datatypes/binary.py class SQLite(Binary): """SQLite file""" file_ext = "sqlite" def set_peek( self, dataset, is_multi_byte=False ): if not dataset.dataset.purged: dataset.peek = "sqlite file" dataset.blurb = data.nice_size( dataset.get_size() ) else: dataset.peek = 'file does not exist' dataset.blurb = 'file purged from disk' def display_peek( self, dataset ): try: return dataset.peek except: return "sqlite file (%s)" % ( data.nice_size( dataset.get_size() ) ) Binary.register_unsniffable_binary_ext("sqlite")
Hi Ido, Is .sqlite a common compressed file type (i.e. zip, gzip, or bz2) just with a different extension? If it is, I think upload.py is always going to try to decompress it. That that has long been a behavior I would like to be able to disable on a per file type basis. If this is the problem and the galaxy team agrees to accept a patch for address the issue, I would be happy to try to implement the fix. -John On Fri, Dec 7, 2012 at 8:39 AM, Ido Tamir <tamir@imp.ac.at> wrote:
from http://wiki.galaxyproject.org/Admin/Datatypes/Adding%20Datatypes I thought that adding subclass="True" in datatypes_conf.xml would be enough to register my new type, but it seems like I stil have to add an entry in binary.py for it to be recognized. But its still corrupted.
thank you very much, ido
------ Tool: Upload File Name: hg19.sqlite Created: Dec 07, 2012 Filesize: 11.1 MB Dbkey: hg19 Format: sqlite
The specification does not mention zip, gzip or bz2 [1] and I am pretty confident, that it does not in addition compresses its content with zip etc.. It starts with a header "SQLite format 3\000" which is visible in my file with less. ido [1]http://www.sqlite.org/fileformat.html On Dec 7, 2012, at 3:50 PM, John Chilton wrote:
Hi Ido,
Is .sqlite a common compressed file type (i.e. zip, gzip, or bz2) just with a different extension?
If it is, I think upload.py is always going to try to decompress it. That that has long been a behavior I would like to be able to disable on a per file type basis. If this is the problem and the galaxy team agrees to accept a patch for address the issue, I would be happy to try to implement the fix.
-John
On Fri, Dec 7, 2012 at 8:39 AM, Ido Tamir <tamir@imp.ac.at> wrote:
from http://wiki.galaxyproject.org/Admin/Datatypes/Adding%20Datatypes I thought that adding subclass="True" in datatypes_conf.xml would be enough to register my new type, but it seems like I stil have to add an entry in binary.py for it to be recognized. But its still corrupted.
thank you very much, ido
------ Tool: Upload File Name: hg19.sqlite Created: Dec 07, 2012 Filesize: 11.1 MB Dbkey: hg19 Format: sqlite
participants (2)
-
Ido Tamir -
John Chilton