Filter tool: how to override automatic guessing of column types ?
Hi, I have an issue with filtering using the Filter tool in the sense that the first line is considered invalid and I think it has to do with the automatic guessing of column types. The condition is like: c4 !='XXX' The value of c4 for the first line is 'NA' When the paster.log is checked one can see that the last argument for filtering.py are the column types: "str,str,int,int,int ". As you can see for whatever reason, the system thinks c4 is an int, it should be 'str'. Indeed, most values for that field look like numbers but they should be treated as string types. In an attempt to override the guessing, a custom tabular type was defined in lib/galaxy/datatypes/tabular.py as such: class CloneAdminTable( Tabular ): """ Tabular format Clone Administration """ column_names = ['Name', 'SeqInAlignment', 'Family', 'Animal/library', 'NR Cluster'] column_types = ['str','str','int','str','int'] def display_peek( self, dataset ): """Returns formated html of peek""" return Tabular.make_html_table( self, dataset, column_names=self.column_names ) But that seems not to help very much; the system still guesses the column types. Any idea how to solve this ? Thanks and regards, Marc ________________________________________ THIS E-MAIL MESSAGE IS INTENDED ONLY FOR THE USE OF THE INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL AND EXEMPT FROM DISCLOSURE. If the reader of this E-mail message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately at ablynx@ablynx.com. Thank you for your co-operation. "NANOBODY" and "NANOCLONE" are registered trademarks of Ablynx N.V. ________________________________________
Hi Marc, You can force the metadata detection to search your entire file by adding a new set_meta method to your datatype class, e.g. (untested): def set_meta( self, *args, **kwd ): kwd['max_data_lines'] = None kwd['max_guess_type_data_lines'] = None return super( CloneAdminTable, self ).set_meta( *args, **kwd ) and also define the default values for your datatype: """Add metadata elements""" MetadataElement( name="column_types", default=['str','str','int','str','int'], desc="Column types", param=metadata.ColumnTypesParameter, readonly=True, visible=False, no_value=['str','str','int','str','int'] ) MetadataElement( name="column_names", default= ['Name', 'SeqInAlignment', 'Family', 'Animal/library', 'NR Cluster'], desc="Column names", readonly=True, visible=False, optional=True, no_value=['Name', 'SeqInAlignment', 'Family', 'Animal/library', 'NR Cluster'] ) alternatively, you can rewrite the set_meta method to handle setting the metadata in anyway that you like. Thanks for using Galaxy, Dan On Feb 22, 2013, at 9:24 AM, Marc Logghe wrote:
Hi, I have an issue with filtering using the Filter tool in the sense that the first line is considered invalid and I think it has to do with the automatic guessing of column types. The condition is like: c4 !='XXX' The value of c4 for the first line is ‘NA’ When the paster.log is checked one can see that the last argument for filtering.py are the column types: "str,str,int,int,int ". As you can see for whatever reason, the system thinks c4 is an int, it should be ‘str’. Indeed, most values for that field look like numbers but they should be treated as string types.
In an attempt to override the guessing, a custom tabular type was defined in lib/galaxy/datatypes/tabular.py as such: class CloneAdminTable( Tabular ): """ Tabular format Clone Administration """ column_names = ['Name', 'SeqInAlignment', 'Family', 'Animal/library', 'NR Cluster'] column_types = ['str','str','int','str','int'] def display_peek( self, dataset ): """Returns formated html of peek""" return Tabular.make_html_table( self, dataset, column_names=self.column_names )
But that seems not to help very much; the system still guesses the column types.
Any idea how to solve this ? Thanks and regards, Marc ________________________________________ THIS E-MAIL MESSAGE IS INTENDED ONLY FOR THE USE OF THE INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL AND EXEMPT FROM DISCLOSURE. If the reader of this E-mail message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately at ablynx@ablynx.com. Thank you for your co-operation. "NANOBODY" and "NANOCLONE" are registered trademarks of Ablynx N.V. ________________________________________ ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Hi Daniel, Thanks for the answer. In my hands it does not work unfortunately but I'm a total python dummy. Marc From: Daniel Blankenberg [mailto:dan@bx.psu.edu] Sent: Friday, February 22, 2013 4:10 PM To: Marc Logghe Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Filter tool: how to override automatic guessing of column types ? Hi Marc, You can force the metadata detection to search your entire file by adding a new set_meta method to your datatype class, e.g. (untested): def set_meta( self, *args, **kwd ): kwd['max_data_lines'] = None kwd['max_guess_type_data_lines'] = None return super( CloneAdminTable, self ).set_meta( *args, **kwd ) and also define the default values for your datatype: """Add metadata elements""" MetadataElement( name="column_types", default=['str','str','int','str','int'], desc="Column types", param=metadata.ColumnTypesParameter, readonly=True, visible=False, no_value=['str','str','int','str','int'] ) MetadataElement( name="column_names", default= ['Name', 'SeqInAlignment', 'Family', 'Animal/library', 'NR Cluster'], desc="Column names", readonly=True, visible=False, optional=True, no_value=['Name', 'SeqInAlignment', 'Family', 'Animal/library', 'NR Cluster'] ) alternatively, you can rewrite the set_meta method to handle setting the metadata in anyway that you like. Thanks for using Galaxy, Dan On Feb 22, 2013, at 9:24 AM, Marc Logghe wrote: Hi, I have an issue with filtering using the Filter tool in the sense that the first line is considered invalid and I think it has to do with the automatic guessing of column types. The condition is like: c4 !='XXX' The value of c4 for the first line is 'NA' When the paster.log is checked one can see that the last argument for filtering.py are the column types: "str,str,int,int,int ". As you can see for whatever reason, the system thinks c4 is an int, it should be 'str'. Indeed, most values for that field look like numbers but they should be treated as string types. In an attempt to override the guessing, a custom tabular type was defined in lib/galaxy/datatypes/tabular.py as such: class CloneAdminTable( Tabular ): """ Tabular format Clone Administration """ column_names = ['Name', 'SeqInAlignment', 'Family', 'Animal/library', 'NR Cluster'] column_types = ['str','str','int','str','int'] def display_peek( self, dataset ): """Returns formated html of peek""" return Tabular.make_html_table( self, dataset, column_names=self.column_names ) But that seems not to help very much; the system still guesses the column types. Any idea how to solve this ? Thanks and regards, Marc ________________________________________ THIS E-MAIL MESSAGE IS INTENDED ONLY FOR THE USE OF THE INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL AND EXEMPT FROM DISCLOSURE. If the reader of this E-mail message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately at ablynx@ablynx.com<mailto:ablynx@ablynx.com>. Thank you for your co-operation. "NANOBODY" and "NANOCLONE" are registered trademarks of Ablynx N.V. ________________________________________ ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ________________________________________ THIS E-MAIL MESSAGE IS INTENDED ONLY FOR THE USE OF THE INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL AND EXEMPT FROM DISCLOSURE. If the reader of this E-mail message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately at ablynx@ablynx.com. Thank you for your co-operation. "NANOBODY" and "NANOCLONE" are registered trademarks of Ablynx N.V. ________________________________________
Hi Marc, Ah, I think I see the issue here, and it has to do with an assumption being made that the first line is a undeclared header, which in your case is wrong. Can you trying adding to your set_meta: kwd['skip'] = 0 and then restart your sever and then create a new dataset with your tool? Thanks for using Galaxy, Dan On Feb 22, 2013, at 10:33 AM, Marc Logghe wrote:
Hi Daniel, Thanks for the answer. In my hands it does not work unfortunately but I’m a total python dummy.
Marc
From: Daniel Blankenberg [mailto:dan@bx.psu.edu] Sent: Friday, February 22, 2013 4:10 PM To: Marc Logghe Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Filter tool: how to override automatic guessing of column types ?
Hi Marc,
You can force the metadata detection to search your entire file by adding a new set_meta method to your datatype class, e.g. (untested):
def set_meta( self, *args, **kwd ): kwd['max_data_lines'] = None kwd['max_guess_type_data_lines'] = None return super( CloneAdminTable, self ).set_meta( *args, **kwd )
and also define the default values for your datatype:
"""Add metadata elements""" MetadataElement( name="column_types", default=['str','str','int','str','int'], desc="Column types", param=metadata.ColumnTypesParameter, readonly=True, visible=False, no_value=['str','str','int','str','int'] ) MetadataElement( name="column_names", default= ['Name', 'SeqInAlignment', 'Family', 'Animal/library', 'NR Cluster'], desc="Column names", readonly=True, visible=False, optional=True, no_value=['Name', 'SeqInAlignment', 'Family', 'Animal/library', 'NR Cluster'] )
alternatively, you can rewrite the set_meta method to handle setting the metadata in anyway that you like.
Thanks for using Galaxy,
Dan
On Feb 22, 2013, at 9:24 AM, Marc Logghe wrote:
Hi, I have an issue with filtering using the Filter tool in the sense that the first line is considered invalid and I think it has to do with the automatic guessing of column types. The condition is like: c4 !='XXX' The value of c4 for the first line is ‘NA’ When the paster.log is checked one can see that the last argument for filtering.py are the column types: "str,str,int,int,int ". As you can see for whatever reason, the system thinks c4 is an int, it should be ‘str’. Indeed, most values for that field look like numbers but they should be treated as string types.
In an attempt to override the guessing, a custom tabular type was defined in lib/galaxy/datatypes/tabular.py as such: class CloneAdminTable( Tabular ): """ Tabular format Clone Administration """ column_names = ['Name', 'SeqInAlignment', 'Family', 'Animal/library', 'NR Cluster'] column_types = ['str','str','int','str','int'] def display_peek( self, dataset ): """Returns formated html of peek""" return Tabular.make_html_table( self, dataset, column_names=self.column_names )
But that seems not to help very much; the system still guesses the column types.
Any idea how to solve this ? Thanks and regards, Marc ________________________________________ THIS E-MAIL MESSAGE IS INTENDED ONLY FOR THE USE OF THE INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL AND EXEMPT FROM DISCLOSURE. If the reader of this E-mail message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately at ablynx@ablynx.com. Thank you for your co-operation. "NANOBODY" and "NANOCLONE" are registered trademarks of Ablynx N.V. ________________________________________ ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
________________________________________ THIS E-MAIL MESSAGE IS INTENDED ONLY FOR THE USE OF THE INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL AND EXEMPT FROM DISCLOSURE. If the reader of this E-mail message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately at ablynx@ablynx.com. Thank you for your co-operation. "NANOBODY" and "NANOCLONE" are registered trademarks of Ablynx N.V. ________________________________________
From: Daniel Blankenberg [mailto:dan@bx.psu.edu] Sent: Friday, February 22, 2013 4:44 PM To: Marc Logghe Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Filter tool: how to override automatic guessing of column types ? Hi Marc, Ah, I think I see the issue here, and it has to do with an assumption being made that the first line is a undeclared header, which in your case is wrong. Can you trying adding to your set_meta: kwd['skip'] = 0 and then restart your sever and then create a new dataset with your tool? OK, got it to work. Guess I combined both of your suggestions in a way. It ended up looking like this: def set_meta( self, dataset, overwrite = True, skip = None, max_data_lines = None, max_guess_type_data_lines = None, **kwd ): dataset.metadata.columns = 5 dataset.metadata.column_types = ['str','str','int','str','int'] It might have worked after all in one of the previous attempts but I did not create a new dataset. I did like you suggested and now it works. Thanks a lot, you saved my day ! Regards, Marc ________________________________________ THIS E-MAIL MESSAGE IS INTENDED ONLY FOR THE USE OF THE INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL AND EXEMPT FROM DISCLOSURE. If the reader of this E-mail message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately at ablynx@ablynx.com. Thank you for your co-operation. "NANOBODY" and "NANOCLONE" are registered trademarks of Ablynx N.V. ________________________________________
participants (2)
-
Daniel Blankenberg -
Marc Logghe