Tool unit-testing with "boolean" parameter
Hello, I'm trying to add a <tests> section for a tool which has a boolean/checkbox parameter. If one of the tests set the parameter's value to "false", the test throws an exception (before starting the actual job). If changed to "true", the test works. I've seen only a few tools with a boolean parameter (grouping.xml, complement.xml, concat.xml, merge.xml, phastOdds_tool.xml) and all of them only test the "true" value. I've tried setting the value to "false", to an empty string, and removing the parameter entirely, but it always throws an exception similar to: ================= Traceback (most recent call last): File "/home/gordon/temp/galaxy/test/functional/test_toolbox.py", line 134, in test_tool self.do_it( td ) File "/home/gordon/temp/galaxy/test/functional/test_toolbox.py", line 56, in do_it self.run_tool( testdef.tool.id, repeat_name=repeat_name, **page_inputs ) File "/home/gordon/temp/galaxy/test/base/twilltestcase.py", line 920, in run_tool self.submit_form( **kwd ) File "/home/gordon/temp/galaxy/test/base/twilltestcase.py", line 870, in submit_form raise AssertionError( errmsg ) AssertionError: Attempting to set field 'header' to value '['false']' in form 'tool_form' threw exception: id=None name='false' label=None control: <CheckboxControl(header=[true])> If the above control is a DataToolparameter whose data type class does not include a sniff() method, make sure to include a proper 'ftype' attribute to the tag for the control within the <test> tag set. ================= Any workarounds ? thanks, -gordon
Assaf, This is a known limitation of the test framework at the moment. Boolean's can't be tested with a 'false' value. It's not ideal, but what I resorted to doing was changing the boolean parameter to a select with 'true' and 'false' options. Kelly On Tue 16 Feb 2010, at 9:34 PM, Assaf Gordon wrote:
Hello,
I'm trying to add a <tests> section for a tool which has a boolean/ checkbox parameter. If one of the tests set the parameter's value to "false", the test throws an exception (before starting the actual job). If changed to "true", the test works.
I've seen only a few tools with a boolean parameter (grouping.xml, complement.xml, concat.xml, merge.xml, phastOdds_tool.xml) and all of them only test the "true" value.
I've tried setting the value to "false", to an empty string, and removing the parameter entirely, but it always throws an exception similar to: ================= Traceback (most recent call last): File "/home/gordon/temp/galaxy/test/functional/test_toolbox.py", line 134, in test_tool self.do_it( td ) File "/home/gordon/temp/galaxy/test/functional/test_toolbox.py", line 56, in do_it self.run_tool( testdef.tool.id, repeat_name=repeat_name, **page_inputs ) File "/home/gordon/temp/galaxy/test/base/twilltestcase.py", line 920, in run_tool self.submit_form( **kwd ) File "/home/gordon/temp/galaxy/test/base/twilltestcase.py", line 870, in submit_form raise AssertionError( errmsg ) AssertionError: Attempting to set field 'header' to value '['false']' in form 'tool_form' threw exception: id=None name='false' label=None control: <CheckboxControl(header=[true])> If the above control is a DataToolparameter whose data type class does not include a sniff() method, make sure to include a proper 'ftype' attribute to the tag for the control within the <test> tag set. =================
Any workarounds ?
thanks, -gordon _______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
Another strange but reproducible testing bug (at least on my computer): if the name of the input file is longer than 30 characters, the test will fail with a weird exception. Example: The following <test> works: ==== <test> <param name="input" value="replace_text_in_column_in1.txt" ftype="tabular" /> ... ... ==== While the following <test> will fail: ==== <test> <param name="input" value="replace_text_in_column_ink1.txt" ftype="tabular" /> ... ... === The only difference between the two is that the input file name is one letter longer ("k" added to the "ink1" part). All other parameters stay exactly the same, and both input files (with and without "k") exist. The second <test> version fails with the following exception: ======== FAIL: Replace Text ( cshl_awk_replace_in_column ) > Test-1 ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/gordon/temp/galaxy/test/functional/test_toolbox.py", line 134, in test_tool self.do_it( td ) File "/home/gordon/temp/galaxy/test/functional/test_toolbox.py", line 56, in do_it self.run_tool( testdef.tool.id, repeat_name=repeat_name, **page_inputs ) File "/home/gordon/temp/galaxy/test/base/twilltestcase.py", line 920, in run_tool self.submit_form( **kwd ) File "/home/gordon/temp/galaxy/test/base/twilltestcase.py", line 870, in submit_form raise AssertionError( errmsg ) AssertionError: Attempting to set field 'input' to value '['replace_text_in_column_ink1.txt']' in form 'tool_form' threw exception: cannot find value/label "replace_text_in_column_ink1.txt" in list control control: <SelectControl(input=[1])> If the above control is a DataToolparameter whose data type class does not include a sniff() method, make sure to include a proper 'ftype' attribute to the tag for the control within the <test> tag set. -------------------- >> begin captured stdout << --------------------- Uploaded file: replace_text_in_column_ink1.txt , ftype: tabular , extra: {'ftype': 'tabular', 'value': 'replace_text_in_column_ink1.txt'} form 'tool_form' contains the following controls ( note the values ) control 0: <HiddenControl(tool_id=cshl_awk_replace_in_column) (readonly)> control 1: <HiddenControl(tool_state=800255ed346436383633663935656435333636643238663535633132373339383564326365323733656165303a3762323236333666366337353664366532323361323032323563323233313563323232323263323032323639366537303735373432323361323032323331323232633230323235663566373036313637363535663566323233613230333032633230323237353732366335663730363137333734363532323361323032323563323234653666366536353563323232323263323032323636363936633635356636343631373436313232336132303232356332323465366636653635356332323232376471002e) (readonly)> control 2: <SelectControl(input=[*1])> control 3: <SelectControl(column=[*1, 2, 3, 4, 5, 6])> control 4: <TextControl(url_paste=)> control 5: <TextControl(file_data=)> control 6: <SubmitControl(runtool_btn=Execute) (readonly)> ======== Looks like somehow the "input" selection box values are overwritten with the single value "1" . I guess there's not much to do about this, except maybe warn and document it in the wiki. -gordon P.S. I assume you (galaxy team) didn't write the testing framework. Is there a documentation of the this framework, with possibly a list of these issues ? I'm trying to be a good Galaxy citizen and write unit tests for my tools, but these quirks waste a lot of time... Kelly Vincent wrote, On 02/16/2010 10:19 PM:
Assaf,
This is a known limitation of the test framework at the moment. Boolean's can't be tested with a 'false' value.
It's not ideal, but what I resorted to doing was changing the boolean parameter to a select with 'true' and 'false' options.
Kelly
On Tue 16 Feb 2010, at 9:34 PM, Assaf Gordon wrote:
Hello,
I'm trying to add a <tests> section for a tool which has a boolean/checkbox parameter. If one of the tests set the parameter's value to "false", the test throws an exception (before starting the actual job). If changed to "true", the test works.
I've seen only a few tools with a boolean parameter (grouping.xml, complement.xml, concat.xml, merge.xml, phastOdds_tool.xml) and all of them only test the "true" value.
I've tried setting the value to "false", to an empty string, and removing the parameter entirely, but it always throws an exception similar to: ================= Traceback (most recent call last): File "/home/gordon/temp/galaxy/test/functional/test_toolbox.py", line 134, in test_tool self.do_it( td ) File "/home/gordon/temp/galaxy/test/functional/test_toolbox.py", line 56, in do_it self.run_tool( testdef.tool.id, repeat_name=repeat_name, **page_inputs ) File "/home/gordon/temp/galaxy/test/base/twilltestcase.py", line 920, in run_tool self.submit_form( **kwd ) File "/home/gordon/temp/galaxy/test/base/twilltestcase.py", line 870, in submit_form raise AssertionError( errmsg ) AssertionError: Attempting to set field 'header' to value '['false']' in form 'tool_form' threw exception: id=None name='false' label=None control: <CheckboxControl(header=[true])> If the above control is a DataToolparameter whose data type class does not include a sniff() method, make sure to include a proper 'ftype' attribute to the tag for the control within the <test> tag set. =================
Any workarounds ?
thanks, -gordon _______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
I looked into this a bit, but could not find the cause in the twill code, which is where the problem lies. For now, just keep test file names shorter than 30 characters and I'll track down a fix if I get a chance. Greg On Feb 16, 2010, at 11:51 PM, Assaf Gordon wrote:
Another strange but reproducible testing bug (at least on my computer):
if the name of the input file is longer than 30 characters, the test will fail with a weird exception.
Example: The following <test> works: ==== <test> <param name="input" value="replace_text_in_column_in1.txt" ftype="tabular" /> ... ... ====
While the following <test> will fail: ==== <test> <param name="input" value="replace_text_in_column_ink1.txt" ftype="tabular" /> ... ... ===
The only difference between the two is that the input file name is one letter longer ("k" added to the "ink1" part). All other parameters stay exactly the same, and both input files (with and without "k") exist.
The second <test> version fails with the following exception: ======== FAIL: Replace Text ( cshl_awk_replace_in_column ) > Test-1 ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/gordon/temp/galaxy/test/functional/test_toolbox.py", line 134, in test_tool self.do_it( td ) File "/home/gordon/temp/galaxy/test/functional/test_toolbox.py", line 56, in do_it self.run_tool( testdef.tool.id, repeat_name=repeat_name, **page_inputs ) File "/home/gordon/temp/galaxy/test/base/twilltestcase.py", line 920, in run_tool self.submit_form( **kwd ) File "/home/gordon/temp/galaxy/test/base/twilltestcase.py", line 870, in submit_form raise AssertionError( errmsg ) AssertionError: Attempting to set field 'input' to value '['replace_text_in_column_ink1.txt']' in form 'tool_form' threw exception: cannot find value/label "replace_text_in_column_ink1.txt" in list control control: <SelectControl(input=[1])> If the above control is a DataToolparameter whose data type class does not include a sniff() method, make sure to include a proper 'ftype' attribute to the tag for the control within the <test> tag set.
-------------------- >> begin captured stdout << --------------------- Uploaded file: replace_text_in_column_ink1.txt , ftype: tabular , extra: {'ftype': 'tabular', 'value': 'replace_text_in_column_ink1.txt'}
form 'tool_form' contains the following controls ( note the values ) control 0: <HiddenControl(tool_id=cshl_awk_replace_in_column) (readonly)> control 1: <HiddenControl (tool_state = 800255ed346436383633663935656435333636643238663535633132373339383564326365323733656165303a3762323236333666366337353664366532323361323032323563323233313563323232323263323032323639366537303735373432323361323032323331323232633230323235663566373036313637363535663566323233613230333032633230323237353732366335663730363137333734363532323361323032323563323234653666366536353563323232323263323032323636363936633635356636343631373436313232336132303232356332323465366636653635356332323232376471002e ) (readonly)> control 2: <SelectControl(input=[*1])> control 3: <SelectControl(column=[*1, 2, 3, 4, 5, 6])> control 4: <TextControl(url_paste=)> control 5: <TextControl(file_data=)> control 6: <SubmitControl(runtool_btn=Execute) (readonly)> ========
Looks like somehow the "input" selection box values are overwritten with the single value "1" .
I guess there's not much to do about this, except maybe warn and document it in the wiki.
-gordon
P.S. I assume you (galaxy team) didn't write the testing framework. Is there a documentation of the this framework, with possibly a list of these issues ? I'm trying to be a good Galaxy citizen and write unit tests for my tools, but these quirks waste a lot of time...
Kelly Vincent wrote, On 02/16/2010 10:19 PM:
Assaf,
This is a known limitation of the test framework at the moment. Boolean's can't be tested with a 'false' value.
It's not ideal, but what I resorted to doing was changing the boolean parameter to a select with 'true' and 'false' options.
Kelly
On Tue 16 Feb 2010, at 9:34 PM, Assaf Gordon wrote:
Hello,
I'm trying to add a <tests> section for a tool which has a boolean/checkbox parameter. If one of the tests set the parameter's value to "false", the test throws an exception (before starting the actual job). If changed to "true", the test works.
I've seen only a few tools with a boolean parameter (grouping.xml, complement.xml, concat.xml, merge.xml, phastOdds_tool.xml) and all of them only test the "true" value.
I've tried setting the value to "false", to an empty string, and removing the parameter entirely, but it always throws an exception similar to: ================= Traceback (most recent call last): File "/home/gordon/temp/galaxy/test/functional/test_toolbox.py", line 134, in test_tool self.do_it( td ) File "/home/gordon/temp/galaxy/test/functional/test_toolbox.py", line 56, in do_it self.run_tool( testdef.tool.id, repeat_name=repeat_name, **page_inputs ) File "/home/gordon/temp/galaxy/test/base/twilltestcase.py", line 920, in run_tool self.submit_form( **kwd ) File "/home/gordon/temp/galaxy/test/base/twilltestcase.py", line 870, in submit_form raise AssertionError( errmsg ) AssertionError: Attempting to set field 'header' to value '['false']' in form 'tool_form' threw exception: id=None name='false' label=None control: <CheckboxControl(header=[true])> If the above control is a DataToolparameter whose data type class does not include a sniff() method, make sure to include a proper 'ftype' attribute to the tag for the control within the <test> tag set. =================
Any workarounds ?
thanks, -gordon _______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
_______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
Greg Von Kuster Galaxy Development Team greg@bx.psu.edu
This was, in fact a problem with Galaxy, not twill, and has been fixed in change set 3408:6471c9b8149c which should be available in the distribution shortly. On Feb 17, 2010, at 11:15 AM, Greg Von Kuster wrote:
I looked into this a bit, but could not find the cause in the twill code, which is where the problem lies. For now, just keep test file names shorter than 30 characters and I'll track down a fix if I get a chance.
Greg
On Feb 16, 2010, at 11:51 PM, Assaf Gordon wrote:
Another strange but reproducible testing bug (at least on my computer):
if the name of the input file is longer than 30 characters, the test will fail with a weird exception.
Example: The following <test> works: ==== <test> <param name="input" value="replace_text_in_column_in1.txt" ftype="tabular" /> ... ... ====
While the following <test> will fail: ==== <test> <param name="input" value="replace_text_in_column_ink1.txt" ftype="tabular" /> ... ... ===
The only difference between the two is that the input file name is one letter longer ("k" added to the "ink1" part). All other parameters stay exactly the same, and both input files (with and without "k") exist.
The second <test> version fails with the following exception: ======== FAIL: Replace Text ( cshl_awk_replace_in_column ) > Test-1 ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/gordon/temp/galaxy/test/functional/test_toolbox.py", line 134, in test_tool self.do_it( td ) File "/home/gordon/temp/galaxy/test/functional/test_toolbox.py", line 56, in do_it self.run_tool( testdef.tool.id, repeat_name=repeat_name, **page_inputs ) File "/home/gordon/temp/galaxy/test/base/twilltestcase.py", line 920, in run_tool self.submit_form( **kwd ) File "/home/gordon/temp/galaxy/test/base/twilltestcase.py", line 870, in submit_form raise AssertionError( errmsg ) AssertionError: Attempting to set field 'input' to value '['replace_text_in_column_ink1.txt']' in form 'tool_form' threw exception: cannot find value/label "replace_text_in_column_ink1.txt" in list control control: <SelectControl(input=[1])> If the above control is a DataToolparameter whose data type class does not include a sniff() method, make sure to include a proper 'ftype' attribute to the tag for the control within the <test> tag set.
-------------------- >> begin captured stdout << --------------------- Uploaded file: replace_text_in_column_ink1.txt , ftype: tabular , extra: {'ftype': 'tabular', 'value': 'replace_text_in_column_ink1.txt'}
form 'tool_form' contains the following controls ( note the values ) control 0: <HiddenControl(tool_id=cshl_awk_replace_in_column) (readonly)> control 1: <HiddenControl (tool_state = 800255ed346436383633663935656435333636643238663535633132373339383564326365323733656165303a3762323236333666366337353664366532323361323032323563323233313563323232323263323032323639366537303735373432323361323032323331323232633230323235663566373036313637363535663566323233613230333032633230323237353732366335663730363137333734363532323361323032323563323234653666366536353563323232323263323032323636363936633635356636343631373436313232336132303232356332323465366636653635356332323232376471002e ) (readonly)> control 2: <SelectControl(input=[*1])> control 3: <SelectControl(column=[*1, 2, 3, 4, 5, 6])> control 4: <TextControl(url_paste=)> control 5: <TextControl(file_data=)> control 6: <SubmitControl(runtool_btn=Execute) (readonly)> ========
Looks like somehow the "input" selection box values are overwritten with the single value "1" .
I guess there's not much to do about this, except maybe warn and document it in the wiki.
-gordon
P.S. I assume you (galaxy team) didn't write the testing framework. Is there a documentation of the this framework, with possibly a list of these issues ? I'm trying to be a good Galaxy citizen and write unit tests for my tools, but these quirks waste a lot of time...
Kelly Vincent wrote, On 02/16/2010 10:19 PM:
Assaf,
This is a known limitation of the test framework at the moment. Boolean's can't be tested with a 'false' value.
It's not ideal, but what I resorted to doing was changing the boolean parameter to a select with 'true' and 'false' options.
Kelly
On Tue 16 Feb 2010, at 9:34 PM, Assaf Gordon wrote:
Hello,
I'm trying to add a <tests> section for a tool which has a boolean/checkbox parameter. If one of the tests set the parameter's value to "false", the test throws an exception (before starting the actual job). If changed to "true", the test works.
I've seen only a few tools with a boolean parameter (grouping.xml, complement.xml, concat.xml, merge.xml, phastOdds_tool.xml) and all of them only test the "true" value.
I've tried setting the value to "false", to an empty string, and removing the parameter entirely, but it always throws an exception similar to: ================= Traceback (most recent call last): File "/home/gordon/temp/galaxy/test/functional/test_toolbox.py", line 134, in test_tool self.do_it( td ) File "/home/gordon/temp/galaxy/test/functional/test_toolbox.py", line 56, in do_it self.run_tool( testdef.tool.id, repeat_name=repeat_name, **page_inputs ) File "/home/gordon/temp/galaxy/test/base/twilltestcase.py", line 920, in run_tool self.submit_form( **kwd ) File "/home/gordon/temp/galaxy/test/base/twilltestcase.py", line 870, in submit_form raise AssertionError( errmsg ) AssertionError: Attempting to set field 'header' to value '['false']' in form 'tool_form' threw exception: id=None name='false' label=None control: <CheckboxControl(header=[true])> If the above control is a DataToolparameter whose data type class does not include a sniff() method, make sure to include a proper 'ftype' attribute to the tag for the control within the <test> tag set. =================
Any workarounds ?
thanks, -gordon _______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
_______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
Greg Von Kuster Galaxy Development Team greg@bx.psu.edu
_______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
Greg Von Kuster Galaxy Development Team greg@bx.psu.edu
This problem has been fixed in change set 3408:6471c9b8149c which should be available in the distribution shortly. On Feb 16, 2010, at 9:34 PM, Assaf Gordon wrote:
Hello,
I'm trying to add a <tests> section for a tool which has a boolean/ checkbox parameter. If one of the tests set the parameter's value to "false", the test throws an exception (before starting the actual job). If changed to "true", the test works.
I've seen only a few tools with a boolean parameter (grouping.xml, complement.xml, concat.xml, merge.xml, phastOdds_tool.xml) and all of them only test the "true" value.
I've tried setting the value to "false", to an empty string, and removing the parameter entirely, but it always throws an exception similar to: ================= Traceback (most recent call last): File "/home/gordon/temp/galaxy/test/functional/test_toolbox.py", line 134, in test_tool self.do_it( td ) File "/home/gordon/temp/galaxy/test/functional/test_toolbox.py", line 56, in do_it self.run_tool( testdef.tool.id, repeat_name=repeat_name, **page_inputs ) File "/home/gordon/temp/galaxy/test/base/twilltestcase.py", line 920, in run_tool self.submit_form( **kwd ) File "/home/gordon/temp/galaxy/test/base/twilltestcase.py", line 870, in submit_form raise AssertionError( errmsg ) AssertionError: Attempting to set field 'header' to value '['false']' in form 'tool_form' threw exception: id=None name='false' label=None control: <CheckboxControl(header=[true])> If the above control is a DataToolparameter whose data type class does not include a sniff() method, make sure to include a proper 'ftype' attribute to the tag for the control within the <test> tag set. =================
Any workarounds ?
thanks, -gordon _______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
Greg Von Kuster Galaxy Development Team greg@bx.psu.edu
participants (3)
-
Assaf Gordon -
Greg Von Kuster -
Kelly Vincent