I recently updated tests for one of my wrappers and came across this strange behaviour: The twill based testing reports a failure: https://travis-ci.org/saketkc/galaxy_tools/jobs/29956682#L1463 whereas, the API based testing shows success: https://travis-ci.org/saketkc/galaxy_tools/jobs/29956683 Unfortunately I cannot run these tests locally since I am behind a system proxy [Refer: http://dev.list.galaxyproject.org/Functional-Tests-and-ftype-td4664233.html] and have to rely on travis.. The place where twii tests fails shows that it is trying to compare the diff between 'chasm_output_genes.tabular' and 'chasm_output_variants.tabular' instead of 'chasm_output_genes.tabular'. [See : https://travis-ci.org/saketkc/galaxy_tools/jobs/29956682#L1469] I tried running my tools locally and I did not come across any case where the 'variants' output gets replaced by the 'genes' output, thus possibly ruling out unexpected behavior from the tool's server end. Is this a possible bug or am I missing something? Saket
Hi Saket,
From memory the Twill tests are fragile with the output file order in the XML.
John was discussing switching the default from the Twill to API backend, not sure when that is happening though... Peter On Tue, Jul 15, 2014 at 9:31 AM, Saket Choudhary <saketkc@gmail.com> wrote:
I recently updated tests for one of my wrappers and came across this strange behaviour:
The twill based testing reports a failure: https://travis-ci.org/saketkc/galaxy_tools/jobs/29956682#L1463
whereas, the API based testing shows success: https://travis-ci.org/saketkc/galaxy_tools/jobs/29956683
Unfortunately I cannot run these tests locally since I am behind a system proxy [Refer: http://dev.list.galaxyproject.org/Functional-Tests-and-ftype-td4664233.html] and have to rely on travis..
The place where twii tests fails shows that it is trying to compare the diff between 'chasm_output_genes.tabular' and 'chasm_output_variants.tabular' instead of 'chasm_output_genes.tabular'. [See : https://travis-ci.org/saketkc/galaxy_tools/jobs/29956682#L1469]
I tried running my tools locally and I did not come across any case where the 'variants' output gets replaced by the 'genes' output, thus possibly ruling out unexpected behavior from the tool's server end.
Is this a possible bug or am I missing something?
Saket
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Thanks Peter, I guess I should then rely on API based tests. On 15 July 2014 14:18, Peter Cock <p.j.a.cock@googlemail.com> wrote:
Hi Saket,
From memory the Twill tests are fragile with the output file order in the XML.
John was discussing switching the default from the Twill to API backend, not sure when that is happening though...
Peter
On Tue, Jul 15, 2014 at 9:31 AM, Saket Choudhary <saketkc@gmail.com> wrote:
I recently updated tests for one of my wrappers and came across this strange behaviour:
The twill based testing reports a failure: https://travis-ci.org/saketkc/galaxy_tools/jobs/29956682#L1463
whereas, the API based testing shows success: https://travis-ci.org/saketkc/galaxy_tools/jobs/29956683
Unfortunately I cannot run these tests locally since I am behind a system proxy [Refer:
http://dev.list.galaxyproject.org/Functional-Tests-and-ftype-td4664233.html ]
and have to rely on travis..
The place where twii tests fails shows that it is trying to compare the diff between 'chasm_output_genes.tabular' and 'chasm_output_variants.tabular' instead of 'chasm_output_genes.tabular'. [See : https://travis-ci.org/saketkc/galaxy_tools/jobs/29956682#L1469]
I tried running my tools locally and I did not come across any case where the 'variants' output gets replaced by the 'genes' output, thus possibly ruling out unexpected behavior from the tool's server end.
Is this a possible bug or am I missing something?
Saket
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
On Wed, Jul 16, 2014 at 7:44 PM, Saket Choudhary <saketkc@gmail.com> wrote:
Thanks Peter, I guess I should then rely on API based tests.
If it is just the order, make sure the order of the output files in the <test> is consistent with that in the <outputs> and it make be OK with Twill... I wonder if I filed a Trello card on this, or just an email? Peter
I think the confusion over two different test methods has reached a high level and Dave and I agreed previously that we should try to make the API driven version the default during this release cycle - but it is not our call :). Greg - is there anything else you feel that needs to happen to at least switch the test tool shed over and see how things go? Once the main tool shed is ready to target API testing we can swap over the default and issues like this and Aaron's earlier this week with collections will hopefully not cause confusion. -John On Wed, Jul 16, 2014 at 2:13 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Wed, Jul 16, 2014 at 7:44 PM, Saket Choudhary <saketkc@gmail.com> wrote:
Thanks Peter, I guess I should then rely on API based tests.
If it is just the order, make sure the order of the output files in the <test> is consistent with that in the <outputs> and it make be OK with Twill... I wonder if I filed a Trello card on this, or just an email?
Peter
John, Peter, The buildbot builders are already using the api interactor for both functional tests and the install and test framework. --Dave B. On Fri 18 Jul 2014 11:34:01 AM EDT, John Chilton wrote:
I think the confusion over two different test methods has reached a high level and Dave and I agreed previously that we should try to make the API driven version the default during this release cycle - but it is not our call :).
Greg - is there anything else you feel that needs to happen to at least switch the test tool shed over and see how things go? Once the main tool shed is ready to target API testing we can swap over the default and issues like this and Aaron's earlier this week with collections will hopefully not cause confusion.
-John
On Wed, Jul 16, 2014 at 2:13 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Wed, Jul 16, 2014 at 7:44 PM, Saket Choudhary <saketkc@gmail.com> wrote:
Thanks Peter, I guess I should then rely on API based tests.
If it is just the order, make sure the order of the output files in the <test> is consistent with that in the <outputs> and it make be OK with Twill... I wonder if I filed a Trello card on this, or just an email?
Peter
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
On Fri, Jul 18, 2014 at 5:14 PM, Dave Bouvier <dave@bx.psu.edu> wrote:
John, Peter,
The buildbot builders are already using the api interactor for both functional tests and the install and test framework.
--Dave B.
Great news. When did that happen? Did it cause any regressions (and can/did you flag those to the repository authors to alert them)? Assuming that change over went smoothly, is the plan for changing the default test back-end in the master branch of galaxy-central (and thus eventually galaxy-stable) for those running tool tests locally? Thanks, Peter
participants (4)
-
Dave Bouvier -
John Chilton -
Peter Cock -
Saket Choudhary