Thanks Dave, The good news is yes, the tests are running again on the Test Tool Shed (although not the main Tool Shed yet), and many of my tools now have successful test results from last night. e.g. My new basic mummer tool which now has a full set of dependency packages thanks to Bjoern: https://testtoolshed.g2.bx.psu.edu/view/peterjc/mummer The bad news is there are many unexpected failures with: "Exception: History in error state." I'm sure you'll learn more once you look over the logs, Thank you, Peter On Wed, Nov 5, 2014 at 6:21 PM, Dave Bouvier <dave@bx.psu.edu> wrote:

Peter,

This was due to a number of issues with the testing framework, the last of which appears to have been resolved, and I see tools being tested in the framework log. I'll check again in the morning to see if any of the tools you listed below were still not tested.

--Dave B.

On 11/03/2014 05:51 AM, Peter Cock wrote:

...

Hello all,

I am currently hoping to review the automated test results for some repositories which I have recently updated, in one case for dependency handling, the other functional changes:

https://testtoolshed.g2.bx.psu.edu/view/peterjc/mummer https://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus

These have not yet been tested. On further investigation of a sample of my other tools, it appears none of them have been tested on the Test Tool Shed since 2014-09-15, e.g.

https://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename https://testtoolshed.g2.bx.psu.edu/view/peterjc/sample_seqs https://testtoolshed.g2.bx.psu.edu/view/peterjc/samtools_idxstats https://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3 https://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod

Similarly, some of my tools on the Main Tool Shed appear not to have been tested since 2014-09-21, e.g.

https://toolshed.g2.bx.psu.edu/view/peterjc/seq_rename https://toolshed.g2.bx.psu.edu/view/peterjc/effectivet3 https://toolshed.g2.bx.psu.edu/view/peterjc/clinod

or 2014-10-27,

https://toolshed.g2.bx.psu.edu/view/peterjc/sample_seqs https://toolshed.g2.bx.psu.edu/view/peterjc/samtools_idxstats

Is there a known problem with the automated tool testing (previously every second night) on the Tool Sheds? Or have you had to further reduce the testing cycle?

Testing less frequently seems fine, say fortnightly, if this can be supplemented by testing updated tools everynight. That would give Tool Authors prompt feedback on their updates, but also catch regressions where changes in Galaxy break a previously working tool.

Regards,

Peter ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

Hey Peter, Dave is out this week - so I have tried to fumble around and see if I could make some progress on this. I found some bugs in a recent commit and fixed them - that might help (https://bitbucket.org/galaxy/galaxy-central/commits/b81798f94dc0fd14de1d585e...). I also have enabled more verbose logging that might help those "History in error state" exceptions (https://bitbucket.org/galaxy/galaxy-central/commits/a799879a82c54c2d1afec6e3...) but I am not sure it will propagate through to the tool shed API - we will see I guess. If the install and test framework just picks up the latest central - these fixes will hopefully be reflected in the next run. -John On Wed, Nov 19, 2014 at 4:48 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:

On Thu, Nov 6, 2014 at 11:08 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:

...

Thanks Dave,

The good news is yes, the tests are running again on the Test Tool Shed (although not the main Tool Shed yet), and many of my tools now have successful test results from last night.

e.g. My new basic mummer tool which now has a full set of dependency packages thanks to Bjoern: https://testtoolshed.g2.bx.psu.edu/view/peterjc/mummer

The bad news is there are many unexpected failures with: "Exception: History in error state."

I'm sure you'll learn more once you look over the logs,

Thank you,

Peter

Hi Dave,

Any progress? All the following seem to have been tested in the last few days on the TestToolShed, but failed with "Exception: History in error state."

https://testtoolshed.g2.bx.psu.edu/view/peterjc/blast2go https://testtoolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr https://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod https://testtoolshed.g2.bx.psu.edu/view/peterjc/fastq_paired_unpaired https://testtoolshed.g2.bx.psu.edu/view/peterjc/get_orfs_or_cdss https://testtoolshed.g2.bx.psu.edu/view/peterjc/mira_assembler https://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus https://testtoolshed.g2.bx.psu.edu/view/peterjc/nlstradamus https://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_filter_by_id https://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_primer_clip https://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename https://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id

That's about half of my TestToolShed repositories - most of the others report their tests passed :)

I am also seeing unexpected problems with packages, e.g.

https://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_29

Error getting revision e78bbab7933d of repository package_blast_plus_2_2_29 owned by iuc: An entry for the repository was not found in the database.

Thanks,

Peter ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

Hi Guys, I retitled this since the TestToolShed does seem to be running tests regularly again. However, I still have a fair number failing with the cryptic status "Exception: History in error state." I guess the logging changes John tried are not propagated to the user-facing output on the ToolShed, but might still offer you some clues? It appears from the stack trace to be something with uploading the test input files into the new history? My current list of tools with failing tests is: https://testtoolshed.g2.bx.psu.edu/view/peterjc/blast2go https://testtoolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr https://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod https://testtoolshed.g2.bx.psu.edu/view/peterjc/fastq_paired_unpaired https://testtoolshed.g2.bx.psu.edu/view/peterjc/get_orfs_or_cdss https://testtoolshed.g2.bx.psu.edu/view/peterjc/mira_assembler https://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus https://testtoolshed.g2.bx.psu.edu/view/peterjc/nlstradamus https://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_primer_clip I have updated some of the tools I listed below (in the older email), so their latest revision has not yet been tested. Regards, Peter On Wed, Nov 19, 2014 at 5:57 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:

Thanks John,

Fingers crossed we'll get some more detailed logs in a day or two :)

Peter

On Wed, Nov 19, 2014 at 5:48 PM, John Chilton <jmchilton@gmail.com> wrote:

...

Hey Peter,

Dave is out this week - so I have tried to fumble around and see if I could make some progress on this. I found some bugs in a recent commit and fixed them - that might help (https://bitbucket.org/galaxy/galaxy-central/commits/b81798f94dc0fd14de1d585e...). I also have enabled more verbose logging that might help those "History in error state" exceptions (https://bitbucket.org/galaxy/galaxy-central/commits/a799879a82c54c2d1afec6e3...) but I am not sure it will propagate through to the tool shed API - we will see I guess.

If the install and test framework just picks up the latest central - these fixes will hopefully be reflected in the next run.

-John

On Wed, Nov 19, 2014 at 4:48 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:

...

Hi Dave,

Any progress? All the following seem to have been tested in the last few days on the TestToolShed, but failed with "Exception: History in error state."

https://testtoolshed.g2.bx.psu.edu/view/peterjc/blast2go https://testtoolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr https://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod https://testtoolshed.g2.bx.psu.edu/view/peterjc/fastq_paired_unpaired https://testtoolshed.g2.bx.psu.edu/view/peterjc/get_orfs_or_cdss https://testtoolshed.g2.bx.psu.edu/view/peterjc/mira_assembler https://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus https://testtoolshed.g2.bx.psu.edu/view/peterjc/nlstradamus https://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_filter_by_id https://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_primer_clip https://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename https://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id

That's about half of my TestToolShed repositories - most of the others report their tests passed :)

I am also seeing unexpected problems with packages, e.g.

https://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_29

Error getting revision e78bbab7933d of repository package_blast_plus_2_2_29 owned by iuc: An entry for the repository was not found in the database.

Thanks,

Peter

On Wed, Dec 10, 2014 at 5:46 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:

On Thu, Nov 27, 2014 at 1:58 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:

...

Hi Guys,

I retitled this since the TestToolShed does seem to be running tests regularly again. However, I still have a fair number failing with the cryptic status "Exception: History in error state."

I guess the logging changes John tried are not propagated to the user-facing output on the ToolShed, but might still offer you some clues?

It appears from the stack trace to be something with uploading the test input files into the new history?

My current list of tools with failing tests is:

https://testtoolshed.g2.bx.psu.edu/view/peterjc/blast2go https://testtoolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr https://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod https://testtoolshed.g2.bx.psu.edu/view/peterjc/fastq_paired_unpaired https://testtoolshed.g2.bx.psu.edu/view/peterjc/get_orfs_or_cdss https://testtoolshed.g2.bx.psu.edu/view/peterjc/mira_assembler https://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus https://testtoolshed.g2.bx.psu.edu/view/peterjc/nlstradamus https://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_primer_clip

I have updated some of the tools I listed below (in the older email), so their latest revision has not yet been tested.

Regards,

Peter

Hi guys,

The good news is my list of failing tools has shrunk to the following, none of which seem to have been tested yet in December, stuck with "Exception: Job in error state.":

https://testtoolshed.g2.bx.psu.edu/view/peterjc/blast2go https://testtoolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr https://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod https://testtoolshed.g2.bx.psu.edu/view/peterjc/mira_assembler https://testtoolshed.g2.bx.psu.edu/view/peterjc/nlstradamus

Meanwhile, on the main tool, "Exception: Job in error state." and not yet tested in December:

https://toolshed.g2.bx.psu.edu/view/peterjc/blast2go

It would be nice to have a more informative error here...

Ugh - yes it clearly would - very sorry about that. https://trello.com/c/BZUaMnJB

Have these tools been automatically earmarked not to be retested? Note that some tools have been tested earlier this month:

I also have a legitimate failure for missing tests, run 2014-12-09, https://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus

Interesting... some tests did run though? Does the framework not report passing tests - seems like it would be hard to know if it working in that case. -John

And again, a legitimate failure for some missing tests on the main Tool Shed, also run on 2014-12-09, https://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus

Regards,

Peter

Peter, After fixing a number of issues, it appears that the tests are now running as they should. I'll keep an eye on the test framework in case any new errors crop up. --Dave B. On 12/10/2014 09:19 AM, Peter Cock wrote:

On Wed, Dec 10, 2014 at 2:03 PM, John Chilton <jmchilton@gmail.com> wrote:

...

On Wed, Dec 10, 2014 at 5:46 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:

...

Hi guys,

The good news is my list of failing tools has shrunk to the following, none of which seem to have been tested yet in December, stuck with "Exception: Job in error state.":

https://testtoolshed.g2.bx.psu.edu/view/peterjc/blast2go https://testtoolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr https://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod https://testtoolshed.g2.bx.psu.edu/view/peterjc/mira_assembler https://testtoolshed.g2.bx.psu.edu/view/peterjc/nlstradamus

Meanwhile, on the main tool, "Exception: Job in error state." and not yet tested in December:

https://toolshed.g2.bx.psu.edu/view/peterjc/blast2go

It would be nice to have a more informative error here...

Ugh - yes it clearly would - very sorry about that.

https://trello.com/c/BZUaMnJB

...

Have these tools been automatically earmarked not to be retested? Note that some tools have been tested earlier this month:

I also have a legitimate failure for missing tests, run 2014-12-09, https://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus

Interesting... some tests did run though? Does the framework not report passing tests - seems like it would be hard to know if it working in that case.

The "Latest revision: all tool tests pass" search returns lots of repositories, but I must examine each one-by-one to see when the tests were last run. The sample I just checked on the main Tool Shed were all lasted tested 2014-11-23,

https://toolshed.g2.bx.psu.edu/view/peterjc/align_back_trans https://toolshed.g2.bx.psu.edu/view/peterjc/blast_rbh https://toolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr https://toolshed.g2.bx.psu.edu/view/peterjc/clinod https://toolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id

However at least some tests have been run in December, e.g. https://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus

Things seem similar on the Test Tool Shed. Presumably you can sort the test results over the entire Tool Shed via a database search? That might help explain any pattern here.

Thanks,

Peter

On Wed, Jan 7, 2015 at 2:07 PM, Dave Bouvier <dave@bx.psu.edu> wrote:

Peter,

After fixing a number of issues, it appears that the tests are now running as they should. I'll keep an eye on the test framework in case any new errors crop up.

--Dave B.

Thanks Dave, Yes, the good news is tests are now being run again more regularly. The bad news is there still seem to be lots of unexpected failures. Also also all the recent RSS entries all zero, e.g. http://testtoolshed.g2.bx.psu.edu/repository/get_functional_test_rss?owner=peterjc&status=all Revision 3070949d8fd7 of samtools_idxstats 25 January 2015 14:41 0 tests passed, 0 tests failed, 0 tests missing test components. [Looks like 1 test passed to me] Revision a8c978f17251 of clinod 25 January 2015 15:55 0 tests passed, 0 tests failed, 0 tests missing test components. [Looks like 1 test failed to me, an unexpected failure which looks like a Galaxy problem in parameter handling on top of the old "Exception: History in error state." problem.] Revision 8e1e3a1ecad7 of samtools_depad 25 January 2015 13:22 0 tests passed, 0 tests failed, 0 tests missing test components. [Looks like 2 tests failed, perhaps a real tool problem] Revision fbe9139eb74b of mira_assembler 25 January 2015 16:22 0 tests passed, 0 tests failed, 0 tests missing test components. [Looks like 1 test failed to me, an unexpected failure which looks like a Galaxy problem as for clinod] Revision 31cb702eb5a8 of blast2go 25 January 2015 16:21 0 tests passed, 0 tests failed, 0 tests missing test components. [Looks like 1 test failed to me, the old "Exception: History in error state." issue again] Regards, Peter

On Tue, Jan 27, 2015 at 1:25 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:

On Tue, Jan 27, 2015 at 11:32 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:

...

On Wed, Jan 7, 2015 at 2:07 PM, Dave Bouvier <dave@bx.psu.edu> wrote:

...

Peter,

After fixing a number of issues, it appears that the tests are now running as they should. I'll keep an eye on the test framework in case any new errors crop up.

--Dave B.

Thanks Dave,

Yes, the good news is tests are now being run again more regularly. The bad news is there still seem to be lots of unexpected failures.

Also also all the recent RSS entries all zero, e.g. http://testtoolshed.g2.bx.psu.edu/repository/get_functional_test_rss?owner=peterjc&status=all

Revision 3070949d8fd7 of samtools_idxstats 25 January 2015 14:41 0 tests passed, 0 tests failed, 0 tests missing test components.

[Looks like 1 test passed to me]

I was going to log that on Trello, but overnight this has changed and the Test Tool Shed RSS feed now fails with: Internal Server Error Galaxy was unable to successfully complete your request URL: http://testtoolshed.g2.bx.psu.edu/repository/get_functional_test_rss?owner=peterjc&status=all Module galaxy.web.framework.middleware.error:149 in __call__

...

app_iter = self.application(environ, sr_checker) Module paste.debug.prints:106 in __call__ environ, self.app) Module paste.wsgilib:543 in intercept_output app_iter = application(environ, replacement_start_response) Module paste.recursive:84 in __call__ return self.application(environ, start_response) Module paste.httpexceptions:633 in __call__ return self.application(environ, start_response) Module galaxy.web.framework.base:132 in __call__ return self.handle_request( environ, start_response ) Module galaxy.web.framework.base:190 in handle_request body = method( trans, **kwargs ) Module galaxy.webapps.tool_shed.controllers.repository:1699 in get_functional_test_rss time_tested = repository_metadata.time_last_tested.strftime( '%a, %d %b %Y %H:%M:%S UT' ) AttributeError: 'NoneType' object has no attribute 'strftime' extra data

I have filed a Trello card for this error instead, https://trello.com/c/RrVBYIBZ/2379-test-toolshed-rss-feed-fails-when-time-la...

Meanwhile on the main Tool Shed, the RSS feed seems broken,

https://toolshed.g2.bx.psu.edu/repository/get_functional_test_rss?owner=peterjc&status=all

Internal Server Error Galaxy was unable to successfully complete your request An error occurred. This may be an intermittent problem due to load or other unpredictable factors, reloading the page may address the problem. The error has been logged to our team.

Trello issue logged, https://trello.com/c/ZOW0NPzb/2378-main-toolshed-rss-feed-for-test-results-o... Peter

On Fri Jan 30 2015 at 5:36:29 AM Peter Cock

...

...

https://toolshed.g2.bx.psu.edu/repository/get_functional_ test_rss?owner=peterjc&status=all

Internal Server Error ...

Trello issue logged, https://trello.com/c/ZOW0NPzb/2378-main-toolshed-rss-feed- for-test-results-offline

Also still broken.

This is the same error from the test toolshed, and this will be fixed with the next main toolshed server update that includes Dave's patch.