Great idea, and thanks. I've committed it in 3281:13e217878147. Assaf Gordon wrote:

Hello all,

While looking for possible slowness causes in my galaxy server, I found the heartbeat log to invaluable, but very frustrating to read: it contains everything that the active threads are doing - while most of them aren't relevant at all.

I suggest the following patch, that adds a second heartbeat log which is easier to use.

This second log has the following properties: 1. Only non-sleeping threads are reported. 2. If all threads are sleeping, the log simply says "All threads are sleeping." 3. For each non-sleeping thread, only the inner-most frame is printed (filename,line,function,text) - if the inner most frame comes from within the "/lib/galaxy" folder (so that deeper frames from eggs aren't printed). 4. for each non-seeping thread, shows a count of how many consecutive hearts-beats this thread is active - meaning the thread is either running for N heartbeats, or is very often waken up.

This log makes it much easier to quickly pin-point what galaxy is doing, and then one can go back to the original heartbeat log and see the full stack trace.

Example of the heartbeat.log.nonsleeping (showing that the "set_meta" in sequence.py is running for 6 heartbeats = 6x20 seconds = 2 minutes: ======================== ... ... Non-Sleeping threads at Thu Jan 7 23:27:28 2010:

Thread 1101138256 <Thread(Thread-2, started)> non-sleeping for 4 heartbeat(s) File /lib/galaxy/datatypes/sequence.py:30 Function "set_meta" for line in file( dataset.file_name ):

Non-Sleeping threads at Thu Jan 7 23:27:48 2010:

Thread 1101138256 <Thread(Thread-2, started)> non-sleeping for 5 heartbeat(s) File /lib/galaxy/datatypes/sequence.py:30 Function "set_meta" for line in file( dataset.file_name ):

Non-Sleeping threads at Thu Jan 7 23:28:08 2010:

Thread 1101138256 <Thread(Thread-2, started)> non-sleeping for 6 heartbeat(s) File /lib/galaxy/datatypes/sequence.py:30 Function "set_meta" for line in file( dataset.file_name ):

Non-Sleeping threads at Thu Jan 7 23:28:28 2010:

All threads are sleeping. ... ... ... ========================

The patch is also available here: http://cancan.cshl.edu/labmembers/gordon/files/nonsleeping_heartbeat.patch

Comments are welcomed, -gordon

------------------------------------------------------------------------

_______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev