Hello all,
While looking for possible slowness causes in my galaxy server, I found the heartbeat log to invaluable, but very frustrating to read: it contains everything that the active threads are doing - while most of them aren't relevant at all.
I suggest the following patch, that adds a second heartbeat log which is easier to use.
This second log has the following properties: 1. Only non-sleeping threads are reported. 2. If all threads are sleeping, the log simply says "All threads are sleeping." 3. For each non-sleeping thread, only the inner-most frame is printed (filename,line,function,text) - if the inner most frame comes from within the "/lib/galaxy" folder (so that deeper frames from eggs aren't printed). 4. for each non-seeping thread, shows a count of how many consecutive hearts-beats this thread is active - meaning the thread is either running for N heartbeats, or is very often waken up.
This log makes it much easier to quickly pin-point what galaxy is doing, and then one can go back to the original heartbeat log and see the full stack trace.
Example of the heartbeat.log.nonsleeping (showing that the "set_meta" in sequence.py is running for 6 heartbeats = 6x20 seconds = 2 minutes: ======================== ... ... Non-Sleeping threads at Thu Jan 7 23:27:28 2010:
Thread 1101138256 <Thread(Thread-2, started)> non-sleeping for 4 heartbeat(s) File /lib/galaxy/datatypes/sequence.py:30 Function "set_meta" for line in file( dataset.file_name ):
Non-Sleeping threads at Thu Jan 7 23:27:48 2010:
Thread 1101138256 <Thread(Thread-2, started)> non-sleeping for 5 heartbeat(s) File /lib/galaxy/datatypes/sequence.py:30 Function "set_meta" for line in file( dataset.file_name ):
Non-Sleeping threads at Thu Jan 7 23:28:08 2010:
Thread 1101138256 <Thread(Thread-2, started)> non-sleeping for 6 heartbeat(s) File /lib/galaxy/datatypes/sequence.py:30 Function "set_meta" for line in file( dataset.file_name ):
Non-Sleeping threads at Thu Jan 7 23:28:28 2010:
All threads are sleeping. ... ... ... ========================
The patch is also available here: http://cancan.cshl.edu/labmembers/gordon/files/nonsleeping_heartbeat.patch
Comments are welcomed, -gordon