New subject: Suggestion: heartbeat log - show only non sleeping threads

8 Jan 2010


      Hello all,
While looking for possible slowness causes in my galaxy server, 
I found the heartbeat log to invaluable, but very frustrating to read: it contains everything that the active threads are doing - while most of them aren't relevant at all.
I suggest the following patch, that adds a second heartbeat log which is easier to use.
This second log has the following properties:
1. Only non-sleeping threads are reported.
2. If all threads are sleeping, the log simply says "All threads are sleeping."
3. For each non-sleeping thread, only the inner-most frame is printed (filename,line,function,text) - if the inner most frame comes from within the "/lib/galaxy" folder (so that deeper frames from eggs aren't printed).
4. for each non-seeping thread, shows a count of how many consecutive hearts-beats this thread is active - meaning the thread is either running for N heartbeats, or is very often waken up.
This log makes it much easier to quickly pin-point what galaxy is doing, and then one can go back to the original heartbeat log and see the full stack trace.
Example of the heartbeat.log.nonsleeping (showing that the "set_meta" in sequence.py is running for 6 heartbeats = 6x20 seconds = 2 minutes:
========================
...
...
Non-Sleeping threads at Thu Jan  7 23:27:28 2010:
Thread 1101138256	<Thread(Thread-2, started)>	non-sleeping for 4 heartbeat(s)
  File /lib/galaxy/datatypes/sequence.py:30
    Function "set_meta"
      for line in file( dataset.file_name ):
Non-Sleeping threads at Thu Jan  7 23:27:48 2010:
Thread 1101138256	<Thread(Thread-2, started)>	non-sleeping for 5 heartbeat(s)
  File /lib/galaxy/datatypes/sequence.py:30
    Function "set_meta"
      for line in file( dataset.file_name ):
Non-Sleeping threads at Thu Jan  7 23:28:08 2010:
Thread 1101138256	<Thread(Thread-2, started)>	non-sleeping for 6 heartbeat(s)
  File /lib/galaxy/datatypes/sequence.py:30
    Function "set_meta"
      for line in file( dataset.file_name ):
Non-Sleeping threads at Thu Jan  7 23:28:28 2010:
All threads are sleeping.
...
...
...
========================
The patch is also available here:
http://cancan.cshl.edu/labmembers/gordon/files/nonsleeping_heartbeat.patch
Comments are welcomed,
  -gordon