SHAUN WEBB wrote:
Further to my email below.
I have a data library that contains many ~117Gb of NGS data, uploaded via file system path. This library was always slow to open (about 10s) but now takes several minutes or not at all.
Thanks for any help on this. I am still experiencing a memory leak that I can't pinpoint and it is only dissipated by restarting the server. At the moment my debugging level is set to INFO. Is there anything I can change in the universe file to try to trace this.
Two things that would help here: Are there other users doing things in Galaxy at this time? It'd help to be able to determine exactly what is triggering this. Also, if you set 'use_heartbeat = True' in universe_wsgi.ini, this will dump the call stack every 30 seconds to the file 'heartbeat.log' (and 'heartbeat.log.nonsleeping'). This should reveal where the thread(s) are hung. --nate
Thanks! Shaun
----- Forwarded message from swebb1@staffmail.ed.ac.uk ----- Date: Wed, 09 Mar 2011 10:15:13 +0000 From: SHAUN WEBB <swebb1@staffmail.ed.ac.uk> Subject: Galaxy process To: galaxy dev <galaxy-dev@bx.psu.edu>
Hi,
since making the last update I have found some new warnings in my paster.log, it also seems as though the galaxy process starts to gather memory and eventually hang (35% of 64G memory).
I've posted the entries below.
If anyone could help me understand what is going on that would be great.
Thanks. Shaun Webb
paste.httpserver.ThreadPool INFO 2011-03-09 09:49:49,962 No idle tasks, and only 0 busy tasks; adding 5 more workers paste.httpserver.ThreadPool INFO 2011-03-09 09:49:58,754 No idle tasks, and only 4 busy tasks; adding 1 more workers paste.httpserver.ThreadPool INFO 2011-03-09 09:51:47,301 Culling 6 extra workers (5 idle workers present) paste.httpserver.ThreadPool INFO 2011-03-09 09:55:17,163 No idle tasks, and only 0 busy tasks; adding 5 more workers 129.215.14.72 - - [09/Mar/2011:09:48:40 +0100] "GET /history HTTP/1.1" 500 - "http://bifx-core.bio.ed.ac.uk:8080/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 (.NET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)" paste.httpserver.ThreadPool INFO 2011-03-09 10:13:16,956 Culling 5 extra workers (7 idle workers present) 212.183.140.59 - - [09/Mar/2011:10:13:17 +0100] "GET / HTTP/1.1" 200 - "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.102 Safari/534.13" paste.httpserver.ThreadPool INFO 2011-03-09 10:14:35,715 No idle tasks, and only 2 busy tasks; adding 3 more workers paste.httpserver.ThreadPool WARNING 2011-03-09 10:15:15,104 Thread 140283224094464 hung (working on task for 3096 seconds) ---------------------------------------- Exception happened during processing of request from ('212.183.140.59', 10871) Traceback (most recent call last): File "/usr/lib/python2.6/SocketServer.py", line 281, in _handle_request_noblock self.process_request(request, client_address) File "/storage/home/galaxy/galaxy_dist/eggs/Paste-1.6-py2.6.egg/paste/httpserver.py", line 1037, in process_request lambda: self.process_request_in_thread(request, client_address)) File "/storage/home/galaxy/galaxy_dist/eggs/Paste-1.6-py2.6.egg/paste/httpserver.py", line 617, in add_task self.kill_hung_threads() File "/storage/home/galaxy/galaxy_dist/eggs/Paste-1.6-py2.6.egg/paste/httpserver.py", line 778, in kill_hung_threads self.kill_worker(worker.thread_id) File "/storage/home/galaxy/galaxy_dist/eggs/Paste-1.6-py2.6.egg/paste/httpserver.py", line 705, in kill_worker killthread.async_raise(thread_id, SystemExit) File "/storage/home/galaxy/galaxy_dist/eggs/Paste-1.6-py2.6.egg/paste/util/killthread.py", line 22, in async_raise raise ValueError("invalid thread id") ValueError: invalid thread id ----------------------------------------
-- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
----- End forwarded message -----
-- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: