Re: [galaxy-dev] wait thread: <1002:Not enough memory.> error > after enabling DRMAA
Hi,
------------------------------
Message: 13 Date: Wed, 18 May 2011 17:15:06 +0200 From: Leandro Hermida <softdev@leandrohermida.com> To: Galaxy Dev <galaxy-dev@bx.psu.edu> Subject: [galaxy-dev] wait thread: <1002:Not enough memory.> error after enabling DRMAA Message-ID: <BANLkTi=J4-ct04U=8dm6ra=sJcR_eDEG1A@mail.gmail.com> Content-Type: text/plain; charset="iso-8859-1"
Hi all,
I enabled DRMAA on my test Galaxy server installation and in the server startup output I get the following strange "E #14ca [ 1.10] * wait
are you using LSF?
thread: <1002:Not enough memory.>" lines after loading the job manager:
... galaxy.jobs.runners.local INFO 2011-05-18 17:10:23,025 starting workers galaxy.jobs.runners.local DEBUG 2011-05-18 17:10:23,026 5 workers ready galaxy.jobs DEBUG 2011-05-18 17:10:23,026 Loaded job runner: galaxy.jobs.runners.local:LocalJobRunner galaxy.jobs.runners.drmaa DEBUG 2011-05-18 17:10:23,130 3 workers ready galaxy.jobs DEBUG 2011-05-18 17:10:23,130 Loaded job runner: galaxy.jobs.runners.drmaa:DRMAAJobRunner galaxy.jobs INFO 2011-05-18 17:10:23,131 job manager started E #14ca [ 0.00] * wait thread: <1002:Not enough memory.> E #14ca [ 0.02] * wait thread: <1002:Not enough memory.> E #14ca [ 0.05] * wait thread: <1002:Not enough memory.> E #14ca [ 0.07] * wait thread: <1002:Not enough memory.> E #14ca [ 0.10] * wait thread: <1002:Not enough memory.> E #14ca [ 0.12] * wait thread: <1002:Not enough memory.> E #14ca [ 0.15] * wait thread: <1002:Not enough memory.> E #14ca [ 0.17] * wait thread: <1002:Not enough memory.> E #14ca [ 0.20] * wait thread: <1002:Not enough memory.> galaxy.jobs INFO 2011-05-18 17:10:23,373 job stopper started E #14ca [ 0.22] * wait thread: <1002:Not enough memory.> galaxy.sample_tracking.external_service_types DEBUG 2011-05-18 17:10:23,383 Loaded external_service_type: Simple unknown sequencer 1.0.0 galaxy.sample_tracking.external_service_types DEBUG 2011-05-18 17:10:23,386 Loaded external_service_type: Applied Biosystems SOLiD 1.0.0 galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,400 Enabling 'admin' controller, class: AdminGalaxy galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,402 Enabling 'async' controller, class: ASync E #14ca [ 0.25] * wait thread: <1002:Not enough memory.> E #14ca [ 0.27] * wait thread: <1002:Not enough memory.> galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,428 Enabling 'dataset' controller, class: DatasetInterface galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,432 Enabling 'error' controller, class: Error galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,436 Enabling 'external_service' controller, class: ExternalService galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,437 Enabling 'external_services' controller, class: ExternalServiceController galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,439 Enabling 'forms' controller, class: Forms galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,441 Enabling 'history' controller, class: HistoryController E #14ca [ 0.30] * wait thread: <1002:Not enough memory.> galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,460 Enabling 'library' controller, class: Library galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,461 Enabling 'library_admin' controller, class: LibraryAdmin galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,462 Enabling 'library_common' controller, class: LibraryCommon galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,475 Enabling 'mobile' controller, class: Mobile galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,478 Enabling 'page' controller, class: PageController E #14ca [ 0.32] * wait thread: <1002:Not enough memory.> galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,482 Enabling 'request_type' controller, class: RequestType galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,483 Enabling 'requests' controller, class: Requests E #14ca [ 0.35] * wait thread: <1002:Not enough memory.> galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,508 Enabling 'requests_admin' controller, class: RequestsAdmin galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,509 Enabling 'requests_common' controller, class: RequestsCommon galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,511 Enabling 'root' controller, class: RootController galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,514 Enabling 'tag' controller, class: TagsController galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,519 Enabling 'tool_runner' controller, class: ToolRunner E #14ca [ 0.37] * wait thread: <1002:Not enough memory.> E #14ca [ 0.40] * wait thread: <1002:Not enough memory.> galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,558 Enabling 'tracks' controller, class: TracksController galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,562 Enabling 'ucsc_proxy' controller, class: UCSCProxy galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,564 Enabling 'user' controller, class: User galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,567 Enabling 'visualization' controller, class: VisualizationController E #14ca [ 0.42] * wait thread: <1002:Not enough memory.> E #14ca [ 0.45] * wait thread: <1002:Not enough memory.> E #14ca [ 0.47] * wait thread: <1002:Not enough memory.> E #14ca [ 0.50] * wait thread: <1002:Not enough memory.> E #14ca [ 0.53] * wait thread: <1002:Not enough memory.> E #14ca [ 0.55] * wait thread: <1002:Not enough memory.> E #14ca [ 0.58] * wait thread: <1002:Not enough memory.> galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,747 Enabling 'workflow' controller, class: WorkflowController galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,755 Enabling 'contents' API controller, class: ContentsController galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,757 Enabling 'forms' API controller, class: FormDefinitionAPIController E #14ca [ 0.60] * wait thread: <1002:Not enough memory.> galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,762 Enabling 'libraries' API controller, class: LibrariesController galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,767 Enabling 'request_types' API controller, class: RequestTypeAPIController galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,768 Enabling 'requests' API controller, class: RequestsAPIController galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,770 Enabling 'roles' API controller, class: RoleAPIController galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,772 Enabling 'samples' API controller, class: SamplesAPIController galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,773 Enabling 'users' API controller, class: UserAPIController E #14ca [ 0.63] * wait thread: <1002:Not enough memory.> E #14ca [ 0.65] * wait thread: <1002:Not enough memory.> E #14ca [ 0.67] * wait thread: <1002:Not enough memory.> E #14ca [ 0.70] * wait thread: <1002:Not enough memory.> galaxy.web.buildapp DEBUG 2011-05-18 17:10:23,862 Enabling 'httpexceptions' middleware galaxy.web.buildapp DEBUG 2011-05-18 17:10:23,869 Enabling 'remote user' middleware galaxy.web.buildapp DEBUG 2011-05-18 17:10:23,871 Enabling 'recursive' middleware E #14ca [ 0.72] * wait thread: <1002:Not enough memory.> galaxy.web.buildapp DEBUG 2011-05-18 17:10:23,882 Enabling 'print debug' middleware E #14ca [ 0.74] * wait thread: <1002:Not enough memory.> E #14ca [ 0.77] * wait thread: <1002:Not enough memory.> E #14ca [ 0.79] * wait thread: <1002:Not enough memory.> E #14ca [ 0.82] * wait thread: <1002:Not enough memory.> E #14ca [ 0.85] * wait thread: <1002:Not enough memory.> /gpfshpc/home/lhermida/soft/galaxy/galaxy_dist/eggs/WebError-0.8a-py2.6.egg/weberror/exceptions/serial_number_generator.py:11: DeprecationWarning: the md5 module is deprecated; use hashlib instead import md5 E #14ca [ 0.87] * wait thread: <1002:Not enough memory.> E #14ca [ 0.90] * wait thread: <1002:Not enough memory.> galaxy.web.buildapp DEBUG 2011-05-18 17:10:24,058 Enabling 'eval exceptions' middleware galaxy.web.buildapp DEBUG 2011-05-18 17:10:24,062 Enabling 'trans logger' middleware galaxy.web.buildapp DEBUG 2011-05-18 17:10:24,062 Enabling 'config' middleware galaxy.web.buildapp DEBUG 2011-05-18 17:10:24,064 Enabling 'x-forwarded-host' middleware Starting server in PID 5229. serving on http://10.155.10.23:8082 E #14ca [ 0.92] * wait thread: <1002:Not enough memory.> E #14ca [ 0.95] * wait thread: <1002:Not enough memory.> E #14ca [ 0.97] * wait thread: <1002:Not enough memory.> ...
Do I need to increase some kind of running memory somewhere or is it another problem?
i think it is a bug. Could you try to edit the LSF_TOP/conf/lsf_drmaa.conf and set there wait_thread: 0 ? Unfortunately i do not have access to any LSF cluster so i can not dig further into this problem.
thanks, Leandro
Hi Mariusz, In another Galaxy post Marina had mentioned it is a bug with the latest FedStage LSF DRMAA 1.0.4 implementation. If you build and use the previous 1.0.3 version it works fine and you don't get this error. regards, Leandro 2011/5/19 Mariusz Mamoński <mamonski@man.poznan.pl>
Hi,
------------------------------
Message: 13 Date: Wed, 18 May 2011 17:15:06 +0200 From: Leandro Hermida <softdev@leandrohermida.com> To: Galaxy Dev <galaxy-dev@bx.psu.edu> Subject: [galaxy-dev] wait thread: <1002:Not enough memory.> error after enabling DRMAA Message-ID: <BANLkTi=J4-ct04U=8dm6ra=sJcR_eDEG1A@mail.gmail.com> Content-Type: text/plain; charset="iso-8859-1"
Hi all,
I enabled DRMAA on my test Galaxy server installation and in the server startup output I get the following strange "E #14ca [ 1.10] * wait
are you using LSF?
thread: <1002:Not enough memory.>" lines after loading the job manager:
... galaxy.jobs.runners.local INFO 2011-05-18 17:10:23,025 starting workers galaxy.jobs.runners.local DEBUG 2011-05-18 17:10:23,026 5 workers ready galaxy.jobs DEBUG 2011-05-18 17:10:23,026 Loaded job runner: galaxy.jobs.runners.local:LocalJobRunner galaxy.jobs.runners.drmaa DEBUG 2011-05-18 17:10:23,130 3 workers ready galaxy.jobs DEBUG 2011-05-18 17:10:23,130 Loaded job runner: galaxy.jobs.runners.drmaa:DRMAAJobRunner galaxy.jobs INFO 2011-05-18 17:10:23,131 job manager started E #14ca [ 0.00] * wait thread: <1002:Not enough memory.> E #14ca [ 0.02] * wait thread: <1002:Not enough memory.> E #14ca [ 0.05] * wait thread: <1002:Not enough memory.> E #14ca [ 0.07] * wait thread: <1002:Not enough memory.> E #14ca [ 0.10] * wait thread: <1002:Not enough memory.> E #14ca [ 0.12] * wait thread: <1002:Not enough memory.> E #14ca [ 0.15] * wait thread: <1002:Not enough memory.> E #14ca [ 0.17] * wait thread: <1002:Not enough memory.> E #14ca [ 0.20] * wait thread: <1002:Not enough memory.> galaxy.jobs INFO 2011-05-18 17:10:23,373 job stopper started E #14ca [ 0.22] * wait thread: <1002:Not enough memory.> galaxy.sample_tracking.external_service_types DEBUG 2011-05-18 17:10:23,383 Loaded external_service_type: Simple unknown sequencer 1.0.0 galaxy.sample_tracking.external_service_types DEBUG 2011-05-18 17:10:23,386 Loaded external_service_type: Applied Biosystems SOLiD 1.0.0 galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,400 Enabling 'admin' controller, class: AdminGalaxy galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,402 Enabling 'async' controller, class: ASync E #14ca [ 0.25] * wait thread: <1002:Not enough memory.> E #14ca [ 0.27] * wait thread: <1002:Not enough memory.> galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,428 Enabling 'dataset' controller, class: DatasetInterface galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,432 Enabling 'error' controller, class: Error galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,436 Enabling 'external_service' controller, class: ExternalService galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,437 Enabling 'external_services' controller, class: ExternalServiceController galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,439 Enabling 'forms' controller, class: Forms galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,441 Enabling 'history' controller, class: HistoryController E #14ca [ 0.30] * wait thread: <1002:Not enough memory.> galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,460 Enabling 'library' controller, class: Library galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,461 Enabling 'library_admin' controller, class: LibraryAdmin galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,462 Enabling 'library_common' controller, class: LibraryCommon galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,475 Enabling 'mobile' controller, class: Mobile galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,478 Enabling 'page' controller, class: PageController E #14ca [ 0.32] * wait thread: <1002:Not enough memory.> galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,482 Enabling 'request_type' controller, class: RequestType galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,483 Enabling 'requests' controller, class: Requests E #14ca [ 0.35] * wait thread: <1002:Not enough memory.> galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,508 Enabling 'requests_admin' controller, class: RequestsAdmin galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,509 Enabling 'requests_common' controller, class: RequestsCommon galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,511 Enabling 'root' controller, class: RootController galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,514 Enabling 'tag' controller, class: TagsController galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,519 Enabling 'tool_runner' controller, class: ToolRunner E #14ca [ 0.37] * wait thread: <1002:Not enough memory.> E #14ca [ 0.40] * wait thread: <1002:Not enough memory.> galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,558 Enabling 'tracks' controller, class: TracksController galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,562 Enabling 'ucsc_proxy' controller, class: UCSCProxy galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,564 Enabling 'user' controller, class: User galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,567 Enabling 'visualization' controller, class: VisualizationController E #14ca [ 0.42] * wait thread: <1002:Not enough memory.> E #14ca [ 0.45] * wait thread: <1002:Not enough memory.> E #14ca [ 0.47] * wait thread: <1002:Not enough memory.> E #14ca [ 0.50] * wait thread: <1002:Not enough memory.> E #14ca [ 0.53] * wait thread: <1002:Not enough memory.> E #14ca [ 0.55] * wait thread: <1002:Not enough memory.> E #14ca [ 0.58] * wait thread: <1002:Not enough memory.> galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,747 Enabling 'workflow' controller, class: WorkflowController galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,755 Enabling 'contents' API controller, class: ContentsController galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,757 Enabling 'forms' API controller, class: FormDefinitionAPIController E #14ca [ 0.60] * wait thread: <1002:Not enough memory.> galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,762 Enabling 'libraries' API controller, class: LibrariesController galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,767 Enabling 'request_types' API controller, class: RequestTypeAPIController galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,768 Enabling 'requests' API controller, class: RequestsAPIController galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,770 Enabling 'roles' API controller, class: RoleAPIController galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,772 Enabling 'samples' API controller, class: SamplesAPIController galaxy.web.framework.base DEBUG 2011-05-18 17:10:23,773 Enabling 'users' API controller, class: UserAPIController E #14ca [ 0.63] * wait thread: <1002:Not enough memory.> E #14ca [ 0.65] * wait thread: <1002:Not enough memory.> E #14ca [ 0.67] * wait thread: <1002:Not enough memory.> E #14ca [ 0.70] * wait thread: <1002:Not enough memory.> galaxy.web.buildapp DEBUG 2011-05-18 17:10:23,862 Enabling 'httpexceptions' middleware galaxy.web.buildapp DEBUG 2011-05-18 17:10:23,869 Enabling 'remote user' middleware galaxy.web.buildapp DEBUG 2011-05-18 17:10:23,871 Enabling 'recursive' middleware E #14ca [ 0.72] * wait thread: <1002:Not enough memory.> galaxy.web.buildapp DEBUG 2011-05-18 17:10:23,882 Enabling 'print debug' middleware E #14ca [ 0.74] * wait thread: <1002:Not enough memory.> E #14ca [ 0.77] * wait thread: <1002:Not enough memory.> E #14ca [ 0.79] * wait thread: <1002:Not enough memory.> E #14ca [ 0.82] * wait thread: <1002:Not enough memory.> E #14ca [ 0.85] * wait thread: <1002:Not enough memory.>
/gpfshpc/home/lhermida/soft/galaxy/galaxy_dist/eggs/WebError-0.8a-py2.6.egg/weberror/exceptions/serial_number_generator.py:11:
DeprecationWarning: the md5 module is deprecated; use hashlib instead import md5 E #14ca [ 0.87] * wait thread: <1002:Not enough memory.> E #14ca [ 0.90] * wait thread: <1002:Not enough memory.> galaxy.web.buildapp DEBUG 2011-05-18 17:10:24,058 Enabling 'eval exceptions' middleware galaxy.web.buildapp DEBUG 2011-05-18 17:10:24,062 Enabling 'trans logger' middleware galaxy.web.buildapp DEBUG 2011-05-18 17:10:24,062 Enabling 'config' middleware galaxy.web.buildapp DEBUG 2011-05-18 17:10:24,064 Enabling 'x-forwarded-host' middleware Starting server in PID 5229. serving on http://10.155.10.23:8082 E #14ca [ 0.92] * wait thread: <1002:Not enough memory.> E #14ca [ 0.95] * wait thread: <1002:Not enough memory.> E #14ca [ 0.97] * wait thread: <1002:Not enough memory.> ...
Do I need to increase some kind of running memory somewhere or is it another problem?
i think it is a bug. Could you try to edit the
LSF_TOP/conf/lsf_drmaa.conf
and set there
wait_thread: 0
?
Unfortunately i do not have access to any LSF cluster so i can not dig further into this problem.
thanks, Leandro
participants (2)
-
Leandro Hermida -
Mariusz Mamoński