[galaxy-dev] Re: Error during tool installation: "inotify instance limit reached"

Thank you very much!

We have 19.01 running and the parameter that I had to put to "false" was "watch_tools"

All the best, Christopher

On 7/10/19 12:00 PM, Marius van den Beek wrote:

Hi Christopher,

you might want to turn off `watch_tools` if you are using it, this is only needed when using the pretty-much undocumented feature where you dump tools into a directory without registering them in any tool_conf.xml file. All other tool reloading happens by another means.

Similarly, if you're using 19.05 and you don't manipulate the tool data table entries manually you can disable `watch_tool_data_dir`.

Finally this is usually a sign of allocating too many processes and threads for your webservers/handlers/mules, as each one starts their own watchdog. A common misconception is that this makes Galaxy run faster. but it actually just increases the amount of concurrent requests Galaxy can handle.

Best, Marius

On Wed, 10 Jul 2019 at 11:43, Previti <christopher.previti@dkfz-heidelberg.de mailto:christopher.previti@dkfz-heidelberg.de> wrote:

Dear all,

I've been getting this error for a few days now, it basically prevents
anything from being installed on the first go.
Individual tools can be installed, but even then I have to try
multiple
times.

Any ideas how to fix this?

Cheers,
Christopher

The details from galaxy.log are (installing an older VCFtools
package):

172.22.24.119 - - [10/Jul/2019:10:44:32 +0200] "POST
/admin_toolshed/prepare_for_install HTTP/1.1" 200 -
"http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare_for_install?change
set_revisions=34a6b690e4b5&repository_ids=2ba42187f2588c4b&tool_shed_url=https%3A%2F%2Ftoolshed.g2.bx.psu.edu%2F
<http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare_for_install?changeset_revisions=34a6b690e4b5&repository_ids=2ba42187f2588c4b&tool_shed_url=https%3A%2F%2Ftoolshed.g2.bx.psu.edu%2F>"
"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:67.0) Gecko/20100101
Firefox/67.0"
galaxy.tools.toolbox.base DEBUG 2019-07-10 10:44:55,592 Appending to
tool panel section: NGS:VCFtools
172.22.24.119 - - [10/Jul/2019:10:44:58 +0200] "POST
/admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 -
"http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare
_for_install" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:67.0)
Gecko/20100101 Firefox/67.0"
172.22.24.119 - - [10/Jul/2019:10:45:01 +0200] "POST
/admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 -
"http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare
_for_install" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:67.0)
Gecko/20100101 Firefox/67.0"
172.22.24.119 - - [10/Jul/2019:10:44:55 +0200] "POST
/admin_toolshed/install_repositories HTTP/1.1" 500 -
"http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare_for_install"
"Moz
illa/5.0 (X11; Fedora; Linux x86_64; rv:67.0) Gecko/20100101
Firefox/67.0"
*Error - <type 'exceptions.OSError'>: inotify instance limit reached*
URL:
http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/install_repositories
File
'/opt/galaxy/galaxy/lib/galaxy/web/framework/middleware/error.py',
line 154 in __call__
  app_iter = self.application(environ, sr_checker)
File
'/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/paste/recursive.py',
line 85 in __call__
  return self.application(environ, start_response)
File
'/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/paste/httpexceptions.py',
line 640 in __call__
  return self.application(environ, start_response)
File '/opt/galaxy/galaxy/lib/galaxy/web/framework/base.py', line
143 in
__call__
  return self.handle_request(environ, start_response)
File '/opt/galaxy/galaxy/lib/galaxy/web/framework/base.py', line
222 in
handle_request
  body = method(trans, **kwargs)
File '/opt/galaxy/galaxy/lib/galaxy/web/framework/decorators.py', line
101 in decorator
  return func(self, trans, *args, **kwargs)
File
'/opt/galaxy/galaxy/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py',
line 575 in install_repositories
  reinstalling=reinstalling,
File
'/opt/galaxy/galaxy/lib/tool_shed/galaxy_install/install_manager.py',
line 846 in install_repositories
  tool_panel_section_mapping=tool_panel_section_mapping)
File
'/opt/galaxy/galaxy/lib/tool_shed/galaxy_install/install_manager.py',
line 893 in install_tool_shed_repository
  tool_panel_section_mapping=tool_panel_section_mapping)
File
'/opt/galaxy/galaxy/lib/tool_shed/galaxy_install/install_manager.py',
line 544 in __handle_repository_contents
  repository_tools_tups = irmm.get_repository_tools_tups()
File
'/opt/galaxy/galaxy/lib/tool_shed/galaxy_install/metadata/installed_repository_metadata_manager.py',
line 75 in get_repository_tools_tups
  tool =
self.app.toolbox.load_tool(os.path.abspath(load_relative_path),
guid=guid, use_cached=False)
File '/opt/galaxy/galaxy/lib/galaxy/tools/toolbox/base.py', line
766 in
load_tool
  self.watch_tool(tool)
File '/opt/galaxy/galaxy/lib/galaxy/tools/toolbox/base.py', line
774 in
watch_tool
  self._tool_watcher.watch_file(tool.config_file, tool.id
<http://tool.id>)
File '/opt/galaxy/galaxy/lib/galaxy/tools/toolbox/watcher.py',
line 198
in watch_file
  self.monitor(tool_dir)
File '/opt/galaxy/galaxy/lib/galaxy/tools/toolbox/watcher.py',
line 190
in monitor
  self.observer.schedule(self.event_handler, dir, recursive=False)
File
'/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/watchdog/observers/api.py',
line 293 in schedule
  emitter.start()
File
'/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/watchdog/utils/__init__.py',
line 110 in start
  self.on_thread_start()
File
'/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/watchdog/observers/inotify.py',
line 121 in on_thread_start
  self._inotify = InotifyBuffer(path, self.watch.is_recursive)
File
'/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/watchdog/observers/inotify_buffer.py',
line 35 in __init__
  self._inotify = Inotify(path, recursive)
File
'/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/watchdog/observers/inotify_c.py',
line 188 in __init__
  Inotify._raise_error()
*File
'/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/watchdog/observers/inotify_c.py',
line 415 in _raise_error**
**  raise OSError("inotify instance limit reached")**
**OSError: inotify instance limit reached**
*
CGI Variables
-------------
  CONTENT_LENGTH: '12169'
  CONTENT_TYPE: 'application/x-www-form-urlencoded; charset=UTF-8'
  HTTP_ACCEPT: 'text/html, */*; q=0.01'
  HTTP_ACCEPT_ENCODING: 'gzip,deflate'
  HTTP_ACCEPT_LANGUAGE: 'en-US,en;q=0.5'
  HTTP_CONNECTION: 'Keep-Alive'
  HTTP_COOKIE:
'galaxysession=c6ca0ddb55be603a9d01bef828912a83dd6a820b8a47ad278d223920b71ba8b43726aaf1140b008e'
  HTTP_HOST: 'dkfzgalaxy.inet.dkfz-heidelberg.de
<http://dkfzgalaxy.inet.dkfz-heidelberg.de>'
  HTTP_REFERER:
'http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare_for_install'
  HTTP_USER_AGENT: 'Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:67.0)
Gecko/20100101 Firefox/67.0'
  HTTP_X_FORWARDED_FOR: '172.22.24.119'
  HTTP_X_FORWARDED_HOST: 'dkfzgalaxy.inet.dkfz-heidelberg.de
<http://dkfzgalaxy.inet.dkfz-heidelberg.de>'
  HTTP_X_FORWARDED_SERVER: 'fe80::216:3eff:fe80:c02b'
  HTTP_X_REQUESTED_WITH: 'XMLHttpRequest'
  ORGINAL_HTTP_HOST: 'localhost:8080'
  ORGINAL_REMOTE_ADDR: '127.0.0.1'
  PATH_INFO: '/admin_toolshed/install_repositories'
  REMOTE_ADDR: '172.22.24.119'
  REQUEST_METHOD: 'POST'
  SERVER_NAME: '127.0.0.1'
  SERVER_PORT: '8080'
  SERVER_PROTOCOL: 'HTTP/1.1'


WSGI Variables
--------------
  application: <paste.recursive.RecursiveMiddleware object at
0x7f9f48207410>
  controller_action_key: u'web.admin_toolshed.install_repositories'
  is_api_request: False
  paste.cookies: (<SimpleCookie:
galaxysession='c6ca0ddb55be603a9d01bef828912a83dd6a820b8a47ad278d223920b71ba8b43726aaf1140b008e'>,
'galaxysession=c6ca0ddb55be603a9d01bef828912a83dd6a820b8a47ad278d223920b71ba8b43726aaf1140b008e')
  paste.expected_exceptions: [<class
'paste.httpexceptions.HTTPException'>]
  paste.httpexceptions: <paste.httpexceptions.HTTPExceptionHandler
object at 0x7f9f48c86fd0>
  paste.httpserver.proxy.host: 'dummy'
  paste.httpserver.proxy.scheme: 'http'
  paste.httpserver.thread_pool: <paste.httpserver.ThreadPool object at
0x7f9f486fcd90>
  paste.recursive.forward: <paste.recursive.Forwarder from />
  paste.recursive.include: <paste.recursive.Includer from />
  paste.recursive.include_app_iter:
<paste.recursive.IncluderAppIter from />
  paste.recursive.script_name: ''
  paste.throw_errors: True
  request_id: 'fd8cc98ea2ee11e9a89100163e80c02b'
  webob._body_file: (<_io.BufferedReader>, <socket._fileobject
object at
0x7f9f3a4b5ad0 length=12169>)
  webob._parsed_post_vars: (MultiDict([('tool_shed_repository_ids',
u"[u'ac218635d795e337', u'e3e09dfaa8fad8c8', u'1efe9eb0bff40152',
u'358d8704ef89cb6b', u'cbc5119f47881258']"), ('encoded_kwd',
u'59b03a7799b953da9827424b9cef2236c5b9f461:7b22737461747573223a2022646f6e65222c20226861735f7265706f7369746f72795f646570656e64656e63696573223a2066616c73652c2022696e636c756465735f746f6f6c735f666f725f646973706c61795f696e5f746f6f6c5f70616e656c223a20747275652c2022746f6f6c5f736865645f7265706f7369746f72795f696473223a205b2261633231383633356437393565333337222c202265336530396466616138666164386338222c202231656665396562306266663430313532222c202233353864383730346566383963623662222c202263626335313139663437383831323538225d2c2022736865645f746f6f6c5f636f6e66223a20222e2f636f6e6669672f736865645f746f6f6c5f636f6e662e786d6c222c2022696e7374616c6c5f7265706f7369746f72795f646570656e64656e63696573223a20747275652c2022746f6f6c5f70617468223a20222f6f70742f67616c6178792f736865645f746f6f6c73222c20227265706f5f696e666f5f6469637473223a2...
0x7f9f35cdfed0>)
  webob._parsed_query_vars: (GET([]), '')
  webob.is_body_seekable: True
  wsgi process: 'Multithreaded'
  wsgi.input_terminated: True




-- 
*Dr. Christopher Previti*
Genomics and Proteomics Core Facility
High Throughput Sequencing (W190)
Bioinformatician

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany
Room: B2.102 (INF580/TP3)
Phone: +49 6221 42-4661

christopher.previti@dkfz.de <mailto:christopher.previti@dkfz.de>
<http://www.dkfz.de/>
www.dkfz.de <http://www.dkfz.de> <http://www.dkfz.de/>

Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
VAT-ID No.: DE143293537

Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die
Personen bestimmt, an die sie adressiert ist.
Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte
Informationen enthalten. Sollten Sie nicht
der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den
Absender und löschen Sie die Mitteilung.
Jegliche unbefugte Verwendung der Informationen in dieser
Nachricht ist
untersagt.


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/