← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #16381

[Bug 1331877] [NEW] Neutron api server stopped working due to Errno 24

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

This traceback below was found in the neutron api server log. It
happened about one minute after the rabbitmq server was restarted after
the msgq service was out.  Afterwards, noticed that the api server is no
longer responding to requests from python-neutronclient.  However, the
api-server continues to respond to RPC calls from L3 and DHCP agents
even after the traceback was reported.

To get the api server to fully functional again, I have to stop and
restart the neutron api server.

There are multiple workers for the api server.

The traceback:

2014-05-28 08:46:16.977 16091 CRITICAL quantum [-] [Errno 24] Too many open files
2014-05-28 08:46:16.977 16091 TRACE quantum Traceback (most recent call last):
2014-05-28 08:46:16.977 16091 TRACE quantum   File "/usr/bin/quantum-server", line 27, in <module>
2014-05-28 08:46:16.977 16091 TRACE quantum     server()
2014-05-28 08:46:16.977 16091 TRACE quantum   File "/usr/lib/python2.7/dist-packages/quantum/server/__init__.py", line 38, in main
2014-05-28 08:46:16.977 16091 TRACE quantum     quantum_service = service.serve_wsgi(service.QuantumApiService)
2014-05-28 08:46:16.977 16091 TRACE quantum   File "/usr/lib/python2.7/dist-packages/quantum/service.py", line 102, in serve_wsgi
2014-05-28 08:46:16.977 16091 TRACE quantum     service.start()
2014-05-28 08:46:16.977 16091 TRACE quantum   File "/usr/lib/python2.7/dist-packages/quantum/service.py", line 69, in start
2014-05-28 08:46:16.977 16091 TRACE quantum     self.wsgi_app = _run_wsgi(self.app_name)
2014-05-28 08:46:16.977 16091 TRACE quantum   File "/usr/lib/python2.7/dist-packages/quantum/service.py", line 113, in _run_wsgi
2014-05-28 08:46:16.977 16091 TRACE quantum     server.start(app, cfg.CONF.bind_port, cfg.CONF.bind_host, workers=cfg.CONF.workers)
2014-05-28 08:46:16.977 16091 TRACE quantum   File "/usr/lib/python2.7/dist-packages/quantum/wsgi.py", line 206, in start
2014-05-28 08:46:16.977 16091 TRACE quantum     self.run_child()
2014-05-28 08:46:16.977 16091 TRACE quantum   File "/usr/lib/python2.7/dist-packages/quantum/wsgi.py", line 265, in run_child
2014-05-28 08:46:16.977 16091 TRACE quantum     self._run(self._application, self._socket)
2014-05-28 08:46:16.977 16091 TRACE quantum   File "/usr/lib/python2.7/dist-packages/quantum/wsgi.py", line 277, in _run
2014-05-28 08:46:16.977 16091 TRACE quantum     log=logging.WritableLogger(logger))
2014-05-28 08:46:16.977 16091 TRACE quantum   File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 655, in server
2014-05-28 08:46:16.977 16091 TRACE quantum     client_socket = sock.accept()
2014-05-28 08:46:16.977 16091 TRACE quantum   File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 154, in accept
2014-05-28 08:46:16.977 16091 TRACE quantum     res = socket_accept(fd)
2014-05-28 08:46:16.977 16091 TRACE quantum   File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 52, in socket_accept
2014-05-28 08:46:16.977 16091 TRACE quantum     return descriptor.accept()
2014-05-28 08:46:16.977 16091 TRACE quantum   File "/usr/lib/python2.7/socket.py", line 202, in accept
2014-05-28 08:46:16.977 16091 TRACE quantum     sock, addr = self._sock.accept()
2014-05-28 08:46:16.977 16091 TRACE quantum error: [Errno 24] Too many open files
2014-05-28 08:46:16.977 16091 TRACE quantum

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: api files many neutron open too

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1331877

Title:
  Neutron api server stopped working due to Errno 24

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  This traceback below was found in the neutron api server log. It
  happened about one minute after the rabbitmq server was restarted
  after the msgq service was out.  Afterwards, noticed that the api
  server is no longer responding to requests from python-neutronclient.
  However, the api-server continues to respond to RPC calls from L3 and
  DHCP agents even after the traceback was reported.

  To get the api server to fully functional again, I have to stop and
  restart the neutron api server.

  There are multiple workers for the api server.

  The traceback:

  2014-05-28 08:46:16.977 16091 CRITICAL quantum [-] [Errno 24] Too many open files
  2014-05-28 08:46:16.977 16091 TRACE quantum Traceback (most recent call last):
  2014-05-28 08:46:16.977 16091 TRACE quantum   File "/usr/bin/quantum-server", line 27, in <module>
  2014-05-28 08:46:16.977 16091 TRACE quantum     server()
  2014-05-28 08:46:16.977 16091 TRACE quantum   File "/usr/lib/python2.7/dist-packages/quantum/server/__init__.py", line 38, in main
  2014-05-28 08:46:16.977 16091 TRACE quantum     quantum_service = service.serve_wsgi(service.QuantumApiService)
  2014-05-28 08:46:16.977 16091 TRACE quantum   File "/usr/lib/python2.7/dist-packages/quantum/service.py", line 102, in serve_wsgi
  2014-05-28 08:46:16.977 16091 TRACE quantum     service.start()
  2014-05-28 08:46:16.977 16091 TRACE quantum   File "/usr/lib/python2.7/dist-packages/quantum/service.py", line 69, in start
  2014-05-28 08:46:16.977 16091 TRACE quantum     self.wsgi_app = _run_wsgi(self.app_name)
  2014-05-28 08:46:16.977 16091 TRACE quantum   File "/usr/lib/python2.7/dist-packages/quantum/service.py", line 113, in _run_wsgi
  2014-05-28 08:46:16.977 16091 TRACE quantum     server.start(app, cfg.CONF.bind_port, cfg.CONF.bind_host, workers=cfg.CONF.workers)
  2014-05-28 08:46:16.977 16091 TRACE quantum   File "/usr/lib/python2.7/dist-packages/quantum/wsgi.py", line 206, in start
  2014-05-28 08:46:16.977 16091 TRACE quantum     self.run_child()
  2014-05-28 08:46:16.977 16091 TRACE quantum   File "/usr/lib/python2.7/dist-packages/quantum/wsgi.py", line 265, in run_child
  2014-05-28 08:46:16.977 16091 TRACE quantum     self._run(self._application, self._socket)
  2014-05-28 08:46:16.977 16091 TRACE quantum   File "/usr/lib/python2.7/dist-packages/quantum/wsgi.py", line 277, in _run
  2014-05-28 08:46:16.977 16091 TRACE quantum     log=logging.WritableLogger(logger))
  2014-05-28 08:46:16.977 16091 TRACE quantum   File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 655, in server
  2014-05-28 08:46:16.977 16091 TRACE quantum     client_socket = sock.accept()
  2014-05-28 08:46:16.977 16091 TRACE quantum   File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 154, in accept
  2014-05-28 08:46:16.977 16091 TRACE quantum     res = socket_accept(fd)
  2014-05-28 08:46:16.977 16091 TRACE quantum   File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 52, in socket_accept
  2014-05-28 08:46:16.977 16091 TRACE quantum     return descriptor.accept()
  2014-05-28 08:46:16.977 16091 TRACE quantum   File "/usr/lib/python2.7/socket.py", line 202, in accept
  2014-05-28 08:46:16.977 16091 TRACE quantum     sock, addr = self._sock.accept()
  2014-05-28 08:46:16.977 16091 TRACE quantum error: [Errno 24] Too many open files
  2014-05-28 08:46:16.977 16091 TRACE quantum

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1331877/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next