yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem@xxxxxxxxxx>]

This confirms it's happening in the community code for keystone and
neutron, but apparently it's not fatal:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiIEFzc2VydGlvbkVycm9yOiBDYWxsaW5nIHdhaXRhbGwoKSBmcm9tIHdpdGhpbiBvbmUgb2YgdGhlIEdyZWVuUG9vbCdzIGdyZWVudGhyZWFkcyB3aWxsIG5ldmVyIHRlcm1pbmF0ZS5cIiBBTkQgdGFnczpcImNvbnNvbGVcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQyNDUzMzY5NDMzMCwibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ==

** Also affects: keystone
   Importance: Undecided
       Status: New

** Changed in: keystone
       Status: New => Confirmed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1423250

Title:
  Terminating neutron-server leads to waitall() inside a GreenPool's
  greenthread

Status in OpenStack Identity (Keystone):
  Confirmed
Status in OpenStack Neutron (virtual network service):
  In Progress

Bug description:
  Sending a termination signal (e.g. SIGHUP or SIGTERM) to neutron
  server with more that one api/rpc worker leads to the following
  traceback:

  2015-02-18 16:05:18.411 CRITICAL neutron [-] AssertionError: Calling
  waitall() from within one of the GreenPool's greenthreads will never
  terminate.

  2015-02-18 16:05:18.411 TRACE neutron Traceback (most recent call last):
  2015-02-18 16:05:18.411 TRACE neutron   File "/usr/local/bin/neutron-server", line 9, in <module>
  2015-02-18 16:05:18.411 TRACE neutron     load_entry_point('neutron==2015.1.dev117', 'console_scripts', 'neutron-server')()
  2015-02-18 16:05:18.411 TRACE neutron   File "/opt/stack/neutron/neutron/server/__init__.py", line 46, in main
  2015-02-18 16:05:18.411 TRACE neutron     neutron_api = service.serve_wsgi(service.NeutronApiService)
  2015-02-18 16:05:18.411 TRACE neutron   File "/opt/stack/neutron/neutron/service.py", line 106, in serve_wsgi
  2015-02-18 16:05:18.411 TRACE neutron     LOG.exception(_LE('Unrecoverable error: please check log '
  2015-02-18 16:05:18.411 TRACE neutron   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 82, in __exit__
  2015-02-18 16:05:18.411 TRACE neutron     six.reraise(self.type_, self.value, self.tb)
  2015-02-18 16:05:18.411 TRACE neutron   File "/opt/stack/neutron/neutron/service.py", line 103, in serve_wsgi
  2015-02-18 16:05:18.411 TRACE neutron     service.start()
  2015-02-18 16:05:18.411 TRACE neutron   File "/opt/stack/neutron/neutron/service.py", line 74, in start
  2015-02-18 16:05:18.411 TRACE neutron     self.wsgi_app = _run_wsgi(self.app_name)
  2015-02-18 16:05:18.411 TRACE neutron   File "/opt/stack/neutron/neutron/service.py", line 175, in _run_wsgi
  2015-02-18 16:05:18.411 TRACE neutron     workers=cfg.CONF.api_workers)
  2015-02-18 16:05:18.411 TRACE neutron   File "/opt/stack/neutron/neutron/wsgi.py", line 224, in start
  2015-02-18 16:05:18.411 TRACE neutron     self._launch(application, workers)
  2015-02-18 16:05:18.411 TRACE neutron   File "/opt/stack/neutron/neutron/wsgi.py", line 238, in _launch
  2015-02-18 16:05:18.411 TRACE neutron     self._server.launch_service(service, workers=workers)
  2015-02-18 16:05:18.411 TRACE neutron   File "/opt/stack/neutron/neutron/openstack/common/service.py", line 333, in launch_service
  2015-02-18 16:05:18.411 TRACE neutron     self._start_child(wrap)
  2015-02-18 16:05:18.411 TRACE neutron   File "/opt/stack/neutron/neutron/openstack/common/service.py", line 314, in _start_child
  2015-02-18 16:05:18.411 TRACE neutron     status, signo = self._child_wait_for_exit_or_signal(launcher)
  2015-02-18 16:05:18.411 TRACE neutron   File "/opt/stack/neutron/neutron/openstack/common/service.py", line 272, in _child_wait_for_exit_or_signal
  2015-02-18 16:05:18.411 TRACE neutron     launcher.stop()
  2015-02-18 16:05:18.411 TRACE neutron   File "/opt/stack/neutron/neutron/openstack/common/service.py", line 126, in stop
  2015-02-18 16:05:18.411 TRACE neutron     self.services.stop()
  2015-02-18 16:05:18.411 TRACE neutron   File "/opt/stack/neutron/neutron/openstack/common/service.py", line 462, in stop
  2015-02-18 16:05:18.411 TRACE neutron     service.stop()
  2015-02-18 16:05:18.411 TRACE neutron   File "/opt/stack/neutron/neutron/wsgi.py", line 117, in stop
  2015-02-18 16:05:18.411 TRACE neutron     self._server.kill()
  2015-02-18 16:05:18.411 TRACE neutron   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 240, in kill
  2015-02-18 16:05:18.411 TRACE neutron     return kill(self, *throw_args)
  2015-02-18 16:05:18.411 TRACE neutron   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 294, in kill
  2015-02-18 16:05:18.411 TRACE neutron     g.throw(*throw_args)
  2015-02-18 16:05:18.411 TRACE neutron   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
  2015-02-18 16:05:18.411 TRACE neutron     result = function(*args, **kwargs)
  2015-02-18 16:05:18.411 TRACE neutron   File "/opt/stack/neutron/neutron/wsgi.py", line 263, in _run
  2015-02-18 16:05:18.411 TRACE neutron     socket_timeout=self.client_socket_timeout)
  2015-02-18 16:05:18.411 TRACE neutron   File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 828, in server
  2015-02-18 16:05:18.411 TRACE neutron     pool.waitall()
  2015-02-18 16:05:18.411 TRACE neutron   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py", line 117, in waitall
  2015-02-18 16:05:18.411 TRACE neutron     "Calling waitall() from within one of the " \
  2015-02-18 16:05:18.411 TRACE neutron AssertionError: Calling waitall() from within one of the GreenPool's greenthreads will never terminate.
  2015-02-18 16:05:18.411 TRACE neutron

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1423250/+subscriptions