yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1443910] [NEW] Zookeeper servicegroup driver crashes

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Roman Dobosz <roman.dobosz@xxxxxxxxx>
Date: Tue, 14 Apr 2015 11:57:47 -0000
Reply-to: Bug 1443910 <1443910@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Zookeeper driver is based on zookeeper and evzookeeper modules. The latter is
the source of nasty crash, which is well visible on nova conductor. To reproduce
it is enough to enable zookeeper in nova.conf, provide configuration for the
zookeeper service address and stack the thing. The traceback:

2015-04-14 13:23:22.622 TRACE nova Traceback (most recent call last):
2015-04-14 13:23:22.622 TRACE nova   File "/usr/local/bin/nova-conductor", line 10, in <module>
2015-04-14 13:23:22.622 TRACE nova     sys.exit(main())
2015-04-14 13:23:22.622 TRACE nova   File "/opt/stack/nova/nova/cmd/conductor.py", line 44, in main
2015-04-14 13:23:22.622 TRACE nova     manager=CONF.conductor.manager)
2015-04-14 13:23:22.622 TRACE nova   File "/opt/stack/nova/nova/service.py", line 277, in create
2015-04-14 13:23:22.622 TRACE nova     db_allowed=db_allowed)
2015-04-14 13:23:22.622 TRACE nova   File "/opt/stack/nova/nova/service.py", line 146, in __init__
2015-04-14 13:23:22.622 TRACE nova     self.servicegroup_api = servicegroup.API(db_allowed=db_allowed)
2015-04-14 13:23:22.622 TRACE nova   File "/opt/stack/nova/nova/servicegroup/api.py", line 76, in __init__
2015-04-14 13:23:22.622 TRACE nova     *args, **kwargs)
2015-04-14 13:23:22.622 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 38, in import_object
2015-04-14 13:23:22.622 TRACE nova     return import_class(import_str)(*args, **kwargs)
2015-04-14 13:23:22.622 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 27, in import_class
2015-04-14 13:23:22.622 TRACE nova     __import__(mod_str)
2015-04-14 13:23:22.622 TRACE nova   File "/opt/stack/nova/nova/servicegroup/drivers/zk.py", line 28, in <module>
2015-04-14 13:23:22.622 TRACE nova     evzookeeper = importutils.try_import('evzookeeper')
2015-04-14 13:23:22.622 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 71, in try_import
2015-04-14 13:23:22.622 TRACE nova     return import_module(import_str)
2015-04-14 13:23:22.622 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 57, in import_module
2015-04-14 13:23:22.622 TRACE nova     __import__(import_str)
2015-04-14 13:23:22.622 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/evzookeeper/__init__.py", line 26, in <module>
2015-04-14 13:23:22.622 TRACE nova     from evzookeeper import utils
2015-04-14 13:23:22.622 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/evzookeeper/utils.py", line 26, in <module>
2015-04-14 13:23:22.622 TRACE nova     class _SocketDuckForFdTimeout(greenio._SocketDuckForFd):
2015-04-14 13:23:22.622 TRACE nova AttributeError: 'module' object has no attribute '_SocketDuckForFd'

The root cause of the problem is the change, which have a place in module
eventlet 0.17, which evzookeeper module is depend on. Because of the change the
way eventlet.greenio is exposed there is no way to reach the class
_SocketDuckForFd other way than explicitly importing it via
eventlet.greenio.py2 module.

Definitely solution for this problem might go upstream to the evzookeeper
author, however the development of evzookeeper module seems stalled (no
activity for last 2 years), maybe it's worth to consider changing the zk driver
implementation to use different zk module (kazoo seems quite active).

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1443910

Title:
  Zookeeper servicegroup driver crashes

Status in OpenStack Compute (Nova):
  New

Bug description:
  Zookeeper driver is based on zookeeper and evzookeeper modules. The latter is
  the source of nasty crash, which is well visible on nova conductor. To reproduce
  it is enough to enable zookeeper in nova.conf, provide configuration for the
  zookeeper service address and stack the thing. The traceback:

  2015-04-14 13:23:22.622 TRACE nova Traceback (most recent call last):
  2015-04-14 13:23:22.622 TRACE nova   File "/usr/local/bin/nova-conductor", line 10, in <module>
  2015-04-14 13:23:22.622 TRACE nova     sys.exit(main())
  2015-04-14 13:23:22.622 TRACE nova   File "/opt/stack/nova/nova/cmd/conductor.py", line 44, in main
  2015-04-14 13:23:22.622 TRACE nova     manager=CONF.conductor.manager)
  2015-04-14 13:23:22.622 TRACE nova   File "/opt/stack/nova/nova/service.py", line 277, in create
  2015-04-14 13:23:22.622 TRACE nova     db_allowed=db_allowed)
  2015-04-14 13:23:22.622 TRACE nova   File "/opt/stack/nova/nova/service.py", line 146, in __init__
  2015-04-14 13:23:22.622 TRACE nova     self.servicegroup_api = servicegroup.API(db_allowed=db_allowed)
  2015-04-14 13:23:22.622 TRACE nova   File "/opt/stack/nova/nova/servicegroup/api.py", line 76, in __init__
  2015-04-14 13:23:22.622 TRACE nova     *args, **kwargs)
  2015-04-14 13:23:22.622 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 38, in import_object
  2015-04-14 13:23:22.622 TRACE nova     return import_class(import_str)(*args, **kwargs)
  2015-04-14 13:23:22.622 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 27, in import_class
  2015-04-14 13:23:22.622 TRACE nova     __import__(mod_str)
  2015-04-14 13:23:22.622 TRACE nova   File "/opt/stack/nova/nova/servicegroup/drivers/zk.py", line 28, in <module>
  2015-04-14 13:23:22.622 TRACE nova     evzookeeper = importutils.try_import('evzookeeper')
  2015-04-14 13:23:22.622 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 71, in try_import
  2015-04-14 13:23:22.622 TRACE nova     return import_module(import_str)
  2015-04-14 13:23:22.622 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 57, in import_module
  2015-04-14 13:23:22.622 TRACE nova     __import__(import_str)
  2015-04-14 13:23:22.622 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/evzookeeper/__init__.py", line 26, in <module>
  2015-04-14 13:23:22.622 TRACE nova     from evzookeeper import utils
  2015-04-14 13:23:22.622 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/evzookeeper/utils.py", line 26, in <module>
  2015-04-14 13:23:22.622 TRACE nova     class _SocketDuckForFdTimeout(greenio._SocketDuckForFd):
  2015-04-14 13:23:22.622 TRACE nova AttributeError: 'module' object has no attribute '_SocketDuckForFd'

  The root cause of the problem is the change, which have a place in module
  eventlet 0.17, which evzookeeper module is depend on. Because of the change the
  way eventlet.greenio is exposed there is no way to reach the class
  _SocketDuckForFd other way than explicitly importing it via
  eventlet.greenio.py2 module.

  Definitely solution for this problem might go upstream to the evzookeeper
  author, however the development of evzookeeper module seems stalled (no
  activity for last 2 years), maybe it's worth to consider changing the zk driver
  implementation to use different zk module (kazoo seems quite active).

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1443910/+subscriptions
Follow ups

[Bug 1443910] Re: Zookeeper servicegroup driver crashes
From: OpenStack Infra, 2016-02-22
[Bug 1443910] [NEW] Zookeeper servicegroup driver crashes
From: Roman Dobosz, 2015-04-14
References

[Bug 1443910] [NEW] Zookeeper servicegroup driver crashes
From: Roman Dobosz, 2015-04-14