openstack team mailing list archive

[Pádraig Brady <P@xxxxxxxxxxxxxx>]

On 02/29/2012 09:08 PM, Johannes Erdfelt wrote:
> On Wed, Feb 29, 2012, Vishvananda Ishaya <vishvananda@xxxxxxxxx> wrote:
>> We have had a memory leak due to an interaction with eventlet for a
>> while that Johannes has just made a fix for.
>>
>> bug:
>> https://bugs.launchpad.net/nova/+bug/903199
>>
>> fix:
>> https://bitbucket.org/which_linden/eventlet/pull-request/10/monkey-patch-threadingcurrent_thread-as
>>
>> Unfortuantely, I don' t think we have a decent workaround for nova
>> while that patch is upstreamed.  I wanted to make sure that all of
>> the distros are aware of it in case they want to carry an eventlet
>> patch to prevent the slow memory leak.
> 
> There is one other possible workaround, but I didn't feel like it was
> safe since we would likely need to audit all of the third party libraries
> to ensure they don't cause problems.
> 
> The memory leak only happens when monkey patching the
> thread/threading/Queue modules. I looked at the nova sources and did
> some tests and it doesn't appear nova needs those modules patches.
> 
> However, third party modules might need it. Also, I only tested on
> xenapi. libvirt and/or vmwareapi might have problems. Or possibly other
> drivers (firewall, volume, etc) in nova that I didn't use in my tests.
> 
> If you're having problems with the memory leak in eventlet and applying
> the patch isn't an option, then monkey patching everything but thread
> might something worth trying.
> 
> eventlet.monkey_patch(os=True, socket=True, time=True)

Note I tried the patch but got errors

# rpm -qa python-greenlet python-eventlet
python-eventlet-0.9.16-4.fc17.noarch
python-greenlet-0.3.1-9.fc17.x86_64

# /usr/bin/nova-cert --flagfile /dev/null --config-file /etc/nova/nova.conf --logfile /var/log/nova/cert.log
2012-03-05 15:09:09 AUDIT nova.service [-] Starting cert node (version 2012.1-LOCALBRANCH:LOCALREVISION)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 336, in fire_timers
    timer()
  File "/usr/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 56, in __call__
    cb(*args, **kw)
  File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 192, in main
    result = function(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/nova/service.py", line 101, in run_server
    server.start()
  File "/usr/lib/python2.7/site-packages/nova/service.py", line 176, in start
    self.conn = rpc.create_connection(new=True)
  File "/usr/lib/python2.7/site-packages/nova/rpc/__init__.py", line 47, in create_connection
    return _get_impl().create_connection(new=new)
  File "/usr/lib/python2.7/site-packages/nova/rpc/impl_qpid.py", line 507, in create_connection
    return rpc_amqp.create_connection(new, Connection.pool)
  File "/usr/lib/python2.7/site-packages/nova/rpc/amqp.py", line 310, in create_connection
    return ConnectionContext(connection_pool, pooled=not new)
  File "/usr/lib/python2.7/site-packages/nova/rpc/amqp.py", line 84, in __init__
    server_params=server_params)
  File "/usr/lib/python2.7/site-packages/nova/rpc/impl_qpid.py", line 294, in __init__
    self.connection = qpid.messaging.Connection(self.broker)
  File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 178, in __init__
    self._driver = Driver(self)
  File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 347, in __init__
    self._selector = Selector.default()
  File "/usr/lib/python2.7/site-packages/qpid/selector.py", line 54, in default
    sel.start()
  File "/usr/lib/python2.7/site-packages/qpid/selector.py", line 99, in start
    self.thread = Thread(target=self.run)
  File "/usr/lib64/python2.7/threading.py", line 446, in __init__
    self.__daemonic = self._set_daemon()
  File "/usr/lib64/python2.7/threading.py", line 470, in _set_daemon
    return current_thread().daemon
AttributeError: '_GreenThread' object has no attribute 'daemon'
2012-03-05 15:09:10 CRITICAL nova [-] '_GreenThread' object has no attribute 'daemon'
Exception KeyError: KeyError(139994820976048,) in <module 'threading' from '/usr/lib64/python2.7/threading.pyc'> ignored
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.7/site-packages/qpid/selector.py", line 138, in stop
    self.thread.join(timeout)
AttributeError: 'NoneType' object has no attribute 'join'
Error in sys.exitfunc:
2012-03-05 15:09:10 CRITICAL nova [-] 'NoneType' object has no attribute 'join'

cheers,
Pádraig.