openstack team mailing list archive
Mailing list archive
Re: Memory leaks from greenthreads
On 02/29/2012 09:08 PM, Johannes Erdfelt wrote:
> On Wed, Feb 29, 2012, Vishvananda Ishaya <vishvananda@xxxxxxxxx> wrote:
>> We have had a memory leak due to an interaction with eventlet for a
>> while that Johannes has just made a fix for.
>> Unfortuantely, I don' t think we have a decent workaround for nova
>> while that patch is upstreamed. I wanted to make sure that all of
>> the distros are aware of it in case they want to carry an eventlet
>> patch to prevent the slow memory leak.
> There is one other possible workaround, but I didn't feel like it was
> safe since we would likely need to audit all of the third party libraries
> to ensure they don't cause problems.
> The memory leak only happens when monkey patching the
> thread/threading/Queue modules. I looked at the nova sources and did
> some tests and it doesn't appear nova needs those modules patches.
> However, third party modules might need it. Also, I only tested on
> xenapi. libvirt and/or vmwareapi might have problems. Or possibly other
> drivers (firewall, volume, etc) in nova that I didn't use in my tests.
> If you're having problems with the memory leak in eventlet and applying
> the patch isn't an option, then monkey patching everything but thread
> might something worth trying.
> eventlet.monkey_patch(os=True, socket=True, time=True)
Note I tried the patch but got errors
# rpm -qa python-greenlet python-eventlet
# /usr/bin/nova-cert --flagfile /dev/null --config-file /etc/nova/nova.conf --logfile /var/log/nova/cert.log
2012-03-05 15:09:09 AUDIT nova.service [-] Starting cert node (version 2012.1-LOCALBRANCH:LOCALREVISION)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 336, in fire_timers
File "/usr/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 56, in __call__
File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 192, in main
result = function(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/nova/service.py", line 101, in run_server
File "/usr/lib/python2.7/site-packages/nova/service.py", line 176, in start
self.conn = rpc.create_connection(new=True)
File "/usr/lib/python2.7/site-packages/nova/rpc/__init__.py", line 47, in create_connection
File "/usr/lib/python2.7/site-packages/nova/rpc/impl_qpid.py", line 507, in create_connection
return rpc_amqp.create_connection(new, Connection.pool)
File "/usr/lib/python2.7/site-packages/nova/rpc/amqp.py", line 310, in create_connection
return ConnectionContext(connection_pool, pooled=not new)
File "/usr/lib/python2.7/site-packages/nova/rpc/amqp.py", line 84, in __init__
File "/usr/lib/python2.7/site-packages/nova/rpc/impl_qpid.py", line 294, in __init__
self.connection = qpid.messaging.Connection(self.broker)
File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 178, in __init__
self._driver = Driver(self)
File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 347, in __init__
self._selector = Selector.default()
File "/usr/lib/python2.7/site-packages/qpid/selector.py", line 54, in default
File "/usr/lib/python2.7/site-packages/qpid/selector.py", line 99, in start
self.thread = Thread(target=self.run)
File "/usr/lib64/python2.7/threading.py", line 446, in __init__
self.__daemonic = self._set_daemon()
File "/usr/lib64/python2.7/threading.py", line 470, in _set_daemon
AttributeError: '_GreenThread' object has no attribute 'daemon'
2012-03-05 15:09:10 CRITICAL nova [-] '_GreenThread' object has no attribute 'daemon'
Exception KeyError: KeyError(139994820976048,) in <module 'threading' from '/usr/lib64/python2.7/threading.pyc'> ignored
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
File "/usr/lib/python2.7/site-packages/qpid/selector.py", line 138, in stop
AttributeError: 'NoneType' object has no attribute 'join'
Error in sys.exitfunc:
2012-03-05 15:09:10 CRITICAL nova [-] 'NoneType' object has no attribute 'join'