yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1169057] Re: fixed ip re-used if instance can't be deleted with libvirtError raised

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Alan Pevec <1169057@xxxxxxxxxxxxxxxxxx>
Date: Thu, 08 Aug 2013 19:53:53 -0000
Reply-to: Bug 1169057 <1169057@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova/grizzly
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1169057

Title:
  fixed ip re-used if instance can't be deleted with libvirtError raised

Status in OpenStack Compute (Nova):
  Fix Released
Status in OpenStack Compute (nova) folsom series:
  Confirmed
Status in OpenStack Compute (nova) grizzly series:
  Fix Released

Bug description:
  the libvirt version is 0.9.12-5 with debian patchs, and stable folsom nova.
  "libvirtError: Timed out during operation: cannot acquire state change lock" was raised when I delete an instance, the strace log is:
  2013-04-07 10:03:27 ERROR nova.openstack.common.rpc.amqp [-] Exception during message handling
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp   File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 275, in _process_data
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt, version, method, **args)
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp   File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 145, in dispatch
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp   File "/usr/local/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp     temp_level, payload)
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp   File "/usr/local/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp     return f(*args, **kw)
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp   File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 325, in decorated_function
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp     kwargs['instance']['uuid'], e, sys.exc_info())
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp   File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 318, in decorated_function
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp   File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 1555, in terminate_instance
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp     do_terminate_instance(instance, del_ebs)
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp   File "/usr/local/lib/python2.7/dist-packages/nova/utils.py", line 845, in inner
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp     retval = f(*args, **kwargs)
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp   File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 1547, in do_terminate_instance
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp     self._delete_instance(context, instance, del_ebs)
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp   File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 1505, in _delete_instance
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp     self._shutdown_instance(context, instance, del_ebs)
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp   File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 1451, in _shutdown_instance
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp     block_device_info)
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp   File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 574, in destroy
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp     self._destroy(instance)
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp   File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 514, in _destroy
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp     virt_dom.destroy()
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in proxy_call
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp     rv = execute(f,*args,**kwargs)
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp     rv = meth(*args,**kwargs)
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 606, in destroy
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp     if ret == -1: raise libvirtError ('virDomainDestroy() failed', dom=self)
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp libvirtError: Timed out during operation: cannot acquire state change lock
  2013-04-07 10:03:27 TRACE nova.openstack.common.rpc.amqp

  this instance can't be deleted with the same error even I deleted it
  several times and stick in deleting task_state, and then the fixed ip
  is used by another instance.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1169057/+subscriptions