← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #05533

[Bug 1202484] Re: Instance backup dir isn't deleted if instance is deleted during confirming resize

  Thread Previous • Date Previous • • Thread Next 
** Changed in: nova
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1202484

Title:
  Instance backup dir isn't deleted if instance is deleted during
  confirming resize

Status in OpenStack Compute (Nova):
  Fix Released

Bug description:
  Reproduce steps with latest master branch of nova(but I believe
  grizzly, folsom also have this bug, and this occurs on our folsom
  stable product env):

  1.hack the nova-compute code as follow:
  diff --git a/nova/compute/manager.py b/nova/compute/manager.py
  index e1302bf..5a82f0b 100755
  --- a/nova/compute/manager.py
  +++ b/nova/compute/manager.py
  @@ -2399,7 +2399,7 @@ class ComputeManager(manager.SchedulerDependentManager):

           self._notify_about_instance_usage(context, instance,
                                             "resize.confirm.start")
  -
  +        LOG.debug(_("--------------------sleep 60s now------------------"))
  +        time.sleep(60)   # sleep here and wait for the instance to be deleted
           with self._error_out_instance_on_exception(context, instance['uuid'],
                                                      reservations):
               # NOTE(danms): delete stashed migration information
  2. create an instance under devstack env
  3. resize it
  4. confirm the resize operation
  5. delete the confirming instance when you saw the hacked log message 'sleep 60s now'
  6. then the instance will be deleted and the instance backup dir 'ddb764e6-c07a-4e81-8347-283213b84a98_resize' will not be deleted.

  the error log is:
  2013-07-18 03:19:14.714 DEBUG nova.compute.manager [req-067c64aa-313f-49fc-b47b-2e83391f72ad admin admin] [instance: ddb764e6-c07a-4e81-8347-283213b84a98] Instance has been destroyed from under us while trying to set it to ERROR from (pid=7552) _set_instance_error_state /opt/stack/nova/nova/compute/manager.py:430
  2013-07-18 03:19:14.717 DEBUG nova.openstack.common.rpc.amqp [req-067c64aa-313f-49fc-b47b-2e83391f72ad admin admin] Making synchronous call on conductor ... from (pid=7552) multicall /opt/stack/nova/nova/openstack/common/rpc/amqp.py:518
  2013-07-18 03:19:14.718 DEBUG nova.openstack.common.rpc.amqp [req-067c64aa-313f-49fc-b47b-2e83391f72ad admin admin] MSG_ID is c57691619c834285bbbf786f16e01a5a from (pid=7552) multicall /opt/stack/nova/nova/openstack/common/rpc/amqp.py:521
  2013-07-18 03:19:14.718 DEBUG nova.openstack.common.rpc.amqp [req-067c64aa-313f-49fc-b47b-2e83391f72ad admin admin] UNIQUE_ID is a1e36d387f0045e5bc9c67bfe4dbdf6a. from (pid=7552) _add_unique_id /opt/stack/nova/nova/openstack/common/rpc/amqp.py:327
  2013-07-18 03:19:14.751 ERROR nova.openstack.common.rpc.amqp [req-067c64aa-313f-49fc-b47b-2e83391f72ad admin admin] Exception during message handling
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 426, in _process_data
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp     **args)
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/nova/nova/exception.py", line 99, in wrapped
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp     temp_level, payload)
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/nova/nova/exception.py", line 76, in wrapped
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/nova/nova/compute/manager.py", line 279, in decorated_function
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/nova/nova/compute/manager.py", line 243, in decorated_function
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/nova/nova/compute/manager.py", line 2410, in confirm_resize
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp     system_metadata=sys_meta)
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/nova/nova/compute/manager.py", line 414, in _instance_update
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp     **kwargs)
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/nova/nova/conductor/api.py", line 414, in instance_update
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp     updates, 'conductor')
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/nova/nova/conductor/rpcapi.py", line 134, in instance_update
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp     version='1.38')
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/nova/nova/openstack/common/rpc/proxy.py", line 126, in call
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp     result = rpc.call(context, real_topic, msg, timeout)
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/nova/nova/openstack/common/rpc/__init__.py", line 140, in call
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp     return _get_impl().call(CONF, context, topic, msg, timeout)
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 824, in call
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp     rpc_amqp.get_connection_pool(conf, Connection))
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 539, in call
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp     rv = list(rv)
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 504, in __iter__
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp     raise result
  2013-07-18 03:19:14.751 TRACE nova.openstack.common.rpc.amqp InstanceNotFound_Remote: Instance ddb764e6-c07a-4e81-8347-283213b84a98 could not be found.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1202484/+subscriptions
  Thread Previous • Date Previous • • Thread Next