yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1745977] [NEW] When source compute service up, will not destroy and clean up those instances which be evacuated then be deleted.

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: jiangyuhao <jiang.yuhao@xxxxxxxxxx>
Date: Mon, 29 Jan 2018 12:03:10 -0000
Reply-to: Bug 1745977 <1745977@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========
When the instance evacuated to destination host successfully, then deleted this instance. The source host up will cleanup this instance failed.

Steps to reproduce
==================
1.deploy a local instance in source host.
2.power off the source host.
3.evacuate the instance to destination host.
4.delete this instance.
5.power on the source host.

Expected result
===============
The source host nova-compute service cleanup this evacuated and deleted instance.

Actual result
=============
This instance still on source host.

Environment
===========
Openstack Pike
Libvirt + KVM
ovs network


Logs & Configs
==============
source host nova-compute log:

2018-01-29 10:28:48.664 9364 ERROR oslo_service.service [req-7bdfe28f-0464-4af8-bdd0-2d433b25d84a - - - - -] Error starting thread.: InstanceNotFound_Remote: Instance 19022200-7abc-423d-90bd-e9dcd0887679 could not be found.
Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 125, in _object_dispatch
    return getattr(target, method)(*args, **kwargs)

  File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
    result = fn(cls, context, *args, **kwargs)

  File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 474, in get_by_uuid
    use_slave=use_slave)

  File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 235, in wrapper
    return f(*args, **kwargs)

  File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 466, in _db_instance_get_by_uuid
    columns_to_join=columns_to_join)

  File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 744, in instance_get_by_uuid
    return IMPL.instance_get_by_uuid(context, uuid, columns_to_join)

  File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 179, in wrapper
    return f(*args, **kwargs)

  File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 280, in wrapped
    return f(context, *args, **kwargs)

  File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1911, in instance_get_by_uuid
    columns_to_join=columns_to_join)

  File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1920, in _instance_get_by_uuid
    raise exception.InstanceNotFound(instance_id=uuid)

InstanceNotFound: Instance 19022200-7abc-423d-90bd-e9dcd0887679 could not be found.
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service Traceback (most recent call last):
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 721, in run_service
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     service.start()
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/service.py", line 156, in start
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     self.manager.init_host()
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1173, in init_host
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     self._destroy_evacuated_instances(context)
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 691, in _destroy_evacuated_instances
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     bdi, destroy_disks)
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 909, in destroy
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     destroy_disks)
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1032, in cleanup
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     attempts = int(instance.system_metadata.get('clean_attempts',
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 67, in getter
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     self.obj_load_attr(name)
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1131, in obj_load_attr
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     self._load_generic(attrname)
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 858, in _load_generic
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     expected_attrs=[attrname])
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 177, in wrapper
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     args, kwargs)
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     args=args, kwargs=kwargs)
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 167, in call
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     retry=self.retry)
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 123, in _send
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     timeout=timeout, retry=retry)
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     retry=retry)
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 557, in _send
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     raise result
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service InstanceNotFound_Remote: Instance 19022200-7abc-423d-90bd-e9dcd0887679 could not be found.
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service Traceback (most recent call last):
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 125, in _object_dispatch
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     return getattr(target, method)(*args, **kwargs)
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     result = fn(cls, context, *args, **kwargs)
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 474, in get_by_uuid
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     use_slave=use_slave)
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 235, in wrapper
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     return f(*args, **kwargs)
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 466, in _db_instance_get_by_uuid
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     columns_to_join=columns_to_join)
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 744, in instance_get_by_uuid
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     return IMPL.instance_get_by_uuid(context, uuid, columns_to_join)
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 179, in wrapper
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     return f(*args, **kwargs)
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 280, in wrapped
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     return f(context, *args, **kwargs)
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1911, in instance_get_by_uuid
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     columns_to_join=columns_to_join)
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1920, in _instance_get_by_uuid
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     raise exception.InstanceNotFound(instance_id=uuid)
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
2018-01-29 10:28:48.664 9364 ERROR oslo_service.service InstanceNotFound: Instance 19022200-7abc-423d-90bd-e9dcd0887679 could not be found.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1745977

Title:
  When source compute service up, will not destroy and clean up those
  instances which be evacuated then be deleted.

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  When the instance evacuated to destination host successfully, then deleted this instance. The source host up will cleanup this instance failed.

  Steps to reproduce
  ==================
  1.deploy a local instance in source host.
  2.power off the source host.
  3.evacuate the instance to destination host.
  4.delete this instance.
  5.power on the source host.

  Expected result
  ===============
  The source host nova-compute service cleanup this evacuated and deleted instance.

  Actual result
  =============
  This instance still on source host.

  Environment
  ===========
  Openstack Pike
  Libvirt + KVM
  ovs network

  
  Logs & Configs
  ==============
  source host nova-compute log:

  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service [req-7bdfe28f-0464-4af8-bdd0-2d433b25d84a - - - - -] Error starting thread.: InstanceNotFound_Remote: Instance 19022200-7abc-423d-90bd-e9dcd0887679 could not be found.
  Traceback (most recent call last):

    File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 125, in _object_dispatch
      return getattr(target, method)(*args, **kwargs)

    File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
      result = fn(cls, context, *args, **kwargs)

    File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 474, in get_by_uuid
      use_slave=use_slave)

    File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 235, in wrapper
      return f(*args, **kwargs)

    File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 466, in _db_instance_get_by_uuid
      columns_to_join=columns_to_join)

    File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 744, in instance_get_by_uuid
      return IMPL.instance_get_by_uuid(context, uuid, columns_to_join)

    File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 179, in wrapper
      return f(*args, **kwargs)

    File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 280, in wrapped
      return f(context, *args, **kwargs)

    File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1911, in instance_get_by_uuid
      columns_to_join=columns_to_join)

    File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1920, in _instance_get_by_uuid
      raise exception.InstanceNotFound(instance_id=uuid)

  InstanceNotFound: Instance 19022200-7abc-423d-90bd-e9dcd0887679 could not be found.
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service Traceback (most recent call last):
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 721, in run_service
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     service.start()
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/service.py", line 156, in start
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     self.manager.init_host()
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1173, in init_host
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     self._destroy_evacuated_instances(context)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 691, in _destroy_evacuated_instances
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     bdi, destroy_disks)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 909, in destroy
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     destroy_disks)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1032, in cleanup
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     attempts = int(instance.system_metadata.get('clean_attempts',
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 67, in getter
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     self.obj_load_attr(name)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1131, in obj_load_attr
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     self._load_generic(attrname)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 858, in _load_generic
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     expected_attrs=[attrname])
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 177, in wrapper
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     args, kwargs)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     args=args, kwargs=kwargs)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 167, in call
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     retry=self.retry)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 123, in _send
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     timeout=timeout, retry=retry)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     retry=retry)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 557, in _send
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     raise result
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service InstanceNotFound_Remote: Instance 19022200-7abc-423d-90bd-e9dcd0887679 could not be found.
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service Traceback (most recent call last):
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 125, in _object_dispatch
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     return getattr(target, method)(*args, **kwargs)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     result = fn(cls, context, *args, **kwargs)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 474, in get_by_uuid
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     use_slave=use_slave)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 235, in wrapper
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     return f(*args, **kwargs)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 466, in _db_instance_get_by_uuid
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     columns_to_join=columns_to_join)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 744, in instance_get_by_uuid
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     return IMPL.instance_get_by_uuid(context, uuid, columns_to_join)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 179, in wrapper
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     return f(*args, **kwargs)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 280, in wrapped
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     return f(context, *args, **kwargs)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1911, in instance_get_by_uuid
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     columns_to_join=columns_to_join)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1920, in _instance_get_by_uuid
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     raise exception.InstanceNotFound(instance_id=uuid)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service InstanceNotFound: Instance 19022200-7abc-423d-90bd-e9dcd0887679 could not be found.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1745977/+subscriptions
Follow ups

[Bug 1745977] Re: When source compute service up, will not destroy and clean up those instances which be evacuated then be deleted.
From: OpenStack Infra, 2018-02-19
[Bug 1745977] Re: When source compute service up, will not destroy and clean up those instances which be evacuated then be deleted.
From: Matt Riedemann, 2018-02-19
[Bug 1745977] Re: When source compute service up, will not destroy and clean up those instances which be evacuated then be deleted.
From: Matt Riedemann, 2018-02-19