yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1745977] Re: When source compute service up, will not destroy and clean up those instances which be evacuated then be deleted.

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1745977@xxxxxxxxxxxxxxxxxx>
Date: Mon, 19 Feb 2018 22:54:51 -0000
Reply-to: Bug 1745977 <1745977@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/543970
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6ba8a35825a7ec839b2d0aab7559351d573130ab
Submitter: Zuul
Branch:    master

commit 6ba8a35825a7ec839b2d0aab7559351d573130ab
Author: Dan Smith <dansmith@xxxxxxxxxx>
Date:   Tue Feb 13 07:16:57 2018 -0800

    Lazy-load instance attributes with read_deleted=yes
    
    If we're doing a lazy-load of a generic attribute on instance, we
    should be using read_deleted=yes. Otherwise we just fail in the load
    process which is confusing and not helpful to a cleanup routine that
    needs to handle the deleted instance. This makes us load those things
    with read_deleted=yes.
    
    Change-Id: Ide6cc5bb1fce2c9aea9fa3efdf940e8308cd9ed0
    Closes-Bug: #1745977


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1745977

Title:
  When source compute service up, will not destroy and clean up those
  instances which be evacuated then be deleted.

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) ocata series:
  In Progress
Status in OpenStack Compute (nova) pike series:
  In Progress
Status in OpenStack Compute (nova) queens series:
  In Progress

Bug description:
  Description
  ===========
  When the instance evacuated to destination host successfully, then deleted this instance. The source host up will cleanup this instance failed.

  Steps to reproduce
  ==================
  1.deploy a local instance in source host.
  2.power off the source host.
  3.evacuate the instance to destination host.
  4.delete this instance.
  5.power on the source host.

  Expected result
  ===============
  The source host nova-compute service cleanup this evacuated and deleted instance.

  Actual result
  =============
  This instance still on source host.

  Environment
  ===========
  Openstack Pike
  Libvirt + KVM
  ovs network

  
  Logs & Configs
  ==============
  source host nova-compute log:

  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service [req-7bdfe28f-0464-4af8-bdd0-2d433b25d84a - - - - -] Error starting thread.: InstanceNotFound_Remote: Instance 19022200-7abc-423d-90bd-e9dcd0887679 could not be found.
  Traceback (most recent call last):

    File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 125, in _object_dispatch
      return getattr(target, method)(*args, **kwargs)

    File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
      result = fn(cls, context, *args, **kwargs)

    File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 474, in get_by_uuid
      use_slave=use_slave)

    File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 235, in wrapper
      return f(*args, **kwargs)

    File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 466, in _db_instance_get_by_uuid
      columns_to_join=columns_to_join)

    File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 744, in instance_get_by_uuid
      return IMPL.instance_get_by_uuid(context, uuid, columns_to_join)

    File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 179, in wrapper
      return f(*args, **kwargs)

    File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 280, in wrapped
      return f(context, *args, **kwargs)

    File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1911, in instance_get_by_uuid
      columns_to_join=columns_to_join)

    File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1920, in _instance_get_by_uuid
      raise exception.InstanceNotFound(instance_id=uuid)

  InstanceNotFound: Instance 19022200-7abc-423d-90bd-e9dcd0887679 could not be found.
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service Traceback (most recent call last):
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 721, in run_service
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     service.start()
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/service.py", line 156, in start
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     self.manager.init_host()
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1173, in init_host
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     self._destroy_evacuated_instances(context)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 691, in _destroy_evacuated_instances
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     bdi, destroy_disks)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 909, in destroy
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     destroy_disks)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1032, in cleanup
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     attempts = int(instance.system_metadata.get('clean_attempts',
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 67, in getter
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     self.obj_load_attr(name)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1131, in obj_load_attr
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     self._load_generic(attrname)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 858, in _load_generic
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     expected_attrs=[attrname])
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 177, in wrapper
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     args, kwargs)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     args=args, kwargs=kwargs)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 167, in call
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     retry=self.retry)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 123, in _send
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     timeout=timeout, retry=retry)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     retry=retry)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 557, in _send
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     raise result
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service InstanceNotFound_Remote: Instance 19022200-7abc-423d-90bd-e9dcd0887679 could not be found.
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service Traceback (most recent call last):
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 125, in _object_dispatch
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     return getattr(target, method)(*args, **kwargs)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     result = fn(cls, context, *args, **kwargs)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 474, in get_by_uuid
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     use_slave=use_slave)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 235, in wrapper
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     return f(*args, **kwargs)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 466, in _db_instance_get_by_uuid
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     columns_to_join=columns_to_join)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 744, in instance_get_by_uuid
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     return IMPL.instance_get_by_uuid(context, uuid, columns_to_join)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 179, in wrapper
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     return f(*args, **kwargs)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 280, in wrapped
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     return f(context, *args, **kwargs)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1911, in instance_get_by_uuid
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     columns_to_join=columns_to_join)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1920, in _instance_get_by_uuid
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service     raise exception.InstanceNotFound(instance_id=uuid)
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service
  2018-01-29 10:28:48.664 9364 ERROR oslo_service.service InstanceNotFound: Instance 19022200-7abc-423d-90bd-e9dcd0887679 could not be found.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1745977/+subscriptions
References

[Bug 1745977] [NEW] When source compute service up, will not destroy and clean up those instances which be evacuated then be deleted.
From: jiangyuhao, 2018-01-29