yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1817670] Re: nova-compute could not destroy evacuated instance which was deleted

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Tue, 26 Feb 2019 14:48:21 -0000
Reply-to: Bug 1817670 <1817670@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
*** This bug is a duplicate of bug 1794996 ***
    https://bugs.launchpad.net/bugs/1794996

This might already be fixed for bug 1794996:

https://review.openstack.org/#/q/I1f4b3540dd453650f94333b36d7504ba164192f7

But that was not backported to stable/ocata.

If that fixes your issue we could work on backporting
https://review.openstack.org/#/q/topic:bug/1550919+branch:stable/pike to
stable/ocata.

** This bug has been marked a duplicate of bug 1794996
   _destroy_evacuated_instances fails and kills n-cpu startup if lazy-loading flavor on a deleted instance

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1817670

Title:
  nova-compute could not destroy evacuated instance which was deleted

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  After evacuation, nova-compute where VMs were evacuated could not restart emitting InstanceNotFound exception when init_host() called.

  
  Steps to reproduce
  ==================
  0) service nova-compute stop at 'compute01'
  1) nova evacuate 'compute01'
  2) nova delete 'server at compute01'
  3) service nova-compute restart at 'compute01'

  
  Expected result
  ===============
  nova-compute normally started

  Actual result
  =============
  nova-compute could not started

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
    list for all releases: http://docs.openstack.org/releases/

    stable/ocata

  2. Which hypervisor did you use?
     (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
     What's the version of that?
   
    libvirt + kvm

  2. Which storage type did you use?
     (For example: Ceph, LVM, GPFS, ...)
     What's the version of that?

     Ceph

  
  3. Which networking type did you use?
     (For example: nova-network, Neutron with OpenVSwitch, ...)
     neutron with linux bridge   

  
  Logs & Configs
  ==============
  2019-02-26 16:24:29.015 8901 INFO nova.virt.libvirt.driver [req-dc2a5061-29aa-4097-9a3e-83de36804460 - - - - -] [instance: 25ab917a-1506-4154-9902-5b8b550ec929] Deleting instance files /var/lib/nova/instances/25ab917a-1506-4154-9902-5b8b550ec929_del
  2019-02-26 16:24:29.017 8901 INFO nova.virt.libvirt.driver [req-dc2a5061-29aa-4097-9a3e-83de36804460 - - - - -] [instance: 25ab917a-1506-4154-9902-5b8b550ec929] Deletion of /var/lib/nova/instances/25ab917a-1506-4154-9902-5b8b550ec929_del complete
  2019-02-26 16:24:29.020 8901 DEBUG oslo_messaging._drivers.amqpdriver [req-dc2a5061-29aa-4097-9a3e-83de36804460 - - - - -] CALL msg_id: 9bbf4371e977450d98a519dac213592b exchange 'nova' topic 'conductor' _send /opt/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:550
  2019-02-26 16:24:29.085 8901 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 9bbf4371e977450d98a519dac213592b __call__ /opt/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:407
  2019-02-26 16:24:29.093 8901 DEBUG oslo_messaging._drivers.amqpdriver [req-dc2a5061-29aa-4097-9a3e-83de36804460 - - - - -] CAST unique_id: d5a0d3b93fbf42b6a379b9507e9c2a9a FANOUT topic 'scheduler' _send /opt/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:539
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service [req-dc2a5061-29aa-4097-9a3e-83de36804460 - - - - -] Error starting thread.
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service Traceback (most recent call last):
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service   File "/opt/openstack/lib/python2.7/site-packages/oslo_service/service.py", line 722, in run_service
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service     service.start()
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service   File "/opt/openstack/src/nova/nova/service.py", line 144, in start
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service     self.manager.init_host()
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service   File "/opt/openstack/src/nova/nova/compute/manager.py", line 1155, in init_host
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service     self._destroy_evacuated_instances(context)
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service   File "/opt/openstack/src/nova/nova/compute/manager.py", line 674, in _destroy_evacuated_instances
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service     bdi, destroy_disks)
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service   File "/opt/openstack/src/nova/nova/virt/libvirt/driver.py", line 915, in destroy
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service     destroy_disks, migrate_data)
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service   File "/opt/openstack/src/nova/nova/virt/libvirt/driver.py", line 1034, in cleanup
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service     instance.save()
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service   File "/opt/openstack/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 210, in wrapper
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service     ctxt, self, fn.__name__, args, kwargs)
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service   File "/opt/openstack/src/nova/nova/conductor/rpcapi.py", line 244, in object_action
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service     objmethod=objmethod, args=args, kwargs=kwargs)
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service   File "/opt/openstack/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service     retry=self.retry)
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service   File "/opt/openstack/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service     timeout=timeout, retry=retry)
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service   File "/opt/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service     retry=retry)
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service   File "/opt/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 557, in _send
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service     raise result
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service InstanceNotFound_Remote: Instance 25ab917a-1506-4154-9902-5b8b550ec929 could not be found.
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service Traceback (most recent call last):
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service   File "/opt/openstack/src/nova/nova/conductor/manager.py", line 93, in _object_dispatch
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service     return getattr(target, method)(*args, **kwargs)
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service   File "/opt/openstack/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service     return fn(self, *args, **kwargs)
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service   File "/opt/openstack/src/nova/nova/objects/instance.py", line 792, in save
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service     columns_to_join=_expected_cols(expected_attrs))
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service   File "/opt/openstack/src/nova/nova/db/api.py", line 840, in instance_update_and_get_original
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service     expected=expected)
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service   File "/opt/openstack/src/nova/nova/db/sqlalchemy/api.py", line 180, in wrapper
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service     return f(*args, **kwargs)
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service   File "/opt/openstack/lib/python2.7/site-packages/oslo_db/api.py", line 151, in wrapper
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service     ectxt.value = e.inner_exc
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service   File "/opt/openstack/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service   File "/opt/openstack/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service   File "/opt/openstack/lib/python2.7/site-packages/oslo_db/api.py", line 139, in wrapper
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service     return f(*args, **kwargs)
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service   File "/opt/openstack/src/nova/nova/db/sqlalchemy/api.py", line 251, in wrapped
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service     return f(context, *args, **kwargs)
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service   File "/opt/openstack/src/nova/nova/db/sqlalchemy/api.py", line 2714, in instance_update_and_get_original
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service     columns_to_join=columns_to_join)
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service   File "/opt/openstack/src/nova/nova/db/sqlalchemy/api.py", line 1910, in _instance_get_by_uuid
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service     raise exception.InstanceNotFound(instance_id=uuid)
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service InstanceNotFound: Instance 25ab917a-1506-4154-9902-5b8b550ec929 could not be found.
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
  2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
  2019-02-26 16:24:29.102 8901 DEBUG oslo_concurrency.lockutils [req-67496f45-7809-41c4-a6a1-365574533016 - - - - -] Acquired semaphore "singleton_lock" lock /opt/openstack/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212
  2019-02-26 16:24:29.103 8901 DEBUG oslo_concurrency.lockutils [req-67496f45-7809-41c4-a6a1-365574533016 - - - - -] Releasing semaphore "singleton_lock" lock /opt/openstack/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:225

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1817670/+subscriptions
References

[Bug 1817670] [NEW] nova-compute could not destroy evacuated instance which was deleted
From: Yang Youseok, 2019-02-26