yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2083503] [NEW] Nova fails to delete a server; KeyError: AsyncDeviceEventsHandler.Waiter ... event_types={<class 'nova.virt.libvirt.event.DeviceRemovalFailedEvent'>, <class 'nova.virt.libvirt.event.DeviceRemovedEvent'>}) in n-cpu logs

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Ihar Hrachyshka <2083503@xxxxxxxxxxxxxxxxxx>
Date: Wed, 02 Oct 2024 12:42:01 -0000
Reply-to: Bug 2083503 <2083503@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Failed here:
https://084ffa20b2af70b3ecc4-dc7530dd32f75128221ba69b2574d743.ssl.cf2.rackcdn.com/930608/3/gate/neutron-
tempest-plugin-ovn-enforce-scope-old-defaults/5c6d504/testr_results.html

The API replied with 500 and KeyError in status:

Body: b'{"server": {"id": "27037e69-e14d-4a72-9348-6e9ce8ad7da3",
"name": "tempest-server-test-1781250568", "status": "ERROR",
"tenant_id": "02d606921ce643fe96d619e5ef472e16", "user_id":
"70d4b28014894d5fa979e2b92f318a28", "metadata": {}, "hostId":
"f2e3e995fe9a0dc59dbfe644042aaa2388de91f7643899cfbb66fb59", "image":
{"id": "076d7543-348f-44cc-a39f-bab79e050305", "links": [{"rel":
"bookmark", "href":
"https://199.19.213.7/compute/images/076d7543-348f-44cc-a39f-bab79e050305"}]},
"flavor": {"id": "42", "links": [{"rel": "bookmark", "href":
"https://199.19.213.7/compute/flavors/42"}]}, "created":
"2024-10-02T09:48:58Z", "updated": "2024-10-02T09:50:56Z", "addresses":
{"tempest-test-network--1177851110": [{"version": 6, "addr":
"2001:db8:0:2:f816:3eff:fef2:b72b", "OS-EXT-IPS:type": "fixed", "OS-EXT-
IPS-MAC:mac_addr": "fa:16:3e:f2:b7:2b"}]}, "accessIPv4": "",
"accessIPv6": "", "links": [{"rel": "self", "href":
"https://199.19.213.7/compute/v2.1/servers/27037e69-e14d-4a72-9348-6e9ce8ad7da3"},
{"rel": "bookmark", "href":
"https://199.19.213.7/compute/servers/27037e69-e14d-4a72-9348-6e9ce8ad7da3"}],
"OS-DCF:diskConfig": "MANUAL", "fault": {"code": 500, "created":
"2024-10-02T09:49:46Z", "message": "KeyError"}, "OS-EXT-
AZ:availability_zone": "nova", "config_drive": "", "key_name": "tempest-
keypair-test-913516318", "OS-SRV-USG:launched_at":
"2024-10-02T09:49:15.000000", "OS-SRV-USG:terminated_at": null, "OS-EXT-
STS:task_state": "deleting", "OS-EXT-STS:vm_state": "error", "OS-EXT-
STS:power_state": 1, "os-extended-volumes:volumes_attached": []}}'

n-cpu log here:
https://084ffa20b2af70b3ecc4-dc7530dd32f75128221ba69b2574d743.ssl.cf2.rackcdn.com/930608/3/gate/neutron-
tempest-plugin-ovn-enforce-scope-old-
defaults/5c6d504/controller/logs/screen-n-cpu.txt

The relevant traceback is probably:

Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server [None req-e8e2e40b-0bb9-4da5-ae52-ea2fe52c8846 tempest-IPv6Test-1538994518 tempest-IPv6Test-1538994518-project-member] Exception during message handling: KeyError: AsyncDeviceEventsHandler.Waiter(instance_uuid=18ccb606-0b6a-4464-9f49-a36ffc815fe8, device_name=net1, event_types={<class 'nova.virt.libvirt.event.DeviceRemovalFailedEvent'>, <class 'nova.virt.libvirt.event.DeviceRemovedEvent'>})
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 172, in _process_incoming
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 65, in wrapped
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     self.force_reraise()
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     raise self.value
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 63, in wrapped
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/utils.py", line 1453, in decorated_function
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 213, in decorated_function
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     self.force_reraise()
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     raise self.value
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 203, in decorated_function
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 8391, in detach_interface
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     do_detach_interface(context, instance, port_id)
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py", line 412, in inner
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 8389, in do_detach_interface
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     self._detach_interface(context, instance, port_id)
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 8432, in _detach_interface
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     self.driver.detach_interface(context, instance, condemned)
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3109, in detach_interface
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     self._detach_with_retry(
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2607, in _detach_with_retry
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     self._detach_from_live_with_retry(
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2663, in _detach_from_live_with_retry
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     self._detach_from_live_and_wait_for_event(
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2741, in _detach_from_live_and_wait_for_event
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     event = self._device_event_handler.wait(
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 368, in wait
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     self._waiters.remove(token)
Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server KeyError: AsyncDeviceEventsHandler.Waiter(instance_uuid=18ccb606-0b6a-4464-9f49-a36ffc815fe8, device_name=net1, event_types={<class 'nova.virt.libvirt.event.DeviceRemovalFailedEvent'>, <class 'nova.virt.libvirt.event.DeviceRemovedEvent'>})

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: gate-failure

** Tags added: gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2083503

Title:
  Nova fails to delete a server; KeyError:
  AsyncDeviceEventsHandler.Waiter ... event_types={<class
  'nova.virt.libvirt.event.DeviceRemovalFailedEvent'>, <class
  'nova.virt.libvirt.event.DeviceRemovedEvent'>}) in n-cpu logs

Status in OpenStack Compute (nova):
  New

Bug description:
  Failed here:
  https://084ffa20b2af70b3ecc4-dc7530dd32f75128221ba69b2574d743.ssl.cf2.rackcdn.com/930608/3/gate/neutron-
  tempest-plugin-ovn-enforce-scope-old-
  defaults/5c6d504/testr_results.html

  The API replied with 500 and KeyError in status:

  Body: b'{"server": {"id": "27037e69-e14d-4a72-9348-6e9ce8ad7da3",
  "name": "tempest-server-test-1781250568", "status": "ERROR",
  "tenant_id": "02d606921ce643fe96d619e5ef472e16", "user_id":
  "70d4b28014894d5fa979e2b92f318a28", "metadata": {}, "hostId":
  "f2e3e995fe9a0dc59dbfe644042aaa2388de91f7643899cfbb66fb59", "image":
  {"id": "076d7543-348f-44cc-a39f-bab79e050305", "links": [{"rel":
  "bookmark", "href":
  "https://199.19.213.7/compute/images/076d7543-348f-44cc-a39f-bab79e050305"}]},
  "flavor": {"id": "42", "links": [{"rel": "bookmark", "href":
  "https://199.19.213.7/compute/flavors/42"}]}, "created":
  "2024-10-02T09:48:58Z", "updated": "2024-10-02T09:50:56Z",
  "addresses": {"tempest-test-network--1177851110": [{"version": 6,
  "addr": "2001:db8:0:2:f816:3eff:fef2:b72b", "OS-EXT-IPS:type":
  "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:f2:b7:2b"}]},
  "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href":
  "https://199.19.213.7/compute/v2.1/servers/27037e69-e14d-4a72-9348-6e9ce8ad7da3"},
  {"rel": "bookmark", "href":
  "https://199.19.213.7/compute/servers/27037e69-e14d-4a72-9348-6e9ce8ad7da3"}],
  "OS-DCF:diskConfig": "MANUAL", "fault": {"code": 500, "created":
  "2024-10-02T09:49:46Z", "message": "KeyError"}, "OS-EXT-
  AZ:availability_zone": "nova", "config_drive": "", "key_name":
  "tempest-keypair-test-913516318", "OS-SRV-USG:launched_at":
  "2024-10-02T09:49:15.000000", "OS-SRV-USG:terminated_at": null, "OS-
  EXT-STS:task_state": "deleting", "OS-EXT-STS:vm_state": "error", "OS-
  EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": []}}'

  n-cpu log here:
  https://084ffa20b2af70b3ecc4-dc7530dd32f75128221ba69b2574d743.ssl.cf2.rackcdn.com/930608/3/gate/neutron-
  tempest-plugin-ovn-enforce-scope-old-
  defaults/5c6d504/controller/logs/screen-n-cpu.txt

  The relevant traceback is probably:

  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server [None req-e8e2e40b-0bb9-4da5-ae52-ea2fe52c8846 tempest-IPv6Test-1538994518 tempest-IPv6Test-1538994518-project-member] Exception during message handling: KeyError: AsyncDeviceEventsHandler.Waiter(instance_uuid=18ccb606-0b6a-4464-9f49-a36ffc815fe8, device_name=net1, event_types={<class 'nova.virt.libvirt.event.DeviceRemovalFailedEvent'>, <class 'nova.virt.libvirt.event.DeviceRemovedEvent'>})
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 172, in _process_incoming
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 65, in wrapped
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     raise self.value
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 63, in wrapped
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/utils.py", line 1453, in decorated_function
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 213, in decorated_function
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     raise self.value
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 203, in decorated_function
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 8391, in detach_interface
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     do_detach_interface(context, instance, port_id)
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py", line 412, in inner
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 8389, in do_detach_interface
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     self._detach_interface(context, instance, port_id)
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 8432, in _detach_interface
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     self.driver.detach_interface(context, instance, condemned)
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3109, in detach_interface
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     self._detach_with_retry(
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2607, in _detach_with_retry
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     self._detach_from_live_with_retry(
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2663, in _detach_from_live_with_retry
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     self._detach_from_live_and_wait_for_event(
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2741, in _detach_from_live_and_wait_for_event
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     event = self._device_event_handler.wait(
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 368, in wait
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server     self._waiters.remove(token)
  Oct 02 09:51:45.626514 np0038682333 nova-compute[68667]: ERROR oslo_messaging.rpc.server KeyError: AsyncDeviceEventsHandler.Waiter(instance_uuid=18ccb606-0b6a-4464-9f49-a36ffc815fe8, device_name=net1, event_types={<class 'nova.virt.libvirt.event.DeviceRemovalFailedEvent'>, <class 'nova.virt.libvirt.event.DeviceRemovedEvent'>})

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2083503/+subscriptions