yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1928922] [NEW] positive evacuation testing in nova-live-migration fails with VirtualInterfaceCreateException

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Balazs Gibizer <1928922@xxxxxxxxxxxxxxxxxx>
Date: Wed, 19 May 2021 08:21:09 -0000
Reply-to: Bug 1928922 <1928922@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Example POST_FAILURE run
https://zuul.opendev.org/t/openstack/build/4d0d5072c1e0479db616a211e0afda42/logs


https://zuul.opendev.org/t/openstack/build/4d0d5072c1e0479db616a211e0afda42/log/controller/logs/screen-n-cpu.txt#13322

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [None req-
778f8336-8e29-4282-a112-91a876303fe3 demo admin] [instance: 021396f6
-40ff-434c-acbe-7092a6b1bcd9] Setting instance vm_state to ERROR:
nova.exception.VirtualInterfaceCreateException: Virtual Interface
creation failed

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9] Traceback (most recent call last):

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]   File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 7188, in
_create_guest_with_network

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]     guest = self._create_guest(

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]   File "/usr/lib/python3.8/contextlib.py", line
120, in __exit__

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]     next(self.gen)

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]   File
"/opt/stack/nova/nova/compute/manager.py", line 479, in
wait_for_instance_event

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]     actual_event = event.wait()

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]   File "/usr/local/lib/python3.8/dist-
packages/eventlet/event.py", line 125, in wait

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]     result = hub.switch()

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]   File "/usr/local/lib/python3.8/dist-
packages/eventlet/hubs/hub.py", line 313, in switch

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]     return self.greenlet.switch()

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9] eventlet.timeout.Timeout: 300 seconds

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9] During handling of the above exception, another
exception occurred:

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9] Traceback (most recent call last):

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]   File
"/opt/stack/nova/nova/compute/manager.py", line 10090, in
_error_out_instance_on_exception

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]     yield

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]   File
"/opt/stack/nova/nova/compute/manager.py", line 3391, in
rebuild_instance

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]     self._do_rebuild_instance_with_claim(

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]   File
"/opt/stack/nova/nova/compute/manager.py", line 3473, in
_do_rebuild_instance_with_claim

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]     self._do_rebuild_instance(

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]   File
"/opt/stack/nova/nova/compute/manager.py", line 3640, in
_do_rebuild_instance

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]     self._rebuild_default_impl(**kwargs)

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]   File
"/opt/stack/nova/nova/compute/manager.py", line 3277, in
_rebuild_default_impl

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]     self.driver.spawn(context, instance,
image_meta, injected_files,

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]   File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 4150, in spawn

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]     self._create_guest_with_network(

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]   File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 7214, in
_create_guest_with_network

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]     raise
exception.VirtualInterfaceCreateException()

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9] nova.exception.VirtualInterfaceCreateException:
Virtual Interface creation failed

May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
434c-acbe-7092a6b1bcd9]

** Affects: nova
     Importance: Medium
     Assignee: Balazs Gibizer (balazs-gibizer)
         Status: Triaged


** Tags: evacuate gate-failure

** Changed in: nova
       Status: New => Triaged

** Changed in: nova
     Assignee: (unassigned) => Balazs Gibizer (balazs-gibizer)

** Changed in: nova
   Importance: Undecided => Medium

** Tags added: evacuate gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1928922

Title:
  positive evacuation testing in nova-live-migration fails with
  VirtualInterfaceCreateException

Status in OpenStack Compute (nova):
  Triaged

Bug description:
  Example POST_FAILURE run
  https://zuul.opendev.org/t/openstack/build/4d0d5072c1e0479db616a211e0afda42/logs

  
  https://zuul.opendev.org/t/openstack/build/4d0d5072c1e0479db616a211e0afda42/log/controller/logs/screen-n-cpu.txt#13322

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [None req-
  778f8336-8e29-4282-a112-91a876303fe3 demo admin] [instance: 021396f6
  -40ff-434c-acbe-7092a6b1bcd9] Setting instance vm_state to ERROR:
  nova.exception.VirtualInterfaceCreateException: Virtual Interface
  creation failed

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9] Traceback (most recent call last):

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]   File
  "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7188, in
  _create_guest_with_network

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]     guest = self._create_guest(

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]   File "/usr/lib/python3.8/contextlib.py",
  line 120, in __exit__

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]     next(self.gen)

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]   File
  "/opt/stack/nova/nova/compute/manager.py", line 479, in
  wait_for_instance_event

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]     actual_event = event.wait()

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]   File "/usr/local/lib/python3.8/dist-
  packages/eventlet/event.py", line 125, in wait

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]     result = hub.switch()

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]   File "/usr/local/lib/python3.8/dist-
  packages/eventlet/hubs/hub.py", line 313, in switch

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]     return self.greenlet.switch()

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9] eventlet.timeout.Timeout: 300 seconds

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9] During handling of the above exception,
  another exception occurred:

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9] Traceback (most recent call last):

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]   File
  "/opt/stack/nova/nova/compute/manager.py", line 10090, in
  _error_out_instance_on_exception

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]     yield

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]   File
  "/opt/stack/nova/nova/compute/manager.py", line 3391, in
  rebuild_instance

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]     self._do_rebuild_instance_with_claim(

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]   File
  "/opt/stack/nova/nova/compute/manager.py", line 3473, in
  _do_rebuild_instance_with_claim

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]     self._do_rebuild_instance(

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]   File
  "/opt/stack/nova/nova/compute/manager.py", line 3640, in
  _do_rebuild_instance

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]     self._rebuild_default_impl(**kwargs)

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]   File
  "/opt/stack/nova/nova/compute/manager.py", line 3277, in
  _rebuild_default_impl

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]     self.driver.spawn(context, instance,
  image_meta, injected_files,

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]   File
  "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4150, in spawn

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]     self._create_guest_with_network(

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]   File
  "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7214, in
  _create_guest_with_network

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]     raise
  exception.VirtualInterfaceCreateException()

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]
  nova.exception.VirtualInterfaceCreateException: Virtual Interface
  creation failed

  May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova-
  compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-
  434c-acbe-7092a6b1bcd9]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1928922/+subscriptions
Follow ups

[Bug 1928922] Re: evacuation tests in nova-live-migration post hook fails with VirtualInterfaceCreateException due to vif-plugged event received before nova starts waiting for it.
From: Bogdan Dobrelya, 2022-02-24