yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1944781] [NEW] finish_migration fails with nova.exception.InternalError: Unexpected vif_type=binding_failed

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: melanie witt <1944781@xxxxxxxxxxxxxxxxxx>
Date: Thu, 23 Sep 2021 23:38:49 -0000
Reply-to: Bug 1944781 <1944781@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Just saw this on a gate failure, at the end of a resize the port binding
fails because the vif type "binding_failed" is not found in the vif
conversion step:

Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [None req-a02b86b5-b31e-4445-9fa3-07d16e10e223 tempest-AttachVolumeMultiAttachTest-2037743988 tempest-AttachVolumeMultiAttachTest-2037743988-project] [instance: c253b31e-afba-4695-ad01-9ec1e55bb987] Setting instance vm_state to ERROR: nova.exception.InternalError: Unexpected vif_type=binding_failed
Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987] Traceback (most recent call last):
Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]   File "/opt/stack/nova/nova/compute/manager.py", line 10228, in _error_out_instance_on_exception
Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]     yield
Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]   File "/opt/stack/nova/nova/compute/manager.py", line 5886, in _finish_resize_helper
Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]     network_info = self._finish_resize(context, instance, migration,
Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]   File "/opt/stack/nova/nova/compute/manager.py", line 5824, in _finish_resize
Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]     self._set_instance_info(instance, old_flavor)
Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]     self.force_reraise()
Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]     raise self.value
Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]   File "/opt/stack/nova/nova/compute/manager.py", line 5807, in _finish_resize
Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]     self.driver.finish_migration(context, migration, instance,
Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 11093, in finish_migration
Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]     xml = self._get_guest_xml(context, instance, network_info,
Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7038, in _get_guest_xml
Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]     conf = self._get_guest_config(instance, network_info, image_meta,
Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6667, in _get_guest_config
Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]     config = self.vif_driver.get_config(
Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]   File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 609, in get_config
Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]     raise exception.InternalError(_('Unexpected vif_type=%s') % vif_type)
Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987] nova.exception.InternalError: Unexpected vif_type=binding_failed
Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987] 


Logstash query:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Setting%20instance%20vm_state%20to%20ERROR%3A%20nova.exception.InternalError%3A%20Unexpected%20vif_type%3Dbinding_failed%5C%22%20AND%20tags%3A%5C%22screen-
n-cpu.txt%5C%22&from=7d

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: gate-failure libvirt network

** Tags added: libvirt network

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1944781

Title:
  finish_migration fails with nova.exception.InternalError: Unexpected
  vif_type=binding_failed

Status in OpenStack Compute (nova):
  New

Bug description:
  Just saw this on a gate failure, at the end of a resize the port
  binding fails because the vif type "binding_failed" is not found in
  the vif conversion step:

  Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [None req-a02b86b5-b31e-4445-9fa3-07d16e10e223 tempest-AttachVolumeMultiAttachTest-2037743988 tempest-AttachVolumeMultiAttachTest-2037743988-project] [instance: c253b31e-afba-4695-ad01-9ec1e55bb987] Setting instance vm_state to ERROR: nova.exception.InternalError: Unexpected vif_type=binding_failed
  Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987] Traceback (most recent call last):
  Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]   File "/opt/stack/nova/nova/compute/manager.py", line 10228, in _error_out_instance_on_exception
  Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]     yield
  Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]   File "/opt/stack/nova/nova/compute/manager.py", line 5886, in _finish_resize_helper
  Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]     network_info = self._finish_resize(context, instance, migration,
  Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]   File "/opt/stack/nova/nova/compute/manager.py", line 5824, in _finish_resize
  Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]     self._set_instance_info(instance, old_flavor)
  Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]     self.force_reraise()
  Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]     raise self.value
  Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]   File "/opt/stack/nova/nova/compute/manager.py", line 5807, in _finish_resize
  Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]     self.driver.finish_migration(context, migration, instance,
  Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 11093, in finish_migration
  Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]     xml = self._get_guest_xml(context, instance, network_info,
  Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7038, in _get_guest_xml
  Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]     conf = self._get_guest_config(instance, network_info, image_meta,
  Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6667, in _get_guest_config
  Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]     config = self.vif_driver.get_config(
  Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]   File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 609, in get_config
  Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987]     raise exception.InternalError(_('Unexpected vif_type=%s') % vif_type)
  Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987] nova.exception.InternalError: Unexpected vif_type=binding_failed
  Sep 23 06:13:22.444045 ubuntu-focal-rax-dfw-0026631000 nova-compute[67008]: ERROR nova.compute.manager [instance: c253b31e-afba-4695-ad01-9ec1e55bb987] 

  
  Logstash query:

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Setting%20instance%20vm_state%20to%20ERROR%3A%20nova.exception.InternalError%3A%20Unexpected%20vif_type%3Dbinding_failed%5C%22%20AND%20tags%3A%5C%22screen-
  n-cpu.txt%5C%22&from=7d

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1944781/+subscriptions