yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1783917] [NEW] live migration fails with NovaException: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound'

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: melanie witt <1783917@xxxxxxxxxxxxxxxxxx>
Date: Fri, 27 Jul 2018 01:55:04 -0000
Reply-to: Bug 1783917 <1783917@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Seen in the gate starting today:

http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Unsupported%20VIF%20type%20unbound%20convert%20'_nova_to_osvif_vif_unbound'%5C%22%20AND%20tags:screen-n-cpu.txt&from=7d

My guess is that it's related to https://review.openstack.org/522537
which began using the new neutron port binding API.

Trace looks like this:

http://logs.openstack.org/52/564452/30/check/nova-live-
migration/d3172a4/logs/screen-n-cpu.txt.gz#_Jul_26_22_44_44_739885

Jul 26 22:44:44.739885 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: WARNING nova.virt.libvirt.driver [None req-7c0f6655-5920-4894-b3d5-062ba4f8270e service nova] [instance: 582afae2-510c-4b73-b41d-1edc93e902a2] Error monitoring migration: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound': NovaException: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound'
Jul 26 22:44:44.740334 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2] Traceback (most recent call last):
Jul 26 22:44:44.740578 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7557, in _live_migration
Jul 26 22:44:44.740839 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     finish_event, disk_paths)
Jul 26 22:44:44.741086 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7421, in _live_migration_monitor
Jul 26 22:44:44.741328 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     migrate_data)
Jul 26 22:44:44.741562 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/exception_wrapper.py", line 79, in wrapped
Jul 26 22:44:44.741785 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     function_name, call_dict, binary, tb)
Jul 26 22:44:44.742008 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Jul 26 22:44:44.742246 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     self.force_reraise()
Jul 26 22:44:44.742485 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jul 26 22:44:44.742791 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     six.reraise(self.type_, self.value, self.tb)
Jul 26 22:44:44.743117 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/exception_wrapper.py", line 69, in wrapped
Jul 26 22:44:44.743332 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     return f(self, context, *args, **kw)
Jul 26 22:44:44.743544 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/compute/manager.py", line 216, in decorated_function
Jul 26 22:44:44.743758 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     kwargs['instance'], e, sys.exc_info())
Jul 26 22:44:44.745586 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Jul 26 22:44:44.747347 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     self.force_reraise()
Jul 26 22:44:44.747790 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jul 26 22:44:44.748166 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     six.reraise(self.type_, self.value, self.tb)
Jul 26 22:44:44.748507 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/compute/manager.py", line 204, in decorated_function
Jul 26 22:44:44.748859 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     return function(self, context, *args, **kwargs)
Jul 26 22:44:44.749207 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/compute/manager.py", line 6582, in _post_live_migration
Jul 26 22:44:44.749626 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     network_info)
Jul 26 22:44:44.750459 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7971, in post_live_migration_at_source
Jul 26 22:44:44.751201 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     self.unplug_vifs(instance, network_info)
Jul 26 22:44:44.751576 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 829, in unplug_vifs
Jul 26 22:44:44.752054 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     self._unplug_vifs(instance, network_info, False)
Jul 26 22:44:44.752517 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 823, in _unplug_vifs
Jul 26 22:44:44.752885 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     self.vif_driver.unplug(instance, vif)
Jul 26 22:44:44.753247 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/virt/libvirt/vif.py", line 928, in unplug
Jul 26 22:44:44.753613 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     vif_obj = os_vif_util.nova_to_osvif_vif(vif)
Jul 26 22:44:44.753983 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/network/os_vif_util.py", line 505, in nova_to_osvif_vif
Jul 26 22:44:44.754411 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     {'type': vif['type'], 'func': funcname})
Jul 26 22:44:44.754752 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2] NovaException: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound'
Jul 26 22:44:44.755092 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]

** Affects: neutron
     Importance: High
         Status: Confirmed

** Affects: nova
     Importance: High
         Status: Confirmed


** Tags: live-migration rocky-rc-potential

** Also affects: neutron
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1783917

Title:
  live migration fails with NovaException: Unsupported VIF type unbound
  convert '_nova_to_osvif_vif_unbound'

Status in neutron:
  Confirmed
Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  Seen in the gate starting today:

  http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Unsupported%20VIF%20type%20unbound%20convert%20'_nova_to_osvif_vif_unbound'%5C%22%20AND%20tags:screen-n-cpu.txt&from=7d

  My guess is that it's related to https://review.openstack.org/522537
  which began using the new neutron port binding API.

  Trace looks like this:

  http://logs.openstack.org/52/564452/30/check/nova-live-
  migration/d3172a4/logs/screen-n-cpu.txt.gz#_Jul_26_22_44_44_739885

  Jul 26 22:44:44.739885 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: WARNING nova.virt.libvirt.driver [None req-7c0f6655-5920-4894-b3d5-062ba4f8270e service nova] [instance: 582afae2-510c-4b73-b41d-1edc93e902a2] Error monitoring migration: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound': NovaException: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound'
  Jul 26 22:44:44.740334 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2] Traceback (most recent call last):
  Jul 26 22:44:44.740578 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7557, in _live_migration
  Jul 26 22:44:44.740839 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     finish_event, disk_paths)
  Jul 26 22:44:44.741086 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7421, in _live_migration_monitor
  Jul 26 22:44:44.741328 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     migrate_data)
  Jul 26 22:44:44.741562 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/exception_wrapper.py", line 79, in wrapped
  Jul 26 22:44:44.741785 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     function_name, call_dict, binary, tb)
  Jul 26 22:44:44.742008 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jul 26 22:44:44.742246 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     self.force_reraise()
  Jul 26 22:44:44.742485 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jul 26 22:44:44.742791 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     six.reraise(self.type_, self.value, self.tb)
  Jul 26 22:44:44.743117 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/exception_wrapper.py", line 69, in wrapped
  Jul 26 22:44:44.743332 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     return f(self, context, *args, **kw)
  Jul 26 22:44:44.743544 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/compute/manager.py", line 216, in decorated_function
  Jul 26 22:44:44.743758 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     kwargs['instance'], e, sys.exc_info())
  Jul 26 22:44:44.745586 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jul 26 22:44:44.747347 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     self.force_reraise()
  Jul 26 22:44:44.747790 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jul 26 22:44:44.748166 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     six.reraise(self.type_, self.value, self.tb)
  Jul 26 22:44:44.748507 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/compute/manager.py", line 204, in decorated_function
  Jul 26 22:44:44.748859 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     return function(self, context, *args, **kwargs)
  Jul 26 22:44:44.749207 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/compute/manager.py", line 6582, in _post_live_migration
  Jul 26 22:44:44.749626 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     network_info)
  Jul 26 22:44:44.750459 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7971, in post_live_migration_at_source
  Jul 26 22:44:44.751201 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     self.unplug_vifs(instance, network_info)
  Jul 26 22:44:44.751576 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 829, in unplug_vifs
  Jul 26 22:44:44.752054 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     self._unplug_vifs(instance, network_info, False)
  Jul 26 22:44:44.752517 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 823, in _unplug_vifs
  Jul 26 22:44:44.752885 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     self.vif_driver.unplug(instance, vif)
  Jul 26 22:44:44.753247 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/virt/libvirt/vif.py", line 928, in unplug
  Jul 26 22:44:44.753613 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     vif_obj = os_vif_util.nova_to_osvif_vif(vif)
  Jul 26 22:44:44.753983 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/network/os_vif_util.py", line 505, in nova_to_osvif_vif
  Jul 26 22:44:44.754411 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     {'type': vif['type'], 'func': funcname})
  Jul 26 22:44:44.754752 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2] NovaException: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound'
  Jul 26 22:44:44.755092 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1783917/+subscriptions
Follow ups

[Bug 1783917] Re: live migration fails with NovaException: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound'
From: OpenStack Infra, 2018-07-27
[Bug 1783917] Re: live migration fails with NovaException: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound'
From: Matt Riedemann, 2018-07-27