yahoo-eng-team team mailing list archive

[OpenStack Infra <1783917@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/586402
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8e17c3784ec2a394f01eb6e6a242a921076cc5dc
Submitter: Zuul
Branch:    master

commit 8e17c3784ec2a394f01eb6e6a242a921076cc5dc
Author: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date:   Thu Jul 26 23:10:58 2018 -0400

    Use source vifs when unplugging on source during post live migrate
    
    With the changes in blueprint neutron-new-port-binding-api, we now
    manage port bindings on both the source and destination host and
    those can be different vif types to support live migrating between
    different network backends, e.g. linuxbridge-ovs, ovs->ovn, etc.
    
    The dest host port bindings are created during pre_live_migration
    and then activated on the source host during post-copy (for the
    libvirt driver) or in _post_live_migration when we call the
    migrate_instance_start() method in the network API. When the dest
    host port bindings are activated, the source host port bindings
    are automatically deactivated (it's an atomic operation in neutron).
    
    Depending on when we refresh the nw info cache on the instance,
    we can end up calling driver.post_live_migration_at_source() with
    information about the unbound source port binding, or the vif type
    for the destination host port binding. Because of that, unplug_vifs
    on the source can fail since it's not using the proper vif type.
    
    To resolve that, we use the stashed VIFMigrateData.source_vif
    when unplugging vifs on the source host. Those source vifs are
    an exact copy of the vifs on the instance before live migration
    started on the source host, so they are appropriate for unplugging
    on the source during _post_live_migration.
    
    Change-Id: Ie6439499cc7c6f05eec692dda677caf649dd7c94
    Closes-Bug: #1783917


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1783917

Title:
  live migration fails with NovaException: Unsupported VIF type unbound
  convert '_nova_to_osvif_vif_unbound'

Status in neutron:
  Invalid
Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Seen in the gate starting today:

  http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Unsupported%20VIF%20type%20unbound%20convert%20'_nova_to_osvif_vif_unbound'%5C%22%20AND%20tags:screen-n-cpu.txt&from=7d

  My guess is that it's related to https://review.openstack.org/522537
  which began using the new neutron port binding API.

  Trace looks like this:

  http://logs.openstack.org/52/564452/30/check/nova-live-
  migration/d3172a4/logs/screen-n-cpu.txt.gz#_Jul_26_22_44_44_739885

  Jul 26 22:44:44.739885 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: WARNING nova.virt.libvirt.driver [None req-7c0f6655-5920-4894-b3d5-062ba4f8270e service nova] [instance: 582afae2-510c-4b73-b41d-1edc93e902a2] Error monitoring migration: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound': NovaException: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound'
  Jul 26 22:44:44.740334 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2] Traceback (most recent call last):
  Jul 26 22:44:44.740578 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7557, in _live_migration
  Jul 26 22:44:44.740839 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     finish_event, disk_paths)
  Jul 26 22:44:44.741086 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7421, in _live_migration_monitor
  Jul 26 22:44:44.741328 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     migrate_data)
  Jul 26 22:44:44.741562 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/exception_wrapper.py", line 79, in wrapped
  Jul 26 22:44:44.741785 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     function_name, call_dict, binary, tb)
  Jul 26 22:44:44.742008 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jul 26 22:44:44.742246 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     self.force_reraise()
  Jul 26 22:44:44.742485 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jul 26 22:44:44.742791 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     six.reraise(self.type_, self.value, self.tb)
  Jul 26 22:44:44.743117 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/exception_wrapper.py", line 69, in wrapped
  Jul 26 22:44:44.743332 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     return f(self, context, *args, **kw)
  Jul 26 22:44:44.743544 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/compute/manager.py", line 216, in decorated_function
  Jul 26 22:44:44.743758 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     kwargs['instance'], e, sys.exc_info())
  Jul 26 22:44:44.745586 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jul 26 22:44:44.747347 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     self.force_reraise()
  Jul 26 22:44:44.747790 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jul 26 22:44:44.748166 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     six.reraise(self.type_, self.value, self.tb)
  Jul 26 22:44:44.748507 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/compute/manager.py", line 204, in decorated_function
  Jul 26 22:44:44.748859 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     return function(self, context, *args, **kwargs)
  Jul 26 22:44:44.749207 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/compute/manager.py", line 6582, in _post_live_migration
  Jul 26 22:44:44.749626 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     network_info)
  Jul 26 22:44:44.750459 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7971, in post_live_migration_at_source
  Jul 26 22:44:44.751201 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     self.unplug_vifs(instance, network_info)
  Jul 26 22:44:44.751576 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 829, in unplug_vifs
  Jul 26 22:44:44.752054 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     self._unplug_vifs(instance, network_info, False)
  Jul 26 22:44:44.752517 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 823, in _unplug_vifs
  Jul 26 22:44:44.752885 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     self.vif_driver.unplug(instance, vif)
  Jul 26 22:44:44.753247 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/virt/libvirt/vif.py", line 928, in unplug
  Jul 26 22:44:44.753613 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     vif_obj = os_vif_util.nova_to_osvif_vif(vif)
  Jul 26 22:44:44.753983 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]   File "/opt/stack/new/nova/nova/network/os_vif_util.py", line 505, in nova_to_osvif_vif
  Jul 26 22:44:44.754411 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]     {'type': vif['type'], 'func': funcname})
  Jul 26 22:44:44.754752 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2] NovaException: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound'
  Jul 26 22:44:44.755092 ubuntu-xenial-rax-dfw-0000999143 nova-compute[2823]: ERROR nova.virt.libvirt.driver [instance: 582afae2-510c-4b73-b41d-1edc93e902a2]

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1783917/+subscriptions