yahoo-eng-team team mailing list archive

[OpenStack Infra <1809095@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/643023
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5a1c385b996090b80f5881680e04c88abc21828a
Submitter: Zuul
Branch:    master

commit 5a1c385b996090b80f5881680e04c88abc21828a
Author: Adrian Chiris <adrianc@xxxxxxxxxxxx>
Date:   Tue Mar 12 14:19:04 2019 +0200

    Move get_pci_mapping_for_migration to MigrationContext
    
    In order to fix Bug #1809095, it is required to update
    PCI related VIFs with the original PCI address on the source
    host to allow virt driver to properly unplug the VIF from hypervisor,
    e.g allow the proper VF representor to be unplugged
    from the integration bridge in case of a hardware offloaded OVS.
    
    To do so, some preliminary work is needed to allow code-sharing
    between nova.network.neutronv2 and nova.compute.manager
    
    This change:
    - Moves common logic to retrieve the PCI mapping between
      the source and destination node from nova.network.neutronv2
      to objects.migration_context.
    - Makes code adjustments to methods in nova.network.neutronv2
      to accomodate the former.
    
    Change-Id: I9a5118373548c525b2b1c2271e7d210cc92e4f4c
    Partial-Bug: #1809095


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1809095

Title:
  Wrong representor port was unplugged from OVS during cold migration

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Description
  ===========
  Wrong representor port was unplugged from OVS during cold migration.
  This happens when VM is scheduled to use a different PCI device to target host vs.
  to what PCI device it is using from source host. Nova uses new PCI device information to unplug
  representor port in source compute.

  
  Steps to reproduce
  ==================
  1. Create representor ports
  $ openstack port create --network private --vnic-type=direct --binding-profile '{"capabilities": ["switchdev"]}' direct_port1 
  $ openstack port create --network private --vnic-type=direct --binding-profile '{"capabilities": ["switchdev"]}' direct_port2
  2. Create VMs using the ports created above:
  openstack server create --flavor m1.small --image fedora24 --nic port-id=direct_port1 --availability-zone=nova:compute-1 vm1 
  openstack server create --flavor m1.small --image fedora24 --nic port-id=direct_port2 --availability-zone=nova:compute-2 vm2
  3. Migrate VM2
  $ openstack server migrate vm2
  $ openstack server resize --confirm vm2
  4. VM2 was migrated to compute-1, however representor port is still attached to OVS 
  $ sudo ovs-dpctl show
  system@ovs-system:
          lookups: hit:466465 missed:5411 lost:0
          flows: 12
          masks: hit:739146 total:2 hit/pkt:1.57
          port 0: ovs-system (internal)
          port 1: br-pro0.0 (internal)
          port 2: br-pro0 (internal)
          port 3: ens6f0
          port 4: br-int (internal)
          port 5: eth3

  Expected result
  ===============
  After cold migration, VM's previously used representor port should be unplugged from OVS

  Actual result
  =============
  VM's previously used representor port is still plugged in source host. In some scenarios, wrong representor port was unplugged from source host. Thus affecting VMs that were not cold migrated.

  Environment
  ===========
  Libvirt+KVM
  $ /usr/libexec/qemu-kvm --version
  QEMU emulator version 2.10.0
  $ virsh --version
  3.9.0
  Neutron+OVS HW Offload 
  Openstack Queens openstack-nova-compute-17.0.7-1

  
  Logs & Configs
  ==============
  1. Plug vif device using pci address 0000:81:00.5
  2018-12-15 13:12:04.871 108055 DEBUG os_vif [req-cd20d9ab-e880-41fa-aee5-97b920abcf77 dd9f16f6b15740e181c9b7cf8ee5795c 52298dbce7024cf89ca9e6d7369a67de - default default] Plugging vif VIFHostDevice(active=False,address=fa:16:3e:1b:0a:21,dev_address=0000:81:00.5,dev_type='ethernet',has_traffic_filtering=True,id=38609ab2-cf36-4782-83c7-7ee2d5c1c163,network=Network(bd30c752-4876-498b-9a36-e9733b635f4f),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True) plug /usr/lib/python2.7/site-packages/os_vif/__init__.py:76

  2. VM was migrated from compute-1 to compute-2. New pci device is now 0000:81:00.4
  2018-12-15 13:13:58.721 108055 DEBUG os_vif [req-afd99706-cf49-4c20-b85b-ea4d990ffbb4 dd9f16f6b15740e181c9b7cf8ee5795c 52298dbce7024cf89ca9e6d7369a67de - default default] Unplugging vif VIFHostDevice(active=True,address=fa:16:3e:1b:0a:21,dev_address=0000:81:00.4,dev_type='ethernet',has_traffic_filtering=True,id=38609ab2-cf36-4782-83c7-7ee2d5c1c163,network=Network(bd30c752-4876-498b-9a36-e9733b635f4f),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True) unplug /usr/lib/python2.7/site-packages/os_vif/__init__.py:109
  2018-12-15 13:13:58.759 108055 INFO os_vif [req-afd99706-cf49-4c20-b85b-ea4d990ffbb4 dd9f16f6b15740e181c9b7cf8ee5795c 52298dbce7024cf89ca9e6d7369a67de - default default] Successfully unplugged vif VIFHostDevice(active=True,address=fa:16:3e:1b:0a:21,dev_address=0000:81:00.4,dev_type='ethernet',has_traffic_filtering=True,id=38609ab2-cf36-4782-83c7-7ee2d5c1c163,network=Network(bd30c752-4876-498b-9a36-e9733b635f4f),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True)

  
  vif_plug_ovs used the new information passed by Nova to unplug representor port:
  https://github.com/openstack/os-vif/blob/db5216357b1be93d91aa48b2878599f2dfef02a8/vif_plug_ovs/ovs.py#L299

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1809095/+subscriptions