yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem.os@xxxxxxxxx>]

** Also affects: nova/queens
   Importance: Undecided
       Status: New

** Also affects: nova/rocky
   Importance: Undecided
       Status: New

** Also affects: nova/stein
   Importance: Undecided
       Status: New

** Changed in: nova/queens
       Status: New => In Progress

** Changed in: nova/rocky
       Status: New => In Progress

** Changed in: nova/stein
       Status: New => In Progress

** Changed in: nova/queens
     Assignee: (unassigned) => Adrian Chiris (adrian.chiris)

** Changed in: nova/rocky
     Assignee: (unassigned) => Adrian Chiris (adrian.chiris)

** Changed in: nova/stein
     Assignee: (unassigned) => Adrian Chiris (adrian.chiris)

** Changed in: nova/rocky
   Importance: Undecided => Medium

** Changed in: nova/queens
   Importance: Undecided => Medium

** Changed in: nova/stein
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1809095

Title:
  Wrong representor port was unplugged from OVS during cold migration

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) queens series:
  In Progress
Status in OpenStack Compute (nova) rocky series:
  In Progress
Status in OpenStack Compute (nova) stein series:
  In Progress

Bug description:
  Description
  ===========
  Wrong representor port was unplugged from OVS during cold migration.
  This happens when VM is scheduled to use a different PCI device to target host vs.
  to what PCI device it is using from source host. Nova uses new PCI device information to unplug
  representor port in source compute.

  
  Steps to reproduce
  ==================
  1. Create representor ports
  $ openstack port create --network private --vnic-type=direct --binding-profile '{"capabilities": ["switchdev"]}' direct_port1 
  $ openstack port create --network private --vnic-type=direct --binding-profile '{"capabilities": ["switchdev"]}' direct_port2
  2. Create VMs using the ports created above:
  openstack server create --flavor m1.small --image fedora24 --nic port-id=direct_port1 --availability-zone=nova:compute-1 vm1 
  openstack server create --flavor m1.small --image fedora24 --nic port-id=direct_port2 --availability-zone=nova:compute-2 vm2
  3. Migrate VM2
  $ openstack server migrate vm2
  $ openstack server resize --confirm vm2
  4. VM2 was migrated to compute-1, however representor port is still attached to OVS 
  $ sudo ovs-dpctl show
  system@ovs-system:
          lookups: hit:466465 missed:5411 lost:0
          flows: 12
          masks: hit:739146 total:2 hit/pkt:1.57
          port 0: ovs-system (internal)
          port 1: br-pro0.0 (internal)
          port 2: br-pro0 (internal)
          port 3: ens6f0
          port 4: br-int (internal)
          port 5: eth3

  Expected result
  ===============
  After cold migration, VM's previously used representor port should be unplugged from OVS

  Actual result
  =============
  VM's previously used representor port is still plugged in source host. In some scenarios, wrong representor port was unplugged from source host. Thus affecting VMs that were not cold migrated.

  Environment
  ===========
  Libvirt+KVM
  $ /usr/libexec/qemu-kvm --version
  QEMU emulator version 2.10.0
  $ virsh --version
  3.9.0
  Neutron+OVS HW Offload 
  Openstack Queens openstack-nova-compute-17.0.7-1

  
  Logs & Configs
  ==============
  1. Plug vif device using pci address 0000:81:00.5
  2018-12-15 13:12:04.871 108055 DEBUG os_vif [req-cd20d9ab-e880-41fa-aee5-97b920abcf77 dd9f16f6b15740e181c9b7cf8ee5795c 52298dbce7024cf89ca9e6d7369a67de - default default] Plugging vif VIFHostDevice(active=False,address=fa:16:3e:1b:0a:21,dev_address=0000:81:00.5,dev_type='ethernet',has_traffic_filtering=True,id=38609ab2-cf36-4782-83c7-7ee2d5c1c163,network=Network(bd30c752-4876-498b-9a36-e9733b635f4f),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True) plug /usr/lib/python2.7/site-packages/os_vif/__init__.py:76

  2. VM was migrated from compute-1 to compute-2. New pci device is now 0000:81:00.4
  2018-12-15 13:13:58.721 108055 DEBUG os_vif [req-afd99706-cf49-4c20-b85b-ea4d990ffbb4 dd9f16f6b15740e181c9b7cf8ee5795c 52298dbce7024cf89ca9e6d7369a67de - default default] Unplugging vif VIFHostDevice(active=True,address=fa:16:3e:1b:0a:21,dev_address=0000:81:00.4,dev_type='ethernet',has_traffic_filtering=True,id=38609ab2-cf36-4782-83c7-7ee2d5c1c163,network=Network(bd30c752-4876-498b-9a36-e9733b635f4f),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True) unplug /usr/lib/python2.7/site-packages/os_vif/__init__.py:109
  2018-12-15 13:13:58.759 108055 INFO os_vif [req-afd99706-cf49-4c20-b85b-ea4d990ffbb4 dd9f16f6b15740e181c9b7cf8ee5795c 52298dbce7024cf89ca9e6d7369a67de - default default] Successfully unplugged vif VIFHostDevice(active=True,address=fa:16:3e:1b:0a:21,dev_address=0000:81:00.4,dev_type='ethernet',has_traffic_filtering=True,id=38609ab2-cf36-4782-83c7-7ee2d5c1c163,network=Network(bd30c752-4876-498b-9a36-e9733b635f4f),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True)

  
  vif_plug_ovs used the new information passed by Nova to unplug representor port:
  https://github.com/openstack/os-vif/blob/db5216357b1be93d91aa48b2878599f2dfef02a8/vif_plug_ovs/ovs.py#L299

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1809095/+subscriptions