← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #76387

[Bug 1809095] [NEW] Wrong representor port was unplugged from OVS during cold migration

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Description
===========
Wrong representor port was unplugged from OVS during cold migration.
This happens when VM is scheduled to use a different PCI device to target host vs.
to what PCI device it is using from source host. Nova uses new PCI device information to unplug
representor port in source compute.


Steps to reproduce
==================
1. Create representor ports
$ openstack port create --network private --vnic-type=direct --binding-profile '{"capabilities": ["switchdev"]}' direct_port1 
$ openstack port create --network private --vnic-type=direct --binding-profile '{"capabilities": ["switchdev"]}' direct_port2
2. Create VMs using the ports created above:
openstack server create --flavor m1.small --image fedora24 --nic port-id=direct_port1 --availability-zone=nova:compute-1 vm1 
openstack server create --flavor m1.small --image fedora24 --nic port-id=direct_port2 --availability-zone=nova:compute-2 vm2
3. Migrate VM2
$ openstack server migrate vm2
$ openstack server resize --confirm vm2
4. VM2 was migrated to compute-1, however representor port is still attached to OVS 
$ sudo ovs-dpctl show
system@ovs-system:
        lookups: hit:466465 missed:5411 lost:0
        flows: 12
        masks: hit:739146 total:2 hit/pkt:1.57
        port 0: ovs-system (internal)
        port 1: br-pro0.0 (internal)
        port 2: br-pro0 (internal)
        port 3: ens6f0
        port 4: br-int (internal)
        port 5: eth3

Expected result
===============
After cold migration, VM's previously used representor port should be unplugged from OVS

Actual result
=============
VM's previously used representor port is still plugged in source host. In some scenarios, wrong representor port was unplugged from source host. Thus affecting VMs that were not cold migrated.

Environment
===========
Libvirt+KVM
$ /usr/libexec/qemu-kvm --version
QEMU emulator version 2.10.0
$ virsh --version
3.9.0
Neutron+OVS HW Offload 
Openstack Queens openstack-nova-compute-17.0.7-1


Logs & Configs
==============
1. Plug vif device using pci address 0000:81:00.5
2018-12-15 13:12:04.871 108055 DEBUG os_vif [req-cd20d9ab-e880-41fa-aee5-97b920abcf77 dd9f16f6b15740e181c9b7cf8ee5795c 52298dbce7024cf89ca9e6d7369a67de - default default] Plugging vif VIFHostDevice(active=False,address=fa:16:3e:1b:0a:21,dev_address=0000:81:00.5,dev_type='ethernet',has_traffic_filtering=True,id=38609ab2-cf36-4782-83c7-7ee2d5c1c163,network=Network(bd30c752-4876-498b-9a36-e9733b635f4f),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True) plug /usr/lib/python2.7/site-packages/os_vif/__init__.py:76

2. VM was migrated from compute-1 to compute-2. New pci device is now 0000:81:00.4
2018-12-15 13:13:58.721 108055 DEBUG os_vif [req-afd99706-cf49-4c20-b85b-ea4d990ffbb4 dd9f16f6b15740e181c9b7cf8ee5795c 52298dbce7024cf89ca9e6d7369a67de - default default] Unplugging vif VIFHostDevice(active=True,address=fa:16:3e:1b:0a:21,dev_address=0000:81:00.4,dev_type='ethernet',has_traffic_filtering=True,id=38609ab2-cf36-4782-83c7-7ee2d5c1c163,network=Network(bd30c752-4876-498b-9a36-e9733b635f4f),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True) unplug /usr/lib/python2.7/site-packages/os_vif/__init__.py:109
2018-12-15 13:13:58.759 108055 INFO os_vif [req-afd99706-cf49-4c20-b85b-ea4d990ffbb4 dd9f16f6b15740e181c9b7cf8ee5795c 52298dbce7024cf89ca9e6d7369a67de - default default] Successfully unplugged vif VIFHostDevice(active=True,address=fa:16:3e:1b:0a:21,dev_address=0000:81:00.4,dev_type='ethernet',has_traffic_filtering=True,id=38609ab2-cf36-4782-83c7-7ee2d5c1c163,network=Network(bd30c752-4876-498b-9a36-e9733b635f4f),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True)


vif_plug_ovs used the new information passed by Nova to unplug representor port:
https://github.com/openstack/os-vif/blob/db5216357b1be93d91aa48b2878599f2dfef02a8/vif_plug_ovs/ovs.py#L299

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1809095

Title:
  Wrong representor port was unplugged from OVS during cold migration

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  Wrong representor port was unplugged from OVS during cold migration.
  This happens when VM is scheduled to use a different PCI device to target host vs.
  to what PCI device it is using from source host. Nova uses new PCI device information to unplug
  representor port in source compute.

  
  Steps to reproduce
  ==================
  1. Create representor ports
  $ openstack port create --network private --vnic-type=direct --binding-profile '{"capabilities": ["switchdev"]}' direct_port1 
  $ openstack port create --network private --vnic-type=direct --binding-profile '{"capabilities": ["switchdev"]}' direct_port2
  2. Create VMs using the ports created above:
  openstack server create --flavor m1.small --image fedora24 --nic port-id=direct_port1 --availability-zone=nova:compute-1 vm1 
  openstack server create --flavor m1.small --image fedora24 --nic port-id=direct_port2 --availability-zone=nova:compute-2 vm2
  3. Migrate VM2
  $ openstack server migrate vm2
  $ openstack server resize --confirm vm2
  4. VM2 was migrated to compute-1, however representor port is still attached to OVS 
  $ sudo ovs-dpctl show
  system@ovs-system:
          lookups: hit:466465 missed:5411 lost:0
          flows: 12
          masks: hit:739146 total:2 hit/pkt:1.57
          port 0: ovs-system (internal)
          port 1: br-pro0.0 (internal)
          port 2: br-pro0 (internal)
          port 3: ens6f0
          port 4: br-int (internal)
          port 5: eth3

  Expected result
  ===============
  After cold migration, VM's previously used representor port should be unplugged from OVS

  Actual result
  =============
  VM's previously used representor port is still plugged in source host. In some scenarios, wrong representor port was unplugged from source host. Thus affecting VMs that were not cold migrated.

  Environment
  ===========
  Libvirt+KVM
  $ /usr/libexec/qemu-kvm --version
  QEMU emulator version 2.10.0
  $ virsh --version
  3.9.0
  Neutron+OVS HW Offload 
  Openstack Queens openstack-nova-compute-17.0.7-1

  
  Logs & Configs
  ==============
  1. Plug vif device using pci address 0000:81:00.5
  2018-12-15 13:12:04.871 108055 DEBUG os_vif [req-cd20d9ab-e880-41fa-aee5-97b920abcf77 dd9f16f6b15740e181c9b7cf8ee5795c 52298dbce7024cf89ca9e6d7369a67de - default default] Plugging vif VIFHostDevice(active=False,address=fa:16:3e:1b:0a:21,dev_address=0000:81:00.5,dev_type='ethernet',has_traffic_filtering=True,id=38609ab2-cf36-4782-83c7-7ee2d5c1c163,network=Network(bd30c752-4876-498b-9a36-e9733b635f4f),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True) plug /usr/lib/python2.7/site-packages/os_vif/__init__.py:76

  2. VM was migrated from compute-1 to compute-2. New pci device is now 0000:81:00.4
  2018-12-15 13:13:58.721 108055 DEBUG os_vif [req-afd99706-cf49-4c20-b85b-ea4d990ffbb4 dd9f16f6b15740e181c9b7cf8ee5795c 52298dbce7024cf89ca9e6d7369a67de - default default] Unplugging vif VIFHostDevice(active=True,address=fa:16:3e:1b:0a:21,dev_address=0000:81:00.4,dev_type='ethernet',has_traffic_filtering=True,id=38609ab2-cf36-4782-83c7-7ee2d5c1c163,network=Network(bd30c752-4876-498b-9a36-e9733b635f4f),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True) unplug /usr/lib/python2.7/site-packages/os_vif/__init__.py:109
  2018-12-15 13:13:58.759 108055 INFO os_vif [req-afd99706-cf49-4c20-b85b-ea4d990ffbb4 dd9f16f6b15740e181c9b7cf8ee5795c 52298dbce7024cf89ca9e6d7369a67de - default default] Successfully unplugged vif VIFHostDevice(active=True,address=fa:16:3e:1b:0a:21,dev_address=0000:81:00.4,dev_type='ethernet',has_traffic_filtering=True,id=38609ab2-cf36-4782-83c7-7ee2d5c1c163,network=Network(bd30c752-4876-498b-9a36-e9733b635f4f),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True)

  
  vif_plug_ovs used the new information passed by Nova to unplug representor port:
  https://github.com/openstack/os-vif/blob/db5216357b1be93d91aa48b2878599f2dfef02a8/vif_plug_ovs/ovs.py#L299

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1809095/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next