← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #91384

[Bug 2008238] [NEW] SRIOV port binding_profile attributes for OVS hardware offload are stripped on instance deletion or port detachment

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Description
===========

This issue applies for systems using SRIOV with Mellanox ASAP2 SDN
offloads.

An SRIOV port capable for ASAP2 SDN acceleration (OVS hardware offloads)
has 'capabilities=[switchdev]' added to the port binding_profile.

After a VM has been created with SRIOV port attached, the port can no
longer be used for subsequent VM builds.  Attempt to reuse the port
results in an error of the form "Cannot set interface MAC/vlanid to
<mac>/<vlan> for ifname ens1f0 vf 7: Operation not supported"

The underlying issue appears to be that when an SRIOV port is detached
from a VM, or the VM is destroyed, the capabilities=[switchdev] property
is removed from the port binding_profile.  This converts the port from
ASAP2 to “Legacy SRIOV” (in Mellanox-speak) and makes it unusable.

If the port binding_profile property is restored then the port can be
successfully reused.

The property is preserved during live migration, instance resizes and
rebuilds.  It only appears to be instance depletion or port detachment
where the binding_profile property is removed.

Steps to reproduce
==================

1. Create SRIOV port with ASAP2 capability:

openstack port create --project <project> --network <network> --vnic-
type=direct --binding-profile '{"capabilities": ["switchdev"]}' sriov-
port-1

2. Check the port binding_profile property:

openstack port show -c binding_profile sriov-port-1

3. Create an instance using the port:

openstack server create --flavor <flavor> --image <image> --key-name
<key> --nic port-id=sriov-port-1 sriov-vm-1

4. Delete the instance:

openstack server delete sriov-vm-1

5. Check the port binding_profile property:

openstack port show -c binding_profile sriov-port-1

Expected Result
===============

Nova sets properties in the binding_profile while the instance is in
use.  Alongside those properties the capabilities='['switchdev']'
property should be preserved.

Actual Result
=============

After the instance is deleted (or port detached), the binding_profile is
empty.

Environment
===========

This has been observed with the following configuration:

- OpenStack Yoga
- OVN Neutron driver

Logs
====

>From Nova Compute:

2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest Traceback (most recent call last):
2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 165, in launch
2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest     return self._domain.createWithFlags(flags)
2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest   File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit
2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest     result = proxy_call(self._autowrap, f, *args, **kwargs)
2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest   File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest     rv = execute(f, *args, **kwargs)
2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest   File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest     six.reraise(c, e, tb)
2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest     raise value
2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest   File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest     rv = meth(*args, **kwargs)
2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest   File "/usr/lib64/python3.6/site-packages/libvirt.py", line 1385, in createWithFlags
2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest     raise libvirtError('virDomainCreateWithFlags() failed')
2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest libvirt.libvirtError: Cannot set interface MAC/vlanid to fa:16:3e:43:1e:ce/2107 for ifname ens1f0 vf 7: Operation not supported
2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest
2023-01-24 19:55:32.273 7 ERROR nova.virt.libvirt.driver [req-581cd9e8-11c8-44be-9ed2-a03a5f70d0f4 802a31d98b364da79be43fe6e9566d63 76f401abee7b4e80b7efd86f2f26e3ca - default default] [instance: d2091824-1f7a-4de1-8776-8f781956130a] Failed to start libvirt guest: libvirt.libvirtError: Cannot set interface MAC/vlanid to fa:16:3e:43:1e:ce/2107 for ifname ens1f0 vf 7: Operation not supported

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2008238

Title:
  SRIOV port binding_profile attributes for OVS hardware offload are
  stripped on instance deletion or port detachment

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========

  This issue applies for systems using SRIOV with Mellanox ASAP2 SDN
  offloads.

  An SRIOV port capable for ASAP2 SDN acceleration (OVS hardware
  offloads) has 'capabilities=[switchdev]' added to the port
  binding_profile.

  After a VM has been created with SRIOV port attached, the port can no
  longer be used for subsequent VM builds.  Attempt to reuse the port
  results in an error of the form "Cannot set interface MAC/vlanid to
  <mac>/<vlan> for ifname ens1f0 vf 7: Operation not supported"

  The underlying issue appears to be that when an SRIOV port is detached
  from a VM, or the VM is destroyed, the capabilities=[switchdev]
  property is removed from the port binding_profile.  This converts the
  port from ASAP2 to “Legacy SRIOV” (in Mellanox-speak) and makes it
  unusable.

  If the port binding_profile property is restored then the port can be
  successfully reused.

  The property is preserved during live migration, instance resizes and
  rebuilds.  It only appears to be instance depletion or port detachment
  where the binding_profile property is removed.

  Steps to reproduce
  ==================

  1. Create SRIOV port with ASAP2 capability:

  openstack port create --project <project> --network <network> --vnic-
  type=direct --binding-profile '{"capabilities": ["switchdev"]}' sriov-
  port-1

  2. Check the port binding_profile property:

  openstack port show -c binding_profile sriov-port-1

  3. Create an instance using the port:

  openstack server create --flavor <flavor> --image <image> --key-name
  <key> --nic port-id=sriov-port-1 sriov-vm-1

  4. Delete the instance:

  openstack server delete sriov-vm-1

  5. Check the port binding_profile property:

  openstack port show -c binding_profile sriov-port-1

  Expected Result
  ===============

  Nova sets properties in the binding_profile while the instance is in
  use.  Alongside those properties the capabilities='['switchdev']'
  property should be preserved.

  Actual Result
  =============

  After the instance is deleted (or port detached), the binding_profile
  is empty.

  Environment
  ===========

  This has been observed with the following configuration:

  - OpenStack Yoga
  - OVN Neutron driver

  Logs
  ====

  From Nova Compute:

  2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest Traceback (most recent call last):
  2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 165, in launch
  2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest     return self._domain.createWithFlags(flags)
  2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest   File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit
  2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest   File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
  2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest     rv = execute(f, *args, **kwargs)
  2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest   File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
  2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest     six.reraise(c, e, tb)
  2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest     raise value
  2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest   File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
  2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest     rv = meth(*args, **kwargs)
  2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest   File "/usr/lib64/python3.6/site-packages/libvirt.py", line 1385, in createWithFlags
  2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest     raise libvirtError('virDomainCreateWithFlags() failed')
  2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest libvirt.libvirtError: Cannot set interface MAC/vlanid to fa:16:3e:43:1e:ce/2107 for ifname ens1f0 vf 7: Operation not supported
  2023-01-24 19:55:32.270 7 ERROR nova.virt.libvirt.guest
  2023-01-24 19:55:32.273 7 ERROR nova.virt.libvirt.driver [req-581cd9e8-11c8-44be-9ed2-a03a5f70d0f4 802a31d98b364da79be43fe6e9566d63 76f401abee7b4e80b7efd86f2f26e3ca - default default] [instance: d2091824-1f7a-4de1-8776-8f781956130a] Failed to start libvirt guest: libvirt.libvirtError: Cannot set interface MAC/vlanid to fa:16:3e:43:1e:ce/2107 for ifname ens1f0 vf 7: Operation not supported

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2008238/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next