yahoo-eng-team team mailing list archive

[Frode Nordahl <1975743@xxxxxxxxxxxxxxxxxx>]

Itai, thank you for reporting this bug.

The Neutron OVN driver does strict validation of the binding profile. As
part of adding support for SmartNIC DPUs the validation was extended to
handle both the existing hardware offload vnic-type direct +
capabilities switchdev workflow as well as the new SmartNIC DPU vnic-
type remote-managed workflow.

What's happening here is that Neutron does not expect Nova to provide
the 'card_serial_number', 'pf_mac_address' and 'vf_num' keys in the
binding profile for the vnic-type direct + capabilities switchdev
workflow, and rejects the request.

The key/value pairs appear to be added whenever a VF from a card with a
serial number in the VPD is used, if the card does not have a serial in
the VPD the key/value pairs are not provided.

This is problematic because there exist cards that do not provide this
information, and cards that do provide the information depending on
which firmware version is in use.

The Neutron validation code does not have a concept of a optional key in
the binding profile, and since the information is not required for the
vnic direct + capabilities switchdev workflow I'm inclined to think Nova
should refrain from providing it in this case.

If this would be hard for Nova to do, we would at the very least need it
to always provide the keys and just not populate the values when they
are not there, with that the Neutron validation code could make the
validation of the values optional for the vnic direct + capabilties
switchdev workflow.

To unblock you while we figure out how to solve this properly you could
apply this patch [0] to your neutron-api units.

0: https://pastebin.ubuntu.com/p/3dsHX4rHdT/

** Also affects: nova
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1975743

Title:
  ML2 OVN - Creating an instance with hardware offloaded port is broken

Status in neutron:
  New
Status in OpenStack Compute (nova):
  New

Bug description:
  OpenStack Release: Yoga
  Platform: Ubuntu focal

  Creating an instance with vnic-type ‘direct’ port and ‘switchdev’ binding-profile is failing over the following validation error:
  ```
  2022-05-25 19:13:40.331 125269 DEBUG neutron.api.v2.base [req-504a0204-6f1a-46ae-8b95-dcfdf2692f91 b2a31335e63b4dd391cc3e6bf4600fe1 - - 654b9b803e6a4a68b31676c16973e3cc 654b9b803e6a4a68b31676c16973e3cc] Request body: {'port': {'device_id': 'd46aef48-e42e-49c8-af9f-a83768747b4f', 'device_owner': 'compute:nova', 'binding:profile': {'capabilities': ['switchdev'], 'pci_vendor_info': '15b3:101e', 'pci_slot': '0000:08:03.2', 'physical_network': None, 'card_serial_number': 'MT2034X11488', 'pf_mac_address': '04:3f:72:9e:0b:a1', 'vf_num': 7}, 'binding:host_id': 'node3.maas', 'dns_name': 'vm1'}} prepare_request_body /usr/lib/python3/dist-packages/neutron/api/v2/base.py:729

  
  2022-05-25 19:13:40.429 125269 DEBUG neutron_lib.callbacks.manager [req-504a0204-6f1a-46ae-8b95-dcfdf2692f91 b2a31335e63b4dd391cc3e6bf4600fe1 - - 654b9b803e6a4a68b31676c16973e3cc 654b9b803e6a4a68b31676c16973e3cc] Publish callbacks ['neutron.plugins.ml2.plugin.SecurityGroupDbMixin._ensure_default_security_group_handler-1311372', 'neutron.services.ovn_l3.plugin.OVNL3RouterPlugin._port_update-8735219071964'] for port (0f1e4e9c-68ef-4b38-a3bc-68e624bca6c7), before_update _notify_loop /usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py:176
  2022-05-25 19:13:41.221 125269 DEBUG neutron.notifiers.nova [req-504a0204-6f1a-46ae-8b95-dcfdf2692f91 b2a31335e63b4dd391cc3e6bf4600fe1 - - 654b9b803e6a4a68b31676c16973e3cc 654b9b803e6a4a68b31676c16973e3cc] Ignoring state change previous_port_status: DOWN current_port_status: DOWN port_id 0f1e4e9c-68ef-4b38-a3bc-68e624bca6c7 record_port_status_changed /usr/lib/python3/dist-packages/neutron/notifiers/nova.py:233
  2022-05-25 19:13:41.229 125269 DEBUG neutron_lib.callbacks.manager [req-504a0204-6f1a-46ae-8b95-dcfdf2692f91 b2a31335e63b4dd391cc3e6bf4600fe1 - - 654b9b803e6a4a68b31676c16973e3cc 654b9b803e6a4a68b31676c16973e3cc] Publish callbacks [] for port (0f1e4e9c-68ef-4b38-a3bc-68e624bca6c7), precommit_update _notify_loop /usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py:176

  
  2022-05-25 19:13:41.229 125269 ERROR neutron.plugins.ml2.managers [req-504a0204-6f1a-46ae-8b95-dcfdf2692f91 b2a31335e63b4dd391cc3e6bf4600fe1 - - 654b9b803e6a4a68b31676c16973e3cc 654b9b803e6a4a68b31676c16973e3cc] Mechanism driver 'ovn' failed in update_port_precommit: neutron_lib.exceptions.InvalidInput: Invalid input for operation: Invalid binding:profile. too many parameters.
  2022-05-25 19:13:41.229 125269 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
  2022-05-25 19:13:41.229 125269 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/managers.py", line 482, in _call_on_drivers
  2022-05-25 19:13:41.229 125269 ERROR neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
  2022-05-25 19:13:41.229 125269 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 792, in update_port_precommit
  2022-05-25 19:13:41.229 125269 ERROR neutron.plugins.ml2.managers     ovn_utils.validate_and_get_data_from_binding_profile(port)
  2022-05-25 19:13:41.229 125269 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3/dist-packages/neutron/common/ovn/utils.py", line 266, in validate_and_get_data_from_binding_profile
  2022-05-25 19:13:41.229 125269 ERROR neutron.plugins.ml2.managers     raise n_exc.InvalidInput(error_message=msg)
  2022-05-25 19:13:41.229 125269 ERROR neutron.plugins.ml2.managers neutron_lib.exceptions.InvalidInput: Invalid input for operation: Invalid binding:profile. too many parameters.
  2022-05-25 19:13:41.229 125269 ERROR neutron.plugins.ml2.managers 
  ```

  Seems like the issue is related to the commit from: 
  https://review.opendev.org/c/openstack/neutron/+/818420

  
  To reproduce:
  https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-ovn.html

  1. Prepare a setup with SR-IOV adjusted for OVN HW Offload
  2. Create a port with switchdev capabilities

  $ openstack port create direct_overlay2 --vnic-type=direct --network
  gen_data --binding-profile '{"capabilities":["switchdev"]}'
  --security-group my_policy

  3. Create an instance

  $ openstack server create --key-name bastion --flavor d1.demo --image
  ubuntu --port direct_overlay1 vm1 --availability-zone nova:node3.maas

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1975743/+subscriptions