yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2122066] [NEW] ovs agent missing openflow rules due to python error

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Mike Lowe <2122066@xxxxxxxxxxxxxxxxxx>
Date: Thu, 04 Sep 2025 18:16:05 -0000
Reply-to: Bug 2122066 <2122066@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

When processing some ports the bug triggering this traceback interrupts
processing of openflow rules for ovs, one symptom is affected instances
not getting the proper wiring within ovs to connect tenant networking
tunnels cutting an affected instance off from DHCP.


2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-f236b063-cbce-4b8e-ba36-2c1a5e6dc940 - - - - - -] Error while processing VIF ports: AttributeError: 'dict' object has no attribute 'id'
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2874, in rpc_loop
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     failed_devices = self.process_network_ports(
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2267, in process_network_ports
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.treat_devices_added_or_updated(
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2088, in treat_devices_added_or_updated
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     need_binding = self.treat_vif_port(port, details['port_id'],
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1952, in treat_vif_port
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_needs_binding = self.port_bound(
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent                          ^^^^^^^^^^^^^^^^
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1221, in port_bound
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.dvr_agent.bind_port_to_dvr(port, lvm,
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3/dist-packages/osprofiler/profiler.py", line 159, in wrapper
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent              ^^^^^^^^^^^^^^^^^^
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py", line 672, in bind_port_to_dvr
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.unbind_port_from_dvr(port, local_vlan_map)
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3/dist-packages/osprofiler/profiler.py", line 159, in wrapper
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent              ^^^^^^^^^^^^^^^^^^
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py", line 875, in unbind_port_from_dvr
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._unbind_port_on_dvr_subnet(vif_port,
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py", line 809, in _unbind_port_on_dvr_subnet
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     if lport.id != port.vif_id:
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent        ^^^^^^^^
2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent AttributeError: 'dict' object has no attribute 'id'

An extra logging statement confirms lport is definitely a dict and id is
a key not an attribute.

2025-09-04 12:56:07.612 546122 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_dvr_neutron_agent [None req-f236b063-cbce-4b8e-ba36-2c1a5e6dc940 - - - - - -] lport: {'id': 'f48ce516-6a37-4ccd-98db-12d06dd654b1', 'name': 'blast-k8s-controlplane-01', 'network_id': '37bfe833-9
4f3-4477-9a20-6e0724f6fa65', 'tenant_id': '8885b9a4c8bb471e9a30b1d67744cbc6', 'mac_address': 'fa:16:3e:4c:3d:af', 'admin_state_up': True, 'status': 'DOWN', 'device_id': 'f3a1bdf2-e47d-41c0-9621-5c0a915158b9', 'device_owner': 'compute:nova', 'standard_attr_id': 2018578, 'project_id
': '8885b9a4c8bb471e9a30b1d67744cbc6'} _unbind_port_on_dvr_subnet /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py:808


Looks to have been introduced by https://opendev.org/openstack/neutron/commit/1364715541247d9aa941f3305acef4b8f9e37274

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2122066

Title:
  ovs agent missing openflow rules due to python error

Status in neutron:
  New

Bug description:
  When processing some ports the bug triggering this traceback
  interrupts processing of openflow rules for ovs, one symptom is
  affected instances not getting the proper wiring within ovs to connect
  tenant networking tunnels cutting an affected instance off from DHCP.

  
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-f236b063-cbce-4b8e-ba36-2c1a5e6dc940 - - - - - -] Error while processing VIF ports: AttributeError: 'dict' object has no attribute 'id'
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2874, in rpc_loop
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     failed_devices = self.process_network_ports(
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2267, in process_network_ports
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.treat_devices_added_or_updated(
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2088, in treat_devices_added_or_updated
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     need_binding = self.treat_vif_port(port, details['port_id'],
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1952, in treat_vif_port
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_needs_binding = self.port_bound(
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent                          ^^^^^^^^^^^^^^^^
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1221, in port_bound
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.dvr_agent.bind_port_to_dvr(port, lvm,
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3/dist-packages/osprofiler/profiler.py", line 159, in wrapper
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent              ^^^^^^^^^^^^^^^^^^
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py", line 672, in bind_port_to_dvr
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.unbind_port_from_dvr(port, local_vlan_map)
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3/dist-packages/osprofiler/profiler.py", line 159, in wrapper
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent              ^^^^^^^^^^^^^^^^^^
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py", line 875, in unbind_port_from_dvr
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._unbind_port_on_dvr_subnet(vif_port,
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py", line 809, in _unbind_port_on_dvr_subnet
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     if lport.id != port.vif_id:
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent        ^^^^^^^^
  2025-09-04 12:56:07.612 546122 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent AttributeError: 'dict' object has no attribute 'id'

  An extra logging statement confirms lport is definitely a dict and id
  is a key not an attribute.

  2025-09-04 12:56:07.612 546122 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_dvr_neutron_agent [None req-f236b063-cbce-4b8e-ba36-2c1a5e6dc940 - - - - - -] lport: {'id': 'f48ce516-6a37-4ccd-98db-12d06dd654b1', 'name': 'blast-k8s-controlplane-01', 'network_id': '37bfe833-9
  4f3-4477-9a20-6e0724f6fa65', 'tenant_id': '8885b9a4c8bb471e9a30b1d67744cbc6', 'mac_address': 'fa:16:3e:4c:3d:af', 'admin_state_up': True, 'status': 'DOWN', 'device_id': 'f3a1bdf2-e47d-41c0-9621-5c0a915158b9', 'device_owner': 'compute:nova', 'standard_attr_id': 2018578, 'project_id
  ': '8885b9a4c8bb471e9a30b1d67744cbc6'} _unbind_port_on_dvr_subnet /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py:808

  
  Looks to have been introduced by https://opendev.org/openstack/neutron/commit/1364715541247d9aa941f3305acef4b8f9e37274

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2122066/+subscriptions