yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2117405] [NEW] test_subport_delete random failure

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: yatin <2117405@xxxxxxxxxxxxxxxxxx>
Date: Mon, 21 Jul 2025 14:37:47 -0000
Reply-to: Bug 2117405 <2117405@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Seen once in
https://60cdea3a73ef3279f0cb-668353d8fd369f78e567fd7dfce5ad3e.ssl.cf1.rackcdn.com/openstack/39f3f456430b4ffaae27910f37ae2ff2/testr_results.html

Failed as:- 
ft1.4: neutron.tests.functional.services.trunk.drivers.ovn.test_trunk_driver.TestOVNTrunkDriver.test_subport_deletetesttools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/services/trunk/drivers/ovn/test_trunk_driver.py", line 139, in test_subport_delete
    self._verify_trunk_info(new_trunk, has_items=False)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/services/trunk/drivers/ovn/test_trunk_driver.py", line 104, in _verify_trunk_info
    self.assertCountEqual(ovn_subports_info, neutron_subports_info)
  File "/usr/lib/python3.12/unittest/case.py", line 1216, in assertCountEqual
    self.fail(msg)
  File "/usr/lib/python3.12/unittest/case.py", line 715, in fail
    raise self.failureException(msg)
AssertionError: Element counts were not equal:
First has 1, Second has 0:  {'port_id': '6f0ba550-0065-4f27-a77f-e54c9410c114', 'parent_port_id': ['af656dfa-0948-4e1c-9d2e-d95bfedac748'], 'tag': [1000], 'device_owner': 'trunk:subport', 'revision_number': '2'}


>From server log:-
2025-07-21 06:54:03.925 77635 INFO neutron.db.ovn_revision_numbers_db [None req-db456857-6070-40f0-ae0d-4f624a863874 - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Successfully bumped revision number for resource 6f0ba550-0065-4f27-a77f-e54c9410c114 (type: ports) to 2
2025-07-21 06:54:03.944 77635 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-db456857-6070-40f0-ae0d-4f624a863874 - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Hash Ring: Node 49831373-7688-4baa-9e24-5b43e7eb41ab (host: npae35f344f6aa4) handling event "delete" for row 33e92390-9736-4a8c-92f1-875b5cc882a0 (table: Port_Binding) notify /home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:852
2025-07-21 06:54:03.945 77635 DEBUG ovsdbapp.backend.ovs_idl.event [None req-db456857-6070-40f0-ae0d-4f624a863874 - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Matched DELETE: WaitForPortBindingDeleteEvent(events=('delete',), table='Port_Binding', conditions=(('logical_port', '=', '6f0ba550-0065-4f27-a77f-e54c9410c114'),), old_conditions=None), priority=10 to row=Port_Binding(parent_port=[], mac=['12:34:56:78:f0:2b 10.0.0.104'], chassis=[], ha_chassis_group=[], options={}, type=, additional_encap=[], tag=[], requested_additional_chassis=[], requested_chassis=[], up=[False], tunnel_key=1, logical_port=6f0ba550-0065-4f27-a77f-e54c9410c114, additional_chassis=[], gateway_chassis=[], encap=[], external_ids={'neutron:cidrs': '10.0.0.104/24', 'neutron:device_id': '', 'neutron:device_owner': '', 'neutron:mtu': '', 'neutron:network_name': 'neutron-6e7d9af6-da58-4834-aa22-227a40632bfc', 'neutron:port_capabilities': '', 'neutron:port_name': '', 'neutron:project_id': '46f70361-ba71-4bd0-9769-3573fd227c4b', 'neutron:revision_number': '1', 'neutron:security_group_ids': 'a351b91c-6962-4145-8bba-2e0d968a30cf', 'neutron:subnet_pool_addr_scope4': '', 'neutron:subnet_pool_addr_scope6': '', 'neutron:vnic_type': 'normal'}, virtual_parent=[], nat_addresses=[], datapath=ea6f29c2-e198-4af4-8a7b-9e112f0f0521, port_security=['12:34:56:78:f0:2b 10.0.0.104'], mirror_rules=[]) old= matches /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/event.py:55
2025-07-21 06:54:03.946 77635 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-db456857-6070-40f0-ae0d-4f624a863874 - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Hash Ring: Node 49831373-7688-4baa-9e24-5b43e7eb41ab (host: npae35f344f6aa4) handling event "create" for row 57ac2acd-4f53-4eae-a0f1-93e5b912648c (table: Port_Binding) notify /home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:852
2025-07-21 06:54:03.947 77635 DEBUG neutron.services.trunk.drivers.ovn.trunk_driver [None req-db456857-6070-40f0-ae0d-4f624a863874 - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Unsetting parent for subport 6f0ba550-0065-4f27-a77f-e54c9410c114 _unset_binding_profile /home/zuul/src/opendev.org/openstack/neutron/neutron/services/trunk/drivers/ovn/trunk_driver.py:121
2025-07-21 06:54:03.978 77635 DEBUG neutron.services.trunk.drivers.ovn.trunk_driver [None req-db456857-6070-40f0-ae0d-4f624a863874 - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Done unsetting parent for subport 6f0ba550-0065-4f27-a77f-e54c9410c114 _unset_binding_profile /home/zuul/src/opendev.org/openstack/neutron/neutron/services/trunk/drivers/ovn/trunk_driver.py:158
2025-07-21 06:54:03.980 77635 DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-db456857-6070-40f0-ae0d-4f624a863874 - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=6f0ba550-0065-4f27-a77f-e54c9410c114, resource=Port(admin_state_up=True,allowed_address_pairs=[],binding_levels=[],bindings=[PortBinding],created_at=2025-07-21T06:54:03Z,data_plane_status=None,description='',device_id='',device_owner='',device_profile=<?>,dhcp_options=[],distributed_bindings=[],dns=None,fixed_ips=[IPAllocation],hardware_offload_type=<?>,hints=None,id=6f0ba550-0065-4f27-a77f-e54c9410c114,mac_address=12:34:56:78:f0:2b,name='',network_id=6e7d9af6-da58-4834-aa22-227a40632bfc,numa_affinity_policy=<?>,project_id='46f70361-ba71-4bd0-9769-3573fd227c4b',qos_network_policy_id=None,qos_policy_id=None,revision_number=1,security=PortSecurity(6f0ba550-0065-4f27-a77f-e54c9410c114),security_group_ids=set([a351b91c-6962-4145-8bba-2e0d968a30cf]),status='DOWN',trusted=<?>,updated_at=2025-07-21T06:54:03Z), resource_type=ports, if_exists=True) do_commit /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89
2025-07-21 06:54:03.981 77635 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-db456857-6070-40f0-ae0d-4f624a863874 - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 121, in run
    txn.results.put(txn.do_commit())
                    ^^^^^^^^^^^^^^^
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 92, in do_commit
    command.run_idl(txn)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/commands.py", line 1046, in run_idl
    raise ovn_exc.RevisionConflict(
neutron.common.ovn.exceptions.RevisionConflict: OVN revision number for 6f0ba550-0065-4f27-a77f-e54c9410c114 (type: ports) is higher than the given resource. Skipping update

2025-07-21 06:54:03.982 77635 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-db456857-6070-40f0-ae0d-4f624a863874 - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Transaction aborted. Reason: OVN revision number for 6f0ba550-0065-4f27-a77f-e54c9410c114 (type: ports) is higher than the given resource. Skipping update
2025-07-21 06:54:03.998 77635 DEBUG neutron.db.ovn_revision_numbers_db [None req-db456857-6070-40f0-ae0d-4f624a863874 - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Skip bumping the revision number for 6f0ba550-0065-4f27-a77f-e54c9410c114 (type: ports) to 1. A higher version is already registered in the database (2) bump_revision /home/zuul/src/opendev.org/openstack/neutron/neutron/db/ovn_revision_numbers_db.py:194

There were two commands[1][2] to be executed as part of the transaction, first failed and second(unsetting tag and parent_name) didn't executed, and this resulted into test to fail as test was expecting no port with parent and tag set to exist.
So we need to check what resulted into this ovn revision command to try to set ovn revision number to lower value(1) and fail.

[1] https://opendev.org/openstack/neutron/src/commit/e69505d293ef12999f948531acbde75e16a65cd4/neutron/services/trunk/drivers/ovn/trunk_driver.py#L128-L130
[2] https://opendev.org/openstack/neutron/src/commit/e69505d293ef12999f948531acbde75e16a65cd4/neutron/services/trunk/drivers/ovn/trunk_driver.py#L150-L157

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2117405

Title:
  test_subport_delete random failure

Status in neutron:
  New

Bug description:
  Seen once in
  https://60cdea3a73ef3279f0cb-668353d8fd369f78e567fd7dfce5ad3e.ssl.cf1.rackcdn.com/openstack/39f3f456430b4ffaae27910f37ae2ff2/testr_results.html

  Failed as:- 
  ft1.4: neutron.tests.functional.services.trunk.drivers.ovn.test_trunk_driver.TestOVNTrunkDriver.test_subport_deletetesttools.testresult.real._StringException: Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
      return f(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/services/trunk/drivers/ovn/test_trunk_driver.py", line 139, in test_subport_delete
      self._verify_trunk_info(new_trunk, has_items=False)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/services/trunk/drivers/ovn/test_trunk_driver.py", line 104, in _verify_trunk_info
      self.assertCountEqual(ovn_subports_info, neutron_subports_info)
    File "/usr/lib/python3.12/unittest/case.py", line 1216, in assertCountEqual
      self.fail(msg)
    File "/usr/lib/python3.12/unittest/case.py", line 715, in fail
      raise self.failureException(msg)
  AssertionError: Element counts were not equal:
  First has 1, Second has 0:  {'port_id': '6f0ba550-0065-4f27-a77f-e54c9410c114', 'parent_port_id': ['af656dfa-0948-4e1c-9d2e-d95bfedac748'], 'tag': [1000], 'device_owner': 'trunk:subport', 'revision_number': '2'}

  
  From server log:-
  2025-07-21 06:54:03.925 77635 INFO neutron.db.ovn_revision_numbers_db [None req-db456857-6070-40f0-ae0d-4f624a863874 - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Successfully bumped revision number for resource 6f0ba550-0065-4f27-a77f-e54c9410c114 (type: ports) to 2
  2025-07-21 06:54:03.944 77635 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-db456857-6070-40f0-ae0d-4f624a863874 - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Hash Ring: Node 49831373-7688-4baa-9e24-5b43e7eb41ab (host: npae35f344f6aa4) handling event "delete" for row 33e92390-9736-4a8c-92f1-875b5cc882a0 (table: Port_Binding) notify /home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:852
  2025-07-21 06:54:03.945 77635 DEBUG ovsdbapp.backend.ovs_idl.event [None req-db456857-6070-40f0-ae0d-4f624a863874 - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Matched DELETE: WaitForPortBindingDeleteEvent(events=('delete',), table='Port_Binding', conditions=(('logical_port', '=', '6f0ba550-0065-4f27-a77f-e54c9410c114'),), old_conditions=None), priority=10 to row=Port_Binding(parent_port=[], mac=['12:34:56:78:f0:2b 10.0.0.104'], chassis=[], ha_chassis_group=[], options={}, type=, additional_encap=[], tag=[], requested_additional_chassis=[], requested_chassis=[], up=[False], tunnel_key=1, logical_port=6f0ba550-0065-4f27-a77f-e54c9410c114, additional_chassis=[], gateway_chassis=[], encap=[], external_ids={'neutron:cidrs': '10.0.0.104/24', 'neutron:device_id': '', 'neutron:device_owner': '', 'neutron:mtu': '', 'neutron:network_name': 'neutron-6e7d9af6-da58-4834-aa22-227a40632bfc', 'neutron:port_capabilities': '', 'neutron:port_name': '', 'neutron:project_id': '46f70361-ba71-4bd0-9769-3573fd227c4b', 'neutron:revision_number': '1', 'neutron:security_group_ids': 'a351b91c-6962-4145-8bba-2e0d968a30cf', 'neutron:subnet_pool_addr_scope4': '', 'neutron:subnet_pool_addr_scope6': '', 'neutron:vnic_type': 'normal'}, virtual_parent=[], nat_addresses=[], datapath=ea6f29c2-e198-4af4-8a7b-9e112f0f0521, port_security=['12:34:56:78:f0:2b 10.0.0.104'], mirror_rules=[]) old= matches /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/event.py:55
  2025-07-21 06:54:03.946 77635 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-db456857-6070-40f0-ae0d-4f624a863874 - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Hash Ring: Node 49831373-7688-4baa-9e24-5b43e7eb41ab (host: npae35f344f6aa4) handling event "create" for row 57ac2acd-4f53-4eae-a0f1-93e5b912648c (table: Port_Binding) notify /home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:852
  2025-07-21 06:54:03.947 77635 DEBUG neutron.services.trunk.drivers.ovn.trunk_driver [None req-db456857-6070-40f0-ae0d-4f624a863874 - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Unsetting parent for subport 6f0ba550-0065-4f27-a77f-e54c9410c114 _unset_binding_profile /home/zuul/src/opendev.org/openstack/neutron/neutron/services/trunk/drivers/ovn/trunk_driver.py:121
  2025-07-21 06:54:03.978 77635 DEBUG neutron.services.trunk.drivers.ovn.trunk_driver [None req-db456857-6070-40f0-ae0d-4f624a863874 - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Done unsetting parent for subport 6f0ba550-0065-4f27-a77f-e54c9410c114 _unset_binding_profile /home/zuul/src/opendev.org/openstack/neutron/neutron/services/trunk/drivers/ovn/trunk_driver.py:158
  2025-07-21 06:54:03.980 77635 DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-db456857-6070-40f0-ae0d-4f624a863874 - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=6f0ba550-0065-4f27-a77f-e54c9410c114, resource=Port(admin_state_up=True,allowed_address_pairs=[],binding_levels=[],bindings=[PortBinding],created_at=2025-07-21T06:54:03Z,data_plane_status=None,description='',device_id='',device_owner='',device_profile=<?>,dhcp_options=[],distributed_bindings=[],dns=None,fixed_ips=[IPAllocation],hardware_offload_type=<?>,hints=None,id=6f0ba550-0065-4f27-a77f-e54c9410c114,mac_address=12:34:56:78:f0:2b,name='',network_id=6e7d9af6-da58-4834-aa22-227a40632bfc,numa_affinity_policy=<?>,project_id='46f70361-ba71-4bd0-9769-3573fd227c4b',qos_network_policy_id=None,qos_policy_id=None,revision_number=1,security=PortSecurity(6f0ba550-0065-4f27-a77f-e54c9410c114),security_group_ids=set([a351b91c-6962-4145-8bba-2e0d968a30cf]),status='DOWN',trusted=<?>,updated_at=2025-07-21T06:54:03Z), resource_type=ports, if_exists=True) do_commit /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89
  2025-07-21 06:54:03.981 77635 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-db456857-6070-40f0-ae0d-4f624a863874 - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 121, in run
      txn.results.put(txn.do_commit())
                      ^^^^^^^^^^^^^^^
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 92, in do_commit
      command.run_idl(txn)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/commands.py", line 1046, in run_idl
      raise ovn_exc.RevisionConflict(
  neutron.common.ovn.exceptions.RevisionConflict: OVN revision number for 6f0ba550-0065-4f27-a77f-e54c9410c114 (type: ports) is higher than the given resource. Skipping update

  2025-07-21 06:54:03.982 77635 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-db456857-6070-40f0-ae0d-4f624a863874 - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Transaction aborted. Reason: OVN revision number for 6f0ba550-0065-4f27-a77f-e54c9410c114 (type: ports) is higher than the given resource. Skipping update
  2025-07-21 06:54:03.998 77635 DEBUG neutron.db.ovn_revision_numbers_db [None req-db456857-6070-40f0-ae0d-4f624a863874 - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Skip bumping the revision number for 6f0ba550-0065-4f27-a77f-e54c9410c114 (type: ports) to 1. A higher version is already registered in the database (2) bump_revision /home/zuul/src/opendev.org/openstack/neutron/neutron/db/ovn_revision_numbers_db.py:194

  There were two commands[1][2] to be executed as part of the transaction, first failed and second(unsetting tag and parent_name) didn't executed, and this resulted into test to fail as test was expecting no port with parent and tag set to exist.
  So we need to check what resulted into this ovn revision command to try to set ovn revision number to lower value(1) and fail.

  [1] https://opendev.org/openstack/neutron/src/commit/e69505d293ef12999f948531acbde75e16a65cd4/neutron/services/trunk/drivers/ovn/trunk_driver.py#L128-L130
  [2] https://opendev.org/openstack/neutron/src/commit/e69505d293ef12999f948531acbde75e16a65cd4/neutron/services/trunk/drivers/ovn/trunk_driver.py#L150-L157

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2117405/+subscriptions