yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2039417] [NEW] [master][stable/2023.2][functional] test_maintenance.TestMaintenance tests fails randomly

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: yatin <2039417@xxxxxxxxxxxxxxxxxx>
Date: Mon, 16 Oct 2023 07:43:42 -0000
Reply-to: Bug 2039417 <2039417@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

The functional test fails randomly as:-
ft1.3: neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_maintenance.TestMaintenance.test_porttesttools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_maintenance.py", line 306, in test_port
    neutron_net = self._create_network('network1')
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_maintenance.py", line 68, in _create_network
    return self.deserialize(self.fmt, res)['network']
KeyError: 'network'

neutron server log when it fails:-
2023-10-06 08:16:33.966 37713 DEBUG neutron.db.ovn_revision_numbers_db [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] create_initial_revision uuid=6ab7bd53-277b-4133-ac32-52b1d0c90f78, type=security_groups, rev=-1 create_initial_revision /home/zuul/src/opendev.org/openstack/neutron/neutron/db/ovn_revision_numbers_db.py:108
2023-10-06 08:16:33.973 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
2023-10-06 08:16:33.974 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
    txn.results.put(txn.do_commit())
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
    raise RuntimeError(msg)
RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it

2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver._create_security_group-11049860 security_group, after_create: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/callbacks/manager.py", line 181, in _notify_loop
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     callback(resource, event, trigger, payload=payload)
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 409, in _create_security_group
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     self._ovn_client.create_security_group(context,
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2328, in create_security_group
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     with self._nb_idl.transaction(check_error=True) as txn:
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     next(self.gen)
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     with super(OvsdbNbOvnIdl, self).transaction(*args, **kwargs) as t:
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     next(self.gen)
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 114, in transaction
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     with self.create_transaction(
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 71, in __exit__
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     self.result = self.commit()
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     raise result.ex
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     txn.results.put(txn.do_commit())
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     raise RuntimeError(msg)
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager
...
2023-10-06 08:16:34.047 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
2023-10-06 08:16:34.048 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
    txn.results.put(txn.do_commit())
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
    raise RuntimeError(msg)
RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it

2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Mechanism driver 'ovn' failed in create_network_postcommit: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 497, in _call_on_drivers
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 579, in create_network_postcommit
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     self._ovn_client.create_network(context.plugin_context, network)
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1927, in create_network
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     with self._nb_idl.transaction(check_error=True) as txn:
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     next(self.gen)
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     with super(OvsdbNbOvnIdl, self).transaction(*args, **kwargs) as t:
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     next(self.gen)
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 114, in transaction
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     with self.create_transaction(
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 71, in __exit__
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     self.result = self.commit()
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     raise result.ex
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     txn.results.put(txn.do_commit())
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     raise RuntimeError(msg)
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers 
2023-10-06 08:16:34.052 37713 ERROR neutron.plugins.ml2.plugin [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] mechanism_manager.create_network_postcommit failed, deleting network '767ee503-3645-4762-b0d6-e673b3c14170': neutron.plugins.ml2.common.exceptions.MechanismDriverError
...
2023-10-06 08:16:34.153 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
2023-10-06 08:16:34.154 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
    txn.results.put(txn.do_commit())
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
    raise RuntimeError(msg)
RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
: neutron.plugins.ml2.common.exceptions.MechanismDriverError
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Mechanism driver 'ovn' failed in delete_network_postcommit: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1219, in _after_create_network
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     self.mechanism_manager.create_network_postcommit(mech_context)
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 547, in create_network_postcommit
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     self._call_on_drivers("create_network_postcommit", context)
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 513, in _call_on_drivers
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     raise ml2_exc.MechanismDriverError(
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers neutron.plugins.ml2.common.exceptions.MechanismDriverError
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers During handling of the above exception, another exception occurred:
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 497, in _call_on_drivers
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 631, in delete_network_postcommit
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     self._ovn_client.delete_network(
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1938, in delete_network
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     with self._nb_idl.transaction(check_error=True) as txn:
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     next(self.gen)
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     with super(OvsdbNbOvnIdl, self).transaction(*args, **kwargs) as t:
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     next(self.gen)
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 114, in transaction
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     with self.create_transaction(
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 71, in __exit__
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     self.result = self.commit()
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     raise result.ex
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     txn.results.put(txn.do_commit())
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     raise RuntimeError(msg)
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers
2023-10-06 08:16:34.162 37713 ERROR neutron.plugins.ml2.plugin [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] mechanism_manager.delete_network_postcommit failed: neutron.plugins.ml2.common.exceptions.MechanismDriverError
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] POST failed.: neutron.plugins.ml2.common.exceptions.MechanismDriverError
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/pecan/core.py", line 693, in __call__
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.invoke_controller(controller, args, kwargs, state)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/pecan/core.py", line 584, in invoke_controller
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     result = controller(*args, **kwargs)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception() as ectxt:
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 181, in wrapped
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/utils.py", line 65, in wrapped
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/resource.py", line 163, in post
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return self.create(resources)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/resource.py", line 181, in create
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return {key: creator(*creator_args, **creator_kwargs)}
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 731, in inner
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 223, in wrapped
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f_with_retry(*args, **kwargs,
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception() as ectxt:
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 181, in wrapped
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1211, in create_network
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return self._after_create_network(context, result, mech_context)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1221, in _after_create_network
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1219, in _after_create_network
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.mechanism_manager.create_network_postcommit(mech_context)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 547, in create_network_postcommit
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self._call_on_drivers("create_network_postcommit", context)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 513, in _call_on_drivers
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     raise ml2_exc.MechanismDriverError(
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation neutron.plugins.ml2.common.exceptions.MechanismDriverError
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation
2023-10-06 08:16:34.166 37713 DEBUG neutron.pecan_wsgi.hooks.notifier [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] No notification will be sent due to unsuccessful status code: 500 after /home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/hooks/notifier.py:78

^ From
https://9ce43a75e3387ceb8909-2b4f2fa211fea8445ec0f4a568f6056b.ssl.cf2.rackcdn.com/897438/1/gate/neutron-
functional-with-uwsgi/7147770/controller/logs/dsvm-functional-
logs.tar.gz

Other example failures:-
- https://5a1da1856f577d094ffa-09419825a71508c658bc83d968b5572f.ssl.cf2.rackcdn.com/890162/3/check/neutron-functional-with-uwsgi/4ad32c0/testr_results.html
- https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d3b/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-uwsgi-fips/d3b6e71/testr_results.html
- https://cf7800dce1b91328e4f8-616b34f6dd94893c803bfdcdd39959b5.ssl.cf5.rackcdn.com/897332/4/gate/neutron-functional-with-uwsgi/fe7d494/testr_results.html
- https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d3b/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-uwsgi-fips/d3b6e71/testr_results.html
- https://cf7800dce1b91328e4f8-616b34f6dd94893c803bfdcdd39959b5.ssl.cf5.rackcdn.com/897332/4/gate/neutron-functional-with-uwsgi/fe7d494/testr_results.html

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2039417

Title:
  [master][stable/2023.2][functional] test_maintenance.TestMaintenance
  tests fails randomly

Status in neutron:
  New

Bug description:
  The functional test fails randomly as:-
  ft1.3: neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_maintenance.TestMaintenance.test_porttesttools.testresult.real._StringException: Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
      return f(self, *args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_maintenance.py", line 306, in test_port
      neutron_net = self._create_network('network1')
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_maintenance.py", line 68, in _create_network
      return self.deserialize(self.fmt, res)['network']
  KeyError: 'network'

  neutron server log when it fails:-
  2023-10-06 08:16:33.966 37713 DEBUG neutron.db.ovn_revision_numbers_db [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] create_initial_revision uuid=6ab7bd53-277b-4133-ac32-52b1d0c90f78, type=security_groups, rev=-1 create_initial_revision /home/zuul/src/opendev.org/openstack/neutron/neutron/db/ovn_revision_numbers_db.py:108
  2023-10-06 08:16:33.973 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
  2023-10-06 08:16:33.974 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
      txn.results.put(txn.do_commit())
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
      raise RuntimeError(msg)
  RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it

  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver._create_security_group-11049860 security_group, after_create: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/callbacks/manager.py", line 181, in _notify_loop
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     callback(resource, event, trigger, payload=payload)
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 409, in _create_security_group
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     self._ovn_client.create_security_group(context,
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2328, in create_security_group
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     with self._nb_idl.transaction(check_error=True) as txn:
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     next(self.gen)
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     with super(OvsdbNbOvnIdl, self).transaction(*args, **kwargs) as t:
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     next(self.gen)
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 114, in transaction
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     with self.create_transaction(
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 71, in __exit__
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     self.result = self.commit()
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     raise result.ex
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     txn.results.put(txn.do_commit())
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     raise RuntimeError(msg)
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager
  ...
  2023-10-06 08:16:34.047 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
  2023-10-06 08:16:34.048 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
      txn.results.put(txn.do_commit())
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
      raise RuntimeError(msg)
  RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it

  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Mechanism driver 'ovn' failed in create_network_postcommit: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 497, in _call_on_drivers
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 579, in create_network_postcommit
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     self._ovn_client.create_network(context.plugin_context, network)
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1927, in create_network
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     with self._nb_idl.transaction(check_error=True) as txn:
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     next(self.gen)
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     with super(OvsdbNbOvnIdl, self).transaction(*args, **kwargs) as t:
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     next(self.gen)
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 114, in transaction
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     with self.create_transaction(
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 71, in __exit__
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     self.result = self.commit()
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     raise result.ex
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     txn.results.put(txn.do_commit())
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     raise RuntimeError(msg)
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers 
  2023-10-06 08:16:34.052 37713 ERROR neutron.plugins.ml2.plugin [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] mechanism_manager.create_network_postcommit failed, deleting network '767ee503-3645-4762-b0d6-e673b3c14170': neutron.plugins.ml2.common.exceptions.MechanismDriverError
  ...
  2023-10-06 08:16:34.153 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
  2023-10-06 08:16:34.154 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
      txn.results.put(txn.do_commit())
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
      raise RuntimeError(msg)
  RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
  : neutron.plugins.ml2.common.exceptions.MechanismDriverError
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Mechanism driver 'ovn' failed in delete_network_postcommit: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1219, in _after_create_network
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     self.mechanism_manager.create_network_postcommit(mech_context)
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 547, in create_network_postcommit
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     self._call_on_drivers("create_network_postcommit", context)
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 513, in _call_on_drivers
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     raise ml2_exc.MechanismDriverError(
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers neutron.plugins.ml2.common.exceptions.MechanismDriverError
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers During handling of the above exception, another exception occurred:
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 497, in _call_on_drivers
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 631, in delete_network_postcommit
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     self._ovn_client.delete_network(
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1938, in delete_network
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     with self._nb_idl.transaction(check_error=True) as txn:
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     next(self.gen)
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     with super(OvsdbNbOvnIdl, self).transaction(*args, **kwargs) as t:
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     next(self.gen)
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 114, in transaction
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     with self.create_transaction(
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 71, in __exit__
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     self.result = self.commit()
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     raise result.ex
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     txn.results.put(txn.do_commit())
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     raise RuntimeError(msg)
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers
  2023-10-06 08:16:34.162 37713 ERROR neutron.plugins.ml2.plugin [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] mechanism_manager.delete_network_postcommit failed: neutron.plugins.ml2.common.exceptions.MechanismDriverError
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] POST failed.: neutron.plugins.ml2.common.exceptions.MechanismDriverError
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/pecan/core.py", line 693, in __call__
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.invoke_controller(controller, args, kwargs, state)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/pecan/core.py", line 584, in invoke_controller
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     result = controller(*args, **kwargs)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 135, in wrapped
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception() as ectxt:
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 181, in wrapped
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/utils.py", line 65, in wrapped
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/resource.py", line 163, in post
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return self.create(resources)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/resource.py", line 181, in create
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return {key: creator(*creator_args, **creator_kwargs)}
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 731, in inner
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 223, in wrapped
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f_with_retry(*args, **kwargs,
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 135, in wrapped
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception() as ectxt:
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 181, in wrapped
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1211, in create_network
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return self._after_create_network(context, result, mech_context)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1221, in _after_create_network
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1219, in _after_create_network
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.mechanism_manager.create_network_postcommit(mech_context)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 547, in create_network_postcommit
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self._call_on_drivers("create_network_postcommit", context)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 513, in _call_on_drivers
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     raise ml2_exc.MechanismDriverError(
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation neutron.plugins.ml2.common.exceptions.MechanismDriverError
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation
  2023-10-06 08:16:34.166 37713 DEBUG neutron.pecan_wsgi.hooks.notifier [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] No notification will be sent due to unsuccessful status code: 500 after /home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/hooks/notifier.py:78

  ^ From
  https://9ce43a75e3387ceb8909-2b4f2fa211fea8445ec0f4a568f6056b.ssl.cf2.rackcdn.com/897438/1/gate/neutron-
  functional-with-uwsgi/7147770/controller/logs/dsvm-functional-
  logs.tar.gz

  Other example failures:-
  - https://5a1da1856f577d094ffa-09419825a71508c658bc83d968b5572f.ssl.cf2.rackcdn.com/890162/3/check/neutron-functional-with-uwsgi/4ad32c0/testr_results.html
  - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d3b/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-uwsgi-fips/d3b6e71/testr_results.html
  - https://cf7800dce1b91328e4f8-616b34f6dd94893c803bfdcdd39959b5.ssl.cf5.rackcdn.com/897332/4/gate/neutron-functional-with-uwsgi/fe7d494/testr_results.html
  - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d3b/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-uwsgi-fips/d3b6e71/testr_results.html
  - https://cf7800dce1b91328e4f8-616b34f6dd94893c803bfdcdd39959b5.ssl.cf5.rackcdn.com/897332/4/gate/neutron-functional-with-uwsgi/fe7d494/testr_results.html

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2039417/+subscriptions
Follow ups

[Bug 2039417] Re: [master][stable/2023.2][functional] test_maintenance.TestMaintenance tests fails randomly
From: OpenStack Infra, 2023-10-26