yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2039417] Re: [master][stable/2023.2][functional] test_maintenance.TestMaintenance tests fails randomly

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <2039417@xxxxxxxxxxxxxxxxxx>
Date: Thu, 26 Oct 2023 10:17:48 -0000
Reply-to: Bug 2039417 <2039417@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Reviewed:  https://review.opendev.org/c/openstack/neutron/+/899244
Committed: https://opendev.org/openstack/neutron/commit/39a1f1e7fd60b9b55dcb4b5ea5027e7759fc3b3b
Submitter: "Zuul (22348)"
Branch:    master

commit 39a1f1e7fd60b9b55dcb4b5ea5027e7759fc3b3b
Author: yatinkarel <ykarel@xxxxxxxxxx>
Date:   Wed Oct 25 15:10:25 2023 +0530

    [FT] Clear the idl lock in TestMaintenance tests
    
    One or the other test in TestMaintenance fails
    randomly with missing idl lock.
    
    DBInconsistenciesPeriodics set's the idl lock
    but in TestMaintenance we don't run it as a seperate
    maintenance worker and instead call check_for_inconsistencies
    when needed so not a requirement to have a dedicated lock.
    This patch clears the lock for these tests to avoid
    such random issues.
    
    Closes-Bug: #2039417
    Change-Id: I931997fe010295120f2400b1ec310dff82497ab6


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2039417

Title:
  [master][stable/2023.2][functional] test_maintenance.TestMaintenance
  tests fails randomly

Status in neutron:
  Fix Released

Bug description:
  The functional test fails randomly as:-
  ft1.3: neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_maintenance.TestMaintenance.test_porttesttools.testresult.real._StringException: Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
      return f(self, *args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_maintenance.py", line 306, in test_port
      neutron_net = self._create_network('network1')
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_maintenance.py", line 68, in _create_network
      return self.deserialize(self.fmt, res)['network']
  KeyError: 'network'

  neutron server log when it fails:-
  2023-10-06 08:16:33.966 37713 DEBUG neutron.db.ovn_revision_numbers_db [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] create_initial_revision uuid=6ab7bd53-277b-4133-ac32-52b1d0c90f78, type=security_groups, rev=-1 create_initial_revision /home/zuul/src/opendev.org/openstack/neutron/neutron/db/ovn_revision_numbers_db.py:108
  2023-10-06 08:16:33.973 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
  2023-10-06 08:16:33.974 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
      txn.results.put(txn.do_commit())
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
      raise RuntimeError(msg)
  RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it

  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver._create_security_group-11049860 security_group, after_create: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/callbacks/manager.py", line 181, in _notify_loop
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     callback(resource, event, trigger, payload=payload)
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 409, in _create_security_group
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     self._ovn_client.create_security_group(context,
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2328, in create_security_group
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     with self._nb_idl.transaction(check_error=True) as txn:
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     next(self.gen)
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     with super(OvsdbNbOvnIdl, self).transaction(*args, **kwargs) as t:
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     next(self.gen)
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 114, in transaction
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     with self.create_transaction(
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 71, in __exit__
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     self.result = self.commit()
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     raise result.ex
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     txn.results.put(txn.do_commit())
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager     raise RuntimeError(msg)
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
  2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager
  ...
  2023-10-06 08:16:34.047 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
  2023-10-06 08:16:34.048 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
      txn.results.put(txn.do_commit())
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
      raise RuntimeError(msg)
  RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it

  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Mechanism driver 'ovn' failed in create_network_postcommit: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 497, in _call_on_drivers
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 579, in create_network_postcommit
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     self._ovn_client.create_network(context.plugin_context, network)
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1927, in create_network
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     with self._nb_idl.transaction(check_error=True) as txn:
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     next(self.gen)
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     with super(OvsdbNbOvnIdl, self).transaction(*args, **kwargs) as t:
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     next(self.gen)
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 114, in transaction
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     with self.create_transaction(
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 71, in __exit__
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     self.result = self.commit()
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     raise result.ex
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     txn.results.put(txn.do_commit())
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers     raise RuntimeError(msg)
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
  2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers 
  2023-10-06 08:16:34.052 37713 ERROR neutron.plugins.ml2.plugin [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] mechanism_manager.create_network_postcommit failed, deleting network '767ee503-3645-4762-b0d6-e673b3c14170': neutron.plugins.ml2.common.exceptions.MechanismDriverError
  ...
  2023-10-06 08:16:34.153 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
  2023-10-06 08:16:34.154 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
      txn.results.put(txn.do_commit())
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
      raise RuntimeError(msg)
  RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
  : neutron.plugins.ml2.common.exceptions.MechanismDriverError
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Mechanism driver 'ovn' failed in delete_network_postcommit: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1219, in _after_create_network
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     self.mechanism_manager.create_network_postcommit(mech_context)
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 547, in create_network_postcommit
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     self._call_on_drivers("create_network_postcommit", context)
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 513, in _call_on_drivers
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     raise ml2_exc.MechanismDriverError(
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers neutron.plugins.ml2.common.exceptions.MechanismDriverError
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers During handling of the above exception, another exception occurred:
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 497, in _call_on_drivers
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 631, in delete_network_postcommit
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     self._ovn_client.delete_network(
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1938, in delete_network
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     with self._nb_idl.transaction(check_error=True) as txn:
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     next(self.gen)
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     with super(OvsdbNbOvnIdl, self).transaction(*args, **kwargs) as t:
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     next(self.gen)
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 114, in transaction
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     with self.create_transaction(
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 71, in __exit__
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     self.result = self.commit()
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     raise result.ex
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     txn.results.put(txn.do_commit())
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers     raise RuntimeError(msg)
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
  2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers
  2023-10-06 08:16:34.162 37713 ERROR neutron.plugins.ml2.plugin [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] mechanism_manager.delete_network_postcommit failed: neutron.plugins.ml2.common.exceptions.MechanismDriverError
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] POST failed.: neutron.plugins.ml2.common.exceptions.MechanismDriverError
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/pecan/core.py", line 693, in __call__
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.invoke_controller(controller, args, kwargs, state)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/pecan/core.py", line 584, in invoke_controller
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     result = controller(*args, **kwargs)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 135, in wrapped
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception() as ectxt:
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 181, in wrapped
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/utils.py", line 65, in wrapped
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/resource.py", line 163, in post
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return self.create(resources)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/resource.py", line 181, in create
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return {key: creator(*creator_args, **creator_kwargs)}
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 731, in inner
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 223, in wrapped
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f_with_retry(*args, **kwargs,
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 135, in wrapped
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception() as ectxt:
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 181, in wrapped
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1211, in create_network
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     return self._after_create_network(context, result, mech_context)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1221, in _after_create_network
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1219, in _after_create_network
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self.mechanism_manager.create_network_postcommit(mech_context)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 547, in create_network_postcommit
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     self._call_on_drivers("create_network_postcommit", context)
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 513, in _call_on_drivers
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation     raise ml2_exc.MechanismDriverError(
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation neutron.plugins.ml2.common.exceptions.MechanismDriverError
  2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation
  2023-10-06 08:16:34.166 37713 DEBUG neutron.pecan_wsgi.hooks.notifier [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] No notification will be sent due to unsuccessful status code: 500 after /home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/hooks/notifier.py:78

  ^ From
  https://9ce43a75e3387ceb8909-2b4f2fa211fea8445ec0f4a568f6056b.ssl.cf2.rackcdn.com/897438/1/gate/neutron-
  functional-with-uwsgi/7147770/controller/logs/dsvm-functional-
  logs.tar.gz

  Other example failures:-
  - https://5a1da1856f577d094ffa-09419825a71508c658bc83d968b5572f.ssl.cf2.rackcdn.com/890162/3/check/neutron-functional-with-uwsgi/4ad32c0/testr_results.html
  - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d3b/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-uwsgi-fips/d3b6e71/testr_results.html
  - https://cf7800dce1b91328e4f8-616b34f6dd94893c803bfdcdd39959b5.ssl.cf5.rackcdn.com/897332/4/gate/neutron-functional-with-uwsgi/fe7d494/testr_results.html
  - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d3b/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-uwsgi-fips/d3b6e71/testr_results.html
  - https://cf7800dce1b91328e4f8-616b34f6dd94893c803bfdcdd39959b5.ssl.cf5.rackcdn.com/897332/4/gate/neutron-functional-with-uwsgi/fe7d494/testr_results.html

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2039417/+subscriptions
References

[Bug 2039417] [NEW] [master][stable/2023.2][functional] test_maintenance.TestMaintenance tests fails randomly
From: yatin, 2023-10-16