yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2088423] [NEW] [functional test] test_ovn_nb_sync* tests fails randomly with Internal Server Error

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: yatin <2088423@xxxxxxxxxxxxxxxxxx>
Date: Mon, 18 Nov 2024 07:00:04 -0000
Reply-to: Bug 2088423 <2088423@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Fails like:-
ft1.3: neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovn_db_sync.TestOvnNbSyncOverSsl.test_ovn_nb_sync_offtesttools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovn_db_sync.py", line 1609, in test_ovn_nb_sync_off
    self._test_ovn_nb_sync_helper('off', should_match_after_sync=False)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovn_db_sync.py", line 1578, in _test_ovn_nb_sync_helper
    self._create_resources(restart_ovsdb_processes)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovn_db_sync.py", line 654, in _create_resources
    self._delete('ports', n4_port_dict['p3'])
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/unit/db/test_db_base_plugin_v2.py", line 725, in _delete
    self.assertEqual(expected_code, res.status_int)
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/testtools/testcase.py", line 419, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/testtools/testcase.py", line 509, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 204 != 500

neutron-server:-
2024-10-31 17:26:09.361 50915 DEBUG neutron.plugins.ml2.plugin [None req-f47f60b2-2d64-4527-bb92-fe15be4792cc - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Calling delete_port for 64720ac5-72a0-4e88-8193-fd54a97ccef3 owned by compute:None delete_port /home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py:2242
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation [None req-f47f60b2-2d64-4527-bb92-fe15be4792cc - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] DELETE failed.: oslo_db.exception.DBReferenceError: (sqlite3.IntegrityError) FOREIGN KEY constraint failed
[SQL: DELETE FROM ports WHERE ports.id = ?]
[parameters: ('64720ac5-72a0-4e88-8193-fd54a97ccef3',)]
(Background on this error at: https://sqlalche.me/e/20/gkpj)
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.dialect.do_execute(
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     cursor.execute(statement, parameters)
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation sqlite3.IntegrityError: FOREIGN KEY constraint failed
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation The above exception was the direct cause of the following exception:
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/pecan/core.py", line 682, in __call__
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.invoke_controller(controller, args, kwargs, state)
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/pecan/core.py", line 573, in invoke_controller
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     result = controller(*args, **kwargs)
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception() as ectxt:
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 181, in wrapped
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/utils.py", line 65, in wrapped
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/utils.py", line 104, in wrapped
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     f(*args, **kwargs)
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/resource.py", line 85, in delete
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return self.plugin_deleter(*deleter_args)
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 723, in inner
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 223, in wrapped
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f_with_retry(*args, **kwargs,
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception() as ectxt:
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 181, in wrapped
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 2244, in delete_port
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     super().delete_port(context, id, port)
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 221, in wrapped
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/db_base_plugin_v2.py", line 1603, in delete_port
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.ipam.delete_port(context, id, port)
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/ipam_pluggable_backend.py", line 523, in delete_port
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self._ipam_deallocate_ips(context, ipam_driver, port,
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/ipam_pluggable_backend.py", line 93, in _ipam_deallocate_ips
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/ipam_pluggable_backend.py", line 86, in _ipam_deallocate_ips
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     ipam_subnet = ipam_driver.get_subnet(ip['subnet_id'])
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 325, in get_subnet
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return NeutronDbSubnet.load(subnet_id, self._context)
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 94, in load
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     ipam_subnet = ipam_db_api.IpamSubnetManager.load_by_neutron_subnet_id(
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/ipam/drivers/neutrondb_ipam/db_api.py", line 29, in load_by_neutron_subnet_id
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     objs = ipam_objs.IpamSubnet.get_objects(
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/objects/base.py", line 668, in get_objects
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     db_objs = obj_db_api.get_objects(
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/objects/db/api.py", line 57, in get_objects
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return model_query.get_collection(
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/model_query.py", line 373, in get_collection
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     items = [
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/model_query.py", line 373, in <listcomp>
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     items = [
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 2813, in __iter__
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     result = self._iter()
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 2827, in _iter
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     result: Union[ScalarResult[_T], Result[_T]] = self.session.execute(
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2362, in execute
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return self._execute_internal(
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2226, in _execute_internal
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     ) = compile_state_cls.orm_pre_session_exec(
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/context.py", line 549, in orm_pre_session_exec
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     session._autoflush()
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3050, in _autoflush
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.flush()
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4352, in flush
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self._flush(objects)
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4487, in _flush
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     with util.safe_reraise():
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise exc_value.with_traceback(exc_tb)
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4448, in _flush
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     flush_context.execute()
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     rec.execute(self)
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 679, in execute
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     util.preloaded.orm_persistence.delete_obj(
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 193, in delete_obj
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     _emit_delete_statements(
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 1465, in _emit_delete_statements
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     c = connection.execute(
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1418, in execute
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return meth(
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return connection._execute_clauseelement(
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     ret = self._execute_context(
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return self._exec_single_context(
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self._handle_dbapi_exception(
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2352, in _handle_dbapi_exception
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise newraise.with_traceback(exc_info[2]) from e
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.dialect.do_execute(
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     cursor.execute(statement, parameters)
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation oslo_db.exception.DBReferenceError: (sqlite3.IntegrityError) FOREIGN KEY constraint failed
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation [SQL: DELETE FROM ports WHERE ports.id = ?]
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation [parameters: ('64720ac5-72a0-4e88-8193-fd54a97ccef3',)]
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation (Background on this error at: https://sqlalche.me/e/20/gkpj)
2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation
2024-10-31 17:26:09.382 50915 DEBUG neutron.pecan_wsgi.hooks.notifier [None req-f47f60b2-2d64-4527-bb92-fe15be4792cc - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] No notification will be sent due to unsuccessful status code: 500 after /home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/hooks/notifier.py:78
2024-10-31 17:26:09.386 50915 DEBUG neutron.agent.linux.utils [None req-f47f60b2-2d64-4527-bb92-fe15be4792cc - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Running command: ['ovs-appctl', '-t', '/tmp/tmp04f99nnu/ovn_northd.ctl', 'exit'] create_process /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/utils.py:84
2024-10-31 17:26:09.389 50915 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-a463b933-9dbc-4181-9bf1-2e384e87b79e - - - - - -] OVN-Southbound DB sync process completed @ 2024-10-31 17:26:09.389608 do_sync /home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1361

Example failures:-
https://efa50ad0d9c799d27a9a-0f5026351f42921faa2f428254ba2d4d.ssl.cf2.rackcdn.com/933458/1/gate/neutron-functional-with-uwsgi/7fe1d18/testr_results.html
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_929/929795/4/gate/neutron-functional-with-uwsgi/929cebc/testr_results.html
https://a8cccf22d53a38c83185-06d350e249369bd6ea2abe2080d2269e.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-functional/fd69d0e/testr_results.html
https://16a347a46b5f9e5955ab-7a435d8b6671949964eefa8586127100.ssl.cf2.rackcdn.com/933752/1/check/neutron-functional-with-uwsgi/8e65fdb/testr_results.html

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: functional-tests gate-failure

** Tags added: functional-tests gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2088423

Title:
  [functional test] test_ovn_nb_sync* tests fails randomly with Internal
  Server Error

Status in neutron:
  New

Bug description:
  Fails like:-
  ft1.3: neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovn_db_sync.TestOvnNbSyncOverSsl.test_ovn_nb_sync_offtesttools.testresult.real._StringException: Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
      return f(self, *args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
      return f(self, *args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovn_db_sync.py", line 1609, in test_ovn_nb_sync_off
      self._test_ovn_nb_sync_helper('off', should_match_after_sync=False)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovn_db_sync.py", line 1578, in _test_ovn_nb_sync_helper
      self._create_resources(restart_ovsdb_processes)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovn_db_sync.py", line 654, in _create_resources
      self._delete('ports', n4_port_dict['p3'])
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/unit/db/test_db_base_plugin_v2.py", line 725, in _delete
      self.assertEqual(expected_code, res.status_int)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/testtools/testcase.py", line 419, in assertEqual
      self.assertThat(observed, matcher, message)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/testtools/testcase.py", line 509, in assertThat
      raise mismatch_error
  testtools.matchers._impl.MismatchError: 204 != 500

  neutron-server:-
  2024-10-31 17:26:09.361 50915 DEBUG neutron.plugins.ml2.plugin [None req-f47f60b2-2d64-4527-bb92-fe15be4792cc - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Calling delete_port for 64720ac5-72a0-4e88-8193-fd54a97ccef3 owned by compute:None delete_port /home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py:2242
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation [None req-f47f60b2-2d64-4527-bb92-fe15be4792cc - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] DELETE failed.: oslo_db.exception.DBReferenceError: (sqlite3.IntegrityError) FOREIGN KEY constraint failed
  [SQL: DELETE FROM ports WHERE ports.id = ?]
  [parameters: ('64720ac5-72a0-4e88-8193-fd54a97ccef3',)]
  (Background on this error at: https://sqlalche.me/e/20/gkpj)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.dialect.do_execute(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     cursor.execute(statement, parameters)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation sqlite3.IntegrityError: FOREIGN KEY constraint failed
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation The above exception was the direct cause of the following exception:
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/pecan/core.py", line 682, in __call__
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.invoke_controller(controller, args, kwargs, state)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/pecan/core.py", line 573, in invoke_controller
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     result = controller(*args, **kwargs)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 135, in wrapped
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception() as ectxt:
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 181, in wrapped
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/utils.py", line 65, in wrapped
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/utils.py", line 104, in wrapped
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     f(*args, **kwargs)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/resource.py", line 85, in delete
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return self.plugin_deleter(*deleter_args)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 723, in inner
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 223, in wrapped
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f_with_retry(*args, **kwargs,
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 135, in wrapped
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception() as ectxt:
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 181, in wrapped
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 2244, in delete_port
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     super().delete_port(context, id, port)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 221, in wrapped
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/db_base_plugin_v2.py", line 1603, in delete_port
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.ipam.delete_port(context, id, port)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/ipam_pluggable_backend.py", line 523, in delete_port
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self._ipam_deallocate_ips(context, ipam_driver, port,
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/ipam_pluggable_backend.py", line 93, in _ipam_deallocate_ips
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/ipam_pluggable_backend.py", line 86, in _ipam_deallocate_ips
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     ipam_subnet = ipam_driver.get_subnet(ip['subnet_id'])
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 325, in get_subnet
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return NeutronDbSubnet.load(subnet_id, self._context)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 94, in load
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     ipam_subnet = ipam_db_api.IpamSubnetManager.load_by_neutron_subnet_id(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/ipam/drivers/neutrondb_ipam/db_api.py", line 29, in load_by_neutron_subnet_id
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     objs = ipam_objs.IpamSubnet.get_objects(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/objects/base.py", line 668, in get_objects
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     db_objs = obj_db_api.get_objects(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/objects/db/api.py", line 57, in get_objects
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return model_query.get_collection(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/model_query.py", line 373, in get_collection
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     items = [
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/model_query.py", line 373, in <listcomp>
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     items = [
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 2813, in __iter__
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     result = self._iter()
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 2827, in _iter
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     result: Union[ScalarResult[_T], Result[_T]] = self.session.execute(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2362, in execute
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return self._execute_internal(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2226, in _execute_internal
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     ) = compile_state_cls.orm_pre_session_exec(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/context.py", line 549, in orm_pre_session_exec
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     session._autoflush()
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3050, in _autoflush
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.flush()
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4352, in flush
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self._flush(objects)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4487, in _flush
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     with util.safe_reraise():
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise exc_value.with_traceback(exc_tb)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4448, in _flush
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     flush_context.execute()
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     rec.execute(self)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 679, in execute
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     util.preloaded.orm_persistence.delete_obj(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 193, in delete_obj
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     _emit_delete_statements(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 1465, in _emit_delete_statements
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     c = connection.execute(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1418, in execute
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return meth(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return connection._execute_clauseelement(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     ret = self._execute_context(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return self._exec_single_context(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self._handle_dbapi_exception(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2352, in _handle_dbapi_exception
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise newraise.with_traceback(exc_info[2]) from e
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.dialect.do_execute(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     cursor.execute(statement, parameters)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation oslo_db.exception.DBReferenceError: (sqlite3.IntegrityError) FOREIGN KEY constraint failed
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation [SQL: DELETE FROM ports WHERE ports.id = ?]
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation [parameters: ('64720ac5-72a0-4e88-8193-fd54a97ccef3',)]
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation (Background on this error at: https://sqlalche.me/e/20/gkpj)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation
  2024-10-31 17:26:09.382 50915 DEBUG neutron.pecan_wsgi.hooks.notifier [None req-f47f60b2-2d64-4527-bb92-fe15be4792cc - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] No notification will be sent due to unsuccessful status code: 500 after /home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/hooks/notifier.py:78
  2024-10-31 17:26:09.386 50915 DEBUG neutron.agent.linux.utils [None req-f47f60b2-2d64-4527-bb92-fe15be4792cc - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Running command: ['ovs-appctl', '-t', '/tmp/tmp04f99nnu/ovn_northd.ctl', 'exit'] create_process /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/utils.py:84
  2024-10-31 17:26:09.389 50915 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-a463b933-9dbc-4181-9bf1-2e384e87b79e - - - - - -] OVN-Southbound DB sync process completed @ 2024-10-31 17:26:09.389608 do_sync /home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1361

  Example failures:-
  https://efa50ad0d9c799d27a9a-0f5026351f42921faa2f428254ba2d4d.ssl.cf2.rackcdn.com/933458/1/gate/neutron-functional-with-uwsgi/7fe1d18/testr_results.html
  https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_929/929795/4/gate/neutron-functional-with-uwsgi/929cebc/testr_results.html
  https://a8cccf22d53a38c83185-06d350e249369bd6ea2abe2080d2269e.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-functional/fd69d0e/testr_results.html
  https://16a347a46b5f9e5955ab-7a435d8b6671949964eefa8586127100.ssl.cf2.rackcdn.com/933752/1/check/neutron-functional-with-uwsgi/8e65fdb/testr_results.html

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2088423/+subscriptions
Follow ups

[Bug 2088423] Re: [functional test] test_ovn_nb_sync* tests fails randomly with Internal Server Error
From: OpenStack Infra, 2024-11-23