yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2088423] Re: [functional test] test_ovn_nb_sync* tests fails randomly with Internal Server Error

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <2088423@xxxxxxxxxxxxxxxxxx>
Date: Sat, 23 Nov 2024 14:35:26 -0000
Reply-to: Bug 2088423 <2088423@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Reviewed:  https://review.opendev.org/c/openstack/neutron/+/935804
Committed: https://opendev.org/openstack/neutron/commit/abb527d1e4ec8ac34f6e277089b59687f9c3307e
Submitter: "Zuul (22348)"
Branch:    master

commit abb527d1e4ec8ac34f6e277089b59687f9c3307e
Author: Rodolfo Alonso Hernandez <ralonsoh@xxxxxxxxxx>
Date:   Wed Nov 20 15:20:16 2024 +0000

    [OVN] Use the MySQL backend for the ``TestOvnNbSync`` tests
    
    The ``TestOvnNbSync`` test cases perform intensive operations on both
    the Neutron database and the OVN databases. These test are frequently
    hitting an issue with the Neutron database, that in the functional test
    is, by default, SQLite.
    
    When a port is being deleted, the database raises an exception with the
    following message:
      DELETE failed.: oslo_db.exception.DBReferenceError:
      (sqlite3.IntegrityError) FOREIGN KEY constraint failed
      [SQL: DELETE FROM ports WHERE ports.id = ?]
      [parameters: ('64720ac5-72a0-4e88-8193-fd54a97ccef3',)]
    
    This resource (port) and the one referring to it (floating IP), have
    been created and updated in previous API calls, thus the transactions
    to the database should be commited and finished.
    
    This patch is changing the database backend to MySQL, that should
    provide better transaction isolation.
    
    Closes-Bug: #2088423
    Change-Id: If1da6c5992aa4635da5a4b5c6eaa06db56d693b4


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2088423

Title:
  [functional test] test_ovn_nb_sync* tests fails randomly with Internal
  Server Error

Status in neutron:
  Fix Released

Bug description:
  Fails like:-
  ft1.3: neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovn_db_sync.TestOvnNbSyncOverSsl.test_ovn_nb_sync_offtesttools.testresult.real._StringException: Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
      return f(self, *args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
      return f(self, *args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovn_db_sync.py", line 1609, in test_ovn_nb_sync_off
      self._test_ovn_nb_sync_helper('off', should_match_after_sync=False)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovn_db_sync.py", line 1578, in _test_ovn_nb_sync_helper
      self._create_resources(restart_ovsdb_processes)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovn_db_sync.py", line 654, in _create_resources
      self._delete('ports', n4_port_dict['p3'])
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/unit/db/test_db_base_plugin_v2.py", line 725, in _delete
      self.assertEqual(expected_code, res.status_int)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/testtools/testcase.py", line 419, in assertEqual
      self.assertThat(observed, matcher, message)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/testtools/testcase.py", line 509, in assertThat
      raise mismatch_error
  testtools.matchers._impl.MismatchError: 204 != 500

  neutron-server:-
  2024-10-31 17:26:09.361 50915 DEBUG neutron.plugins.ml2.plugin [None req-f47f60b2-2d64-4527-bb92-fe15be4792cc - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Calling delete_port for 64720ac5-72a0-4e88-8193-fd54a97ccef3 owned by compute:None delete_port /home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py:2242
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation [None req-f47f60b2-2d64-4527-bb92-fe15be4792cc - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] DELETE failed.: oslo_db.exception.DBReferenceError: (sqlite3.IntegrityError) FOREIGN KEY constraint failed
  [SQL: DELETE FROM ports WHERE ports.id = ?]
  [parameters: ('64720ac5-72a0-4e88-8193-fd54a97ccef3',)]
  (Background on this error at: https://sqlalche.me/e/20/gkpj)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.dialect.do_execute(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     cursor.execute(statement, parameters)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation sqlite3.IntegrityError: FOREIGN KEY constraint failed
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation The above exception was the direct cause of the following exception:
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/pecan/core.py", line 682, in __call__
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.invoke_controller(controller, args, kwargs, state)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/pecan/core.py", line 573, in invoke_controller
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     result = controller(*args, **kwargs)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 135, in wrapped
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception() as ectxt:
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 181, in wrapped
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/utils.py", line 65, in wrapped
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/utils.py", line 104, in wrapped
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     f(*args, **kwargs)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/resource.py", line 85, in delete
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return self.plugin_deleter(*deleter_args)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 723, in inner
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 223, in wrapped
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f_with_retry(*args, **kwargs,
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 135, in wrapped
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception() as ectxt:
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 181, in wrapped
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 2244, in delete_port
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     super().delete_port(context, id, port)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 221, in wrapped
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/db_base_plugin_v2.py", line 1603, in delete_port
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.ipam.delete_port(context, id, port)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/ipam_pluggable_backend.py", line 523, in delete_port
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self._ipam_deallocate_ips(context, ipam_driver, port,
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/ipam_pluggable_backend.py", line 93, in _ipam_deallocate_ips
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     with excutils.save_and_reraise_exception():
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/ipam_pluggable_backend.py", line 86, in _ipam_deallocate_ips
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     ipam_subnet = ipam_driver.get_subnet(ip['subnet_id'])
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 325, in get_subnet
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return NeutronDbSubnet.load(subnet_id, self._context)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 94, in load
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     ipam_subnet = ipam_db_api.IpamSubnetManager.load_by_neutron_subnet_id(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/ipam/drivers/neutrondb_ipam/db_api.py", line 29, in load_by_neutron_subnet_id
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     objs = ipam_objs.IpamSubnet.get_objects(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/objects/base.py", line 668, in get_objects
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     db_objs = obj_db_api.get_objects(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/objects/db/api.py", line 57, in get_objects
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return model_query.get_collection(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/model_query.py", line 373, in get_collection
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     items = [
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/model_query.py", line 373, in <listcomp>
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     items = [
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 2813, in __iter__
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     result = self._iter()
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 2827, in _iter
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     result: Union[ScalarResult[_T], Result[_T]] = self.session.execute(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2362, in execute
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return self._execute_internal(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2226, in _execute_internal
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     ) = compile_state_cls.orm_pre_session_exec(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/context.py", line 549, in orm_pre_session_exec
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     session._autoflush()
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3050, in _autoflush
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.flush()
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4352, in flush
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self._flush(objects)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4487, in _flush
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     with util.safe_reraise():
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise exc_value.with_traceback(exc_tb)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4448, in _flush
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     flush_context.execute()
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     rec.execute(self)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 679, in execute
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     util.preloaded.orm_persistence.delete_obj(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 193, in delete_obj
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     _emit_delete_statements(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 1465, in _emit_delete_statements
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     c = connection.execute(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1418, in execute
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return meth(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return connection._execute_clauseelement(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     ret = self._execute_context(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     return self._exec_single_context(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self._handle_dbapi_exception(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2352, in _handle_dbapi_exception
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     raise newraise.with_traceback(exc_info[2]) from e
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     self.dialect.do_execute(
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation     cursor.execute(statement, parameters)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation oslo_db.exception.DBReferenceError: (sqlite3.IntegrityError) FOREIGN KEY constraint failed
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation [SQL: DELETE FROM ports WHERE ports.id = ?]
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation [parameters: ('64720ac5-72a0-4e88-8193-fd54a97ccef3',)]
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation (Background on this error at: https://sqlalche.me/e/20/gkpj)
  2024-10-31 17:26:09.377 50915 ERROR neutron.pecan_wsgi.hooks.translation
  2024-10-31 17:26:09.382 50915 DEBUG neutron.pecan_wsgi.hooks.notifier [None req-f47f60b2-2d64-4527-bb92-fe15be4792cc - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] No notification will be sent due to unsuccessful status code: 500 after /home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/hooks/notifier.py:78
  2024-10-31 17:26:09.386 50915 DEBUG neutron.agent.linux.utils [None req-f47f60b2-2d64-4527-bb92-fe15be4792cc - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Running command: ['ovs-appctl', '-t', '/tmp/tmp04f99nnu/ovn_northd.ctl', 'exit'] create_process /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/utils.py:84
  2024-10-31 17:26:09.389 50915 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-a463b933-9dbc-4181-9bf1-2e384e87b79e - - - - - -] OVN-Southbound DB sync process completed @ 2024-10-31 17:26:09.389608 do_sync /home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1361

  Example failures:-
  https://efa50ad0d9c799d27a9a-0f5026351f42921faa2f428254ba2d4d.ssl.cf2.rackcdn.com/933458/1/gate/neutron-functional-with-uwsgi/7fe1d18/testr_results.html
  https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_929/929795/4/gate/neutron-functional-with-uwsgi/929cebc/testr_results.html
  https://a8cccf22d53a38c83185-06d350e249369bd6ea2abe2080d2269e.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-functional/fd69d0e/testr_results.html
  https://16a347a46b5f9e5955ab-7a435d8b6671949964eefa8586127100.ssl.cf2.rackcdn.com/933752/1/check/neutron-functional-with-uwsgi/8e65fdb/testr_results.html

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2088423/+subscriptions
References

[Bug 2088423] [NEW] [functional test] test_ovn_nb_sync* tests fails randomly with Internal Server Error
From: yatin, 2024-11-18