yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2069442] [NEW] Deleting a VM may produce a traceback because of a race

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Jakub Libosvar <2069442@xxxxxxxxxxxxxxxxxx>
Date: Fri, 14 Jun 2024 16:56:13 -0000
Reply-to: Bug 2069442 <2069442@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

When a VM port is deleted, it is moved to down state in OVN first and
then deleted by Neutron. The down state emits an event in Neutron and if
the port was deleted before Neutron got to setting its state to down, it
produces a traceback.

2024-06-04 17:18:03.563 44 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [req-939ad74c-8400-412f-b537-5e0250c001de - - - - -] OVN reports status down for port: 6d26299c-dab5-4b8d-bba4-37665f4dfcdd
2024-06-04 17:18:03.858 44 INFO neutron.notifiers.nova [-] Nova event matching ['req-6b15230f-3d03-4e94-bcbd-b87c3dd526fc'] response: {'server_uuid': '0043ea52-85ca-40bf-9242-1507159547a9', 'name': 'network-vif-unplugged', 'status': 'completed', 'tag': '6d26299c-dab5-4b8d-bba4-37665
f4dfcdd', 'code': 200}
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers [req-939ad74c-8400-412f-b537-5e0250c001de - - - - -] Mechanism driver 'ovn' failed in update_port_postcommit: oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a fo
reign key constraint fails (`ovs_neutron`.`ovn_revision_numbers`, CONSTRAINT `ovn_revision_numbers_ibfk_1` FOREIGN KEY (`standard_attr_id`) REFERENCES `standardattributes` (`id`) ON DELETE SET NULL)')
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     self.dialect.do_execute(
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     cursor.execute(statement, parameters)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     result = self._query(query)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     conn.query(q)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     result.read()
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     first_packet = self.connection._read_packet()
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     packet.raise_for_error()
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     err.raise_mysql_exception(self._data)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     raise errorclass(errno, errval)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers pymysql.err.IntegrityError: (1452, 'Cannot add or update a child row: a foreign key constraint fails (`ovs_neutron`.`ovn_revision_numbers`, CONSTRAINT `ovn_revision_numbers_ibfk_1` FOREIGN KEY (`standard_attr_id`) REFEREN
CES `standardattributes` (`id`) ON DELETE SET NULL)')
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers The above exception was the direct cause of the following exception:
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/managers.py", line 493, in _call_on_drivers
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 869, in update_port_postcommit
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     self._ovn_update_port(context._plugin_context, port, original_port,
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 754, in _ovn_update_port
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     self._ovn_client.update_port(plugin_context, port,
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 833, in update_port
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     db_rev.bump_revision(context, port, ovn_const.TYPE_PORTS)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 217, in wrapped
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     return method(*args, **kwargs)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     setattr(e, '_RETRY_EXCEEDED', True)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     self.force_reraise()
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     raise self.value
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     return f(*args, **kwargs)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     ectxt.value = e.inner_exc
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     self.force_reraise()
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     raise self.value
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     return f(*args, **kwargs)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     LOG.debug("Retry wrapper got retriable exception: %s", e)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     self.force_reraise()
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     raise self.value
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     return f(*dup_args, **dup_kwargs)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron/db/ovn_revision_numbers_db.py", line 190, in bump_revision
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     context.session.merge(row)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2160, in merge
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     self._autoflush()
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1622, in _autoflush
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     self.flush()
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2540, in flush
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     self._flush(objects)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2682, in _flush
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     transaction.rollback(_capture_exception=True)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     compat.raise_(
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     raise exception
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2642, in _flush
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     flush_context.execute()
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     rec.execute(self)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 586, in execute
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     persistence.save_obj(
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 230, in save_obj
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     _emit_update_statements(
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 994, in _emit_update_statements
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     c = cached_connections[connection].execute(
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     return meth(self, multiparams, params)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     return connection._execute_clauseelement(self, multiparams, params)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     ret = self._execute_context(
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     self._handle_dbapi_exception(
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1508, in _handle_dbapi_exception
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     util.raise_(newraise, with_traceback=exc_info[2], from_=e)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     raise exception
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     self.dialect.do_execute(
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     cursor.execute(statement, parameters)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     result = self._query(query)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     conn.query(q)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     result.read()
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     first_packet = self.connection._read_packet()
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     packet.raise_for_error()
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     err.raise_mysql_exception(self._data)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     raise errorclass(errno, errval)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`ovs_neutron`.`ovn_revision_numbers`, CONSTRAINT `ovn_revision_numbers_ibfk_1` FOREIGN KEY (`standard_attr_id`) REFERENCES `standardattributes` (`id`) ON DELETE SET NULL)')
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers [SQL: UPDATE ovn_revision_numbers SET standard_attr_id=%(standard_attr_id)s, revision_number=%(revision_number)s, updated_at=now() WHERE ovn_revision_numbers.resource_uuid = %(ovn_revision_numbers_resource_uuid)s AND ovn_revision_numbers.resource_type = %(ovn_revision_numbers_resource_type)s]
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers [parameters: {'standard_attr_id': 298849487, 'revision_number': 5, 'ovn_revision_numbers_resource_uuid': '6d26299c-dab5-4b8d-bba4-37665f4dfcdd', 'ovn_revision_numbers_resource_type': 'ports'}]
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers (Background on this error at: http://sqlalche.me/e/13/gkpj)
2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers


We should catch the exception but we need to do it in the mech driver itself and not the event, since ML2 plugin mechanism driver manager swallows exceptions.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2069442

Title:
  Deleting a VM may produce a traceback because of a race

Status in neutron:
  New

Bug description:
  When a VM port is deleted, it is moved to down state in OVN first and
  then deleted by Neutron. The down state emits an event in Neutron and
  if the port was deleted before Neutron got to setting its state to
  down, it produces a traceback.

  2024-06-04 17:18:03.563 44 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [req-939ad74c-8400-412f-b537-5e0250c001de - - - - -] OVN reports status down for port: 6d26299c-dab5-4b8d-bba4-37665f4dfcdd
  2024-06-04 17:18:03.858 44 INFO neutron.notifiers.nova [-] Nova event matching ['req-6b15230f-3d03-4e94-bcbd-b87c3dd526fc'] response: {'server_uuid': '0043ea52-85ca-40bf-9242-1507159547a9', 'name': 'network-vif-unplugged', 'status': 'completed', 'tag': '6d26299c-dab5-4b8d-bba4-37665
  f4dfcdd', 'code': 200}
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers [req-939ad74c-8400-412f-b537-5e0250c001de - - - - -] Mechanism driver 'ovn' failed in update_port_postcommit: oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a fo
  reign key constraint fails (`ovs_neutron`.`ovn_revision_numbers`, CONSTRAINT `ovn_revision_numbers_ibfk_1` FOREIGN KEY (`standard_attr_id`) REFERENCES `standardattributes` (`id`) ON DELETE SET NULL)')
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     self.dialect.do_execute(
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     cursor.execute(statement, parameters)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     result = self._query(query)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     conn.query(q)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     result.read()
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     first_packet = self.connection._read_packet()
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     packet.raise_for_error()
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     err.raise_mysql_exception(self._data)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     raise errorclass(errno, errval)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers pymysql.err.IntegrityError: (1452, 'Cannot add or update a child row: a foreign key constraint fails (`ovs_neutron`.`ovn_revision_numbers`, CONSTRAINT `ovn_revision_numbers_ibfk_1` FOREIGN KEY (`standard_attr_id`) REFEREN
  CES `standardattributes` (`id`) ON DELETE SET NULL)')
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers The above exception was the direct cause of the following exception:
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/managers.py", line 493, in _call_on_drivers
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 869, in update_port_postcommit
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     self._ovn_update_port(context._plugin_context, port, original_port,
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 754, in _ovn_update_port
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     self._ovn_client.update_port(plugin_context, port,
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 833, in update_port
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     db_rev.bump_revision(context, port, ovn_const.TYPE_PORTS)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 217, in wrapped
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     return method(*args, **kwargs)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     setattr(e, '_RETRY_EXCEEDED', True)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     self.force_reraise()
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     raise self.value
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     return f(*args, **kwargs)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     ectxt.value = e.inner_exc
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     self.force_reraise()
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     raise self.value
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     return f(*args, **kwargs)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     LOG.debug("Retry wrapper got retriable exception: %s", e)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     self.force_reraise()
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     raise self.value
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     return f(*dup_args, **dup_kwargs)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron/db/ovn_revision_numbers_db.py", line 190, in bump_revision
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     context.session.merge(row)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2160, in merge
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     self._autoflush()
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1622, in _autoflush
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     self.flush()
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2540, in flush
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     self._flush(objects)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2682, in _flush
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     transaction.rollback(_capture_exception=True)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     compat.raise_(
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     raise exception
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2642, in _flush
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     flush_context.execute()
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     rec.execute(self)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 586, in execute
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     persistence.save_obj(
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 230, in save_obj
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     _emit_update_statements(
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 994, in _emit_update_statements
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     c = cached_connections[connection].execute(
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     return meth(self, multiparams, params)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     return connection._execute_clauseelement(self, multiparams, params)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     ret = self._execute_context(
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     self._handle_dbapi_exception(
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1508, in _handle_dbapi_exception
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     util.raise_(newraise, with_traceback=exc_info[2], from_=e)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     raise exception
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     self.dialect.do_execute(
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     cursor.execute(statement, parameters)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     result = self._query(query)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     conn.query(q)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     result.read()
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     first_packet = self.connection._read_packet()
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     packet.raise_for_error()
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     err.raise_mysql_exception(self._data)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers     raise errorclass(errno, errval)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`ovs_neutron`.`ovn_revision_numbers`, CONSTRAINT `ovn_revision_numbers_ibfk_1` FOREIGN KEY (`standard_attr_id`) REFERENCES `standardattributes` (`id`) ON DELETE SET NULL)')
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers [SQL: UPDATE ovn_revision_numbers SET standard_attr_id=%(standard_attr_id)s, revision_number=%(revision_number)s, updated_at=now() WHERE ovn_revision_numbers.resource_uuid = %(ovn_revision_numbers_resource_uuid)s AND ovn_revision_numbers.resource_type = %(ovn_revision_numbers_resource_type)s]
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers [parameters: {'standard_attr_id': 298849487, 'revision_number': 5, 'ovn_revision_numbers_resource_uuid': '6d26299c-dab5-4b8d-bba4-37665f4dfcdd', 'ovn_revision_numbers_resource_type': 'ports'}]
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers (Background on this error at: http://sqlalche.me/e/13/gkpj)
  2024-06-04 17:18:04.218 44 ERROR neutron.plugins.ml2.managers

  
  We should catch the exception but we need to do it in the mech driver itself and not the event, since ML2 plugin mechanism driver manager swallows exceptions.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2069442/+subscriptions
Follow ups

[Bug 2069442] Re: Deleting a VM may produce a traceback because of a race
From: OpenStack Infra, 2024-07-16