← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #95683

[Bug 2106374] [NEW] neutron-server error with ml2_distributed_port_bindings

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Under some conditions, Neutron tries to create a record in the Database
a few times and fails with an unexpected exception that should be
handled by existing code.

Neutron mode: ovs+dvr

How to reproduce:
1. Tenant network connected to distributed router
2. VM with tenant network created or migrated to an empty hypervisor

When VM is placed to empty node L3-agent notifies neutron-server to bind router distributed port to node and we get ERROR traceback in neutron-server log:
```
ERROR oslo_messaging.rpc.server [parameters: {'port_id': 'f269cf3d-8a00-4b5b-99c2-63cae0968f42', 'host': 'gm-os-hv-017', 'router_id': 'e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}]
ERROR oslo_messaging.rpc.server [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]
ERROR oslo_messaging.rpc.server sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'f269cf3d-8a00-4b5b-99c2-63cae0968f42-gm-os-hv-017' for key 'PRIMARY'")
ERROR oslo_messaging.rpc.server     raise sa_exc.PendingRollbackError(
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active
ERROR oslo_messaging.rpc.server     self._assert_active()
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind
ERROR oslo_messaging.rpc.server     return self._transaction._connection_for_bind(
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind
ERROR oslo_messaging.rpc.server     conn = self._connection_for_bind(bind, close_with_result=True)
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute
ERROR oslo_messaging.rpc.server     result = self.session.execute(
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter
ERROR oslo_messaging.rpc.server     return self._iter().one()
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one
ERROR oslo_messaging.rpc.server     return (context.session.query(models.DistributedPortBinding).
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding
ERROR oslo_messaging.rpc.server     binding = db.ensure_distributed_port_binding(
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding
ERROR oslo_messaging.rpc.server     return f(*dup_args, **dup_kwargs)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped
ERROR oslo_messaging.rpc.server     raise self.value
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
ERROR oslo_messaging.rpc.server     self.force_reraise()
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server     LOG.debug("Retry wrapper got retriable exception: %s", e)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped
ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper
ERROR oslo_messaging.rpc.server     raise self.value
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
ERROR oslo_messaging.rpc.server     self.force_reraise()
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server     ectxt.value = e.inner_exc
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper
ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped
ERROR oslo_messaging.rpc.server     raise self.value
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
ERROR oslo_messaging.rpc.server     self.force_reraise()
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server     setattr(e, '_RETRY_EXCEEDED', True)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped
ERROR oslo_messaging.rpc.server     return method(*args, **kwargs)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped
ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner
ERROR oslo_messaging.rpc.server     self.plugin.update_distributed_port_binding(
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port
ERROR oslo_messaging.rpc.server     self._ensure_host_set_on_port(
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports
ERROR oslo_messaging.rpc.server     self._ensure_host_set_on_ports(context, host, routers)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers
ERROR oslo_messaging.rpc.server     return f(*dup_args, **dup_kwargs)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped
ERROR oslo_messaging.rpc.server     raise self.value
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
ERROR oslo_messaging.rpc.server     self.force_reraise()
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server     LOG.debug("Retry wrapper got retriable exception: %s", e)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped
ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper
ERROR oslo_messaging.rpc.server     raise self.value
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
ERROR oslo_messaging.rpc.server     self.force_reraise()
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server     ectxt.value = e.inner_exc
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper
ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped
ERROR oslo_messaging.rpc.server     raise self.value
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
ERROR oslo_messaging.rpc.server     self.force_reraise()
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server     setattr(e, '_RETRY_EXCEEDED', True)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped
ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
ERROR oslo_messaging.rpc.server
ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
ERROR oslo_messaging.rpc.server
ERROR oslo_messaging.rpc.server (Background on this error at: https://sqlalche.me/e/14/gkpj)
ERROR oslo_messaging.rpc.server [parameters: {'port_id': 'f269cf3d-8a00-4b5b-99c2-63cae0968f42', 'host': 'gm-os-hv-017', 'router_id': 'e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}]
ERROR oslo_messaging.rpc.server [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]
ERROR oslo_messaging.rpc.server oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'f269cf3d-8a00-4b5b-99c2-63cae0968f42-gm-os-hv-017' for key 'PRIMARY'")
ERROR oslo_messaging.rpc.server     raise errorclass(errno, errval)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
ERROR oslo_messaging.rpc.server     err.raise_mysql_exception(self._data)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error
ERROR oslo_messaging.rpc.server     packet.raise_for_error()
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet
ERROR oslo_messaging.rpc.server     first_packet = self.connection._read_packet()
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read
ERROR oslo_messaging.rpc.server     result.read()
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result
ERROR oslo_messaging.rpc.server     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query
ERROR oslo_messaging.rpc.server     conn.query(q)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query
ERROR oslo_messaging.rpc.server     result = self._query(query)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute
ERROR oslo_messaging.rpc.server     cursor.execute(statement, parameters)
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
ERROR oslo_messaging.rpc.server     self.dialect.do_execute(
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
ERROR oslo_messaging.rpc.server     raise exception
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
ERROR oslo_messaging.rpc.server     util.raise_(newraise, with_traceback=exc_info[2], from_=e)
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception
ERROR oslo_messaging.rpc.server     self._handle_dbapi_exception(
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
ERROR oslo_messaging.rpc.server     ret = self._execute_context(
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
ERROR oslo_messaging.rpc.server     return connection._execute_clauseelement(
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
ERROR oslo_messaging.rpc.server     return meth(self, args_10style, kwargs_10style, execution_options)
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20
ERROR oslo_messaging.rpc.server     c = connection._execute_20(
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements
ERROR oslo_messaging.rpc.server     _emit_insert_statements(
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
ERROR oslo_messaging.rpc.server     util.preloaded.orm_persistence.save_obj(
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute
ERROR oslo_messaging.rpc.server     rec.execute(self)
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
ERROR oslo_messaging.rpc.server     flush_context.execute()
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush
ERROR oslo_messaging.rpc.server     raise exception
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
ERROR oslo_messaging.rpc.server     compat.raise_(
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
ERROR oslo_messaging.rpc.server     transaction.rollback(_capture_exception=True)
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush
ERROR oslo_messaging.rpc.server     self._flush(objects)
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush
ERROR oslo_messaging.rpc.server     self.session.flush()
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session
ERROR oslo_messaging.rpc.server     next(self.gen)
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__
ERROR oslo_messaging.rpc.server     yield resource
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope
ERROR oslo_messaging.rpc.server     next(self.gen)
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__
ERROR oslo_messaging.rpc.server     return record
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
ERROR oslo_messaging.rpc.server
ERROR oslo_messaging.rpc.server The above exception was the direct cause of the following exception:
ERROR oslo_messaging.rpc.server
ERROR oslo_messaging.rpc.server pymysql.err.IntegrityError: (1062, "Duplicate entry 'f269cf3d-8a00-4b5b-99c2-63cae0968f42-gm-os-hv-017' for key 'PRIMARY'")
ERROR oslo_messaging.rpc.server     raise errorclass(errno, errval)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
ERROR oslo_messaging.rpc.server     err.raise_mysql_exception(self._data)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error
ERROR oslo_messaging.rpc.server     packet.raise_for_error()
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet
ERROR oslo_messaging.rpc.server     first_packet = self.connection._read_packet()
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read
ERROR oslo_messaging.rpc.server     result.read()
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result
ERROR oslo_messaging.rpc.server     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query
ERROR oslo_messaging.rpc.server     conn.query(q)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query
ERROR oslo_messaging.rpc.server     result = self._query(query)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute
ERROR oslo_messaging.rpc.server     cursor.execute(statement, parameters)
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
ERROR oslo_messaging.rpc.server     self.dialect.do_execute(
ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
ERROR oslo_messaging.rpc.server [req-1112fb3c-05bc-4754-b4d9-e0813bba902c - - - - -] Exception during message handling: sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'f269cf3d-8a00-4b5b-99c2-63cae0968f42-gm-os-hv-017' for key 'PRIMARY'")
```

As you can see neutron-server should handle exception for `DuplicateEntry` and return existing record:
https://github.com/openstack/neutron/blob/20.3.1/neutron/plugins/ml2/db.py#L111

But this code is not doing it and failed with:
```
sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry ..."
```

It seems that this part of the code is not covered by tests and not work
correctly.

It is reproduced on our side only in one production system and we can
not reproduce it on our lab env.

Environment
===========
AlmaLinux 9.5
```
openstack-neutron-20.3.1-1.el9s
python-oslo-db-11.2.0-1.el9s
python-oslo-messaging-12.13.3-1.el9s
python-sqlalchemy-1.4.45-3.el9
```

** Affects: neutron
     Importance: Undecided
         Status: New

** Description changed:

  Under some conditions, Neutron tries to create a record in the Database
  a few times and fails with an unexpected exception that should be
  handled by existing code.
  
  Neutron mode: ovs+dvr
  
  How to reproduce:
  1. Tenant network connected to distributed router
  2. VM with tenant network created or migrated to an empty hypervisor
  
  When VM is placed to empty node L3-agent notifies neutron-server to bind router distributed port to node and we get ERROR traceback in neutron-server log:
  ```
  ERROR oslo_messaging.rpc.server [parameters: {'port_id': 'f269cf3d-8a00-4b5b-99c2-63cae0968f42', 'host': 'gm-os-hv-017', 'router_id': 'e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}]
  ERROR oslo_messaging.rpc.server [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]
  ERROR oslo_messaging.rpc.server sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'f269cf3d-8a00-4b5b-99c2-63cae0968f42-gm-os-hv-017' for key 'PRIMARY'")
  ERROR oslo_messaging.rpc.server     raise sa_exc.PendingRollbackError(
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active
  ERROR oslo_messaging.rpc.server     self._assert_active()
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind
  ERROR oslo_messaging.rpc.server     return self._transaction._connection_for_bind(
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind
  ERROR oslo_messaging.rpc.server     conn = self._connection_for_bind(bind, close_with_result=True)
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute
  ERROR oslo_messaging.rpc.server     result = self.session.execute(
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter
  ERROR oslo_messaging.rpc.server     return self._iter().one()
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one
  ERROR oslo_messaging.rpc.server     return (context.session.query(models.DistributedPortBinding).
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding
  ERROR oslo_messaging.rpc.server     binding = db.ensure_distributed_port_binding(
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding
  ERROR oslo_messaging.rpc.server     return f(*dup_args, **dup_kwargs)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped
  ERROR oslo_messaging.rpc.server     raise self.value
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  ERROR oslo_messaging.rpc.server     self.force_reraise()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  ERROR oslo_messaging.rpc.server     LOG.debug("Retry wrapper got retriable exception: %s", e)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper
  ERROR oslo_messaging.rpc.server     raise self.value
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  ERROR oslo_messaging.rpc.server     self.force_reraise()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  ERROR oslo_messaging.rpc.server     ectxt.value = e.inner_exc
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper
  ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped
  ERROR oslo_messaging.rpc.server     raise self.value
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  ERROR oslo_messaging.rpc.server     self.force_reraise()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  ERROR oslo_messaging.rpc.server     setattr(e, '_RETRY_EXCEEDED', True)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped
  ERROR oslo_messaging.rpc.server     return method(*args, **kwargs)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped
  ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner
  ERROR oslo_messaging.rpc.server     self.plugin.update_distributed_port_binding(
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port
  ERROR oslo_messaging.rpc.server     self._ensure_host_set_on_port(
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports
  ERROR oslo_messaging.rpc.server     self._ensure_host_set_on_ports(context, host, routers)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers
  ERROR oslo_messaging.rpc.server     return f(*dup_args, **dup_kwargs)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped
  ERROR oslo_messaging.rpc.server     raise self.value
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  ERROR oslo_messaging.rpc.server     self.force_reraise()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  ERROR oslo_messaging.rpc.server     LOG.debug("Retry wrapper got retriable exception: %s", e)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper
  ERROR oslo_messaging.rpc.server     raise self.value
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  ERROR oslo_messaging.rpc.server     self.force_reraise()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  ERROR oslo_messaging.rpc.server     ectxt.value = e.inner_exc
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper
  ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped
  ERROR oslo_messaging.rpc.server     raise self.value
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  ERROR oslo_messaging.rpc.server     self.force_reraise()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  ERROR oslo_messaging.rpc.server     setattr(e, '_RETRY_EXCEEDED', True)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped
  ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
  ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
  ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
  ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  ERROR oslo_messaging.rpc.server
  ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
  ERROR oslo_messaging.rpc.server
  ERROR oslo_messaging.rpc.server (Background on this error at: https://sqlalche.me/e/14/gkpj)
  ERROR oslo_messaging.rpc.server [parameters: {'port_id': 'f269cf3d-8a00-4b5b-99c2-63cae0968f42', 'host': 'gm-os-hv-017', 'router_id': 'e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}]
  ERROR oslo_messaging.rpc.server [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]
  ERROR oslo_messaging.rpc.server oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'f269cf3d-8a00-4b5b-99c2-63cae0968f42-gm-os-hv-017' for key 'PRIMARY'")
  ERROR oslo_messaging.rpc.server     raise errorclass(errno, errval)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
  ERROR oslo_messaging.rpc.server     err.raise_mysql_exception(self._data)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error
  ERROR oslo_messaging.rpc.server     packet.raise_for_error()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet
  ERROR oslo_messaging.rpc.server     first_packet = self.connection._read_packet()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read
  ERROR oslo_messaging.rpc.server     result.read()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result
  ERROR oslo_messaging.rpc.server     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query
  ERROR oslo_messaging.rpc.server     conn.query(q)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query
  ERROR oslo_messaging.rpc.server     result = self._query(query)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute
  ERROR oslo_messaging.rpc.server     cursor.execute(statement, parameters)
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
  ERROR oslo_messaging.rpc.server     self.dialect.do_execute(
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
  ERROR oslo_messaging.rpc.server     raise exception
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
  ERROR oslo_messaging.rpc.server     util.raise_(newraise, with_traceback=exc_info[2], from_=e)
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception
  ERROR oslo_messaging.rpc.server     self._handle_dbapi_exception(
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
  ERROR oslo_messaging.rpc.server     ret = self._execute_context(
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
  ERROR oslo_messaging.rpc.server     return connection._execute_clauseelement(
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
  ERROR oslo_messaging.rpc.server     return meth(self, args_10style, kwargs_10style, execution_options)
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20
  ERROR oslo_messaging.rpc.server     c = connection._execute_20(
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements
  ERROR oslo_messaging.rpc.server     _emit_insert_statements(
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
  ERROR oslo_messaging.rpc.server     util.preloaded.orm_persistence.save_obj(
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute
  ERROR oslo_messaging.rpc.server     rec.execute(self)
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
  ERROR oslo_messaging.rpc.server     flush_context.execute()
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush
  ERROR oslo_messaging.rpc.server     raise exception
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
  ERROR oslo_messaging.rpc.server     compat.raise_(
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
  ERROR oslo_messaging.rpc.server     transaction.rollback(_capture_exception=True)
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush
  ERROR oslo_messaging.rpc.server     self._flush(objects)
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush
  ERROR oslo_messaging.rpc.server     self.session.flush()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session
  ERROR oslo_messaging.rpc.server     next(self.gen)
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__
  ERROR oslo_messaging.rpc.server     yield resource
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope
  ERROR oslo_messaging.rpc.server     next(self.gen)
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__
  ERROR oslo_messaging.rpc.server     return record
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding
  ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  ERROR oslo_messaging.rpc.server
  ERROR oslo_messaging.rpc.server The above exception was the direct cause of the following exception:
  ERROR oslo_messaging.rpc.server
  ERROR oslo_messaging.rpc.server pymysql.err.IntegrityError: (1062, "Duplicate entry 'f269cf3d-8a00-4b5b-99c2-63cae0968f42-gm-os-hv-017' for key 'PRIMARY'")
  ERROR oslo_messaging.rpc.server     raise errorclass(errno, errval)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
  ERROR oslo_messaging.rpc.server     err.raise_mysql_exception(self._data)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error
  ERROR oslo_messaging.rpc.server     packet.raise_for_error()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet
  ERROR oslo_messaging.rpc.server     first_packet = self.connection._read_packet()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read
  ERROR oslo_messaging.rpc.server     result.read()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result
  ERROR oslo_messaging.rpc.server     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query
  ERROR oslo_messaging.rpc.server     conn.query(q)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query
  ERROR oslo_messaging.rpc.server     result = self._query(query)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute
  ERROR oslo_messaging.rpc.server     cursor.execute(statement, parameters)
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
  ERROR oslo_messaging.rpc.server     self.dialect.do_execute(
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
  ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  ERROR oslo_messaging.rpc.server [req-1112fb3c-05bc-4754-b4d9-e0813bba902c - - - - -] Exception during message handling: sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'f269cf3d-8a00-4b5b-99c2-63cae0968f42-gm-os-hv-017' for key 'PRIMARY'")
  ```
  
  As you can see neutron-server should handle exception for `DuplicateEntry` and return existing record:
  https://github.com/openstack/neutron/blob/20.3.1/neutron/plugins/ml2/db.py#L111
  
  But this code is not doing it and failed with:
  ```
  sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry ..."
  ```
  
  It seems that this part of the code is not covered by tests and not work
  correctly.
  
  It is reproduced on our side only in one production system and we can
  not reproduce it on our lab env.
  
- versions:
+ Environment
+ ===========
+ AlmaLinux 9.5
  ```
  openstack-neutron-20.3.1-1.el9s
  python-oslo-db-11.2.0-1.el9s
  python-oslo-messaging-12.13.3-1.el9s
  python-sqlalchemy-1.4.45-3.el9
  ```

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2106374

Title:
  neutron-server error with ml2_distributed_port_bindings

Status in neutron:
  New

Bug description:
  Under some conditions, Neutron tries to create a record in the
  Database a few times and fails with an unexpected exception that
  should be handled by existing code.

  Neutron mode: ovs+dvr

  How to reproduce:
  1. Tenant network connected to distributed router
  2. VM with tenant network created or migrated to an empty hypervisor

  When VM is placed to empty node L3-agent notifies neutron-server to bind router distributed port to node and we get ERROR traceback in neutron-server log:
  ```
  ERROR oslo_messaging.rpc.server [parameters: {'port_id': 'f269cf3d-8a00-4b5b-99c2-63cae0968f42', 'host': 'gm-os-hv-017', 'router_id': 'e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}]
  ERROR oslo_messaging.rpc.server [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]
  ERROR oslo_messaging.rpc.server sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'f269cf3d-8a00-4b5b-99c2-63cae0968f42-gm-os-hv-017' for key 'PRIMARY'")
  ERROR oslo_messaging.rpc.server     raise sa_exc.PendingRollbackError(
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active
  ERROR oslo_messaging.rpc.server     self._assert_active()
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind
  ERROR oslo_messaging.rpc.server     return self._transaction._connection_for_bind(
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind
  ERROR oslo_messaging.rpc.server     conn = self._connection_for_bind(bind, close_with_result=True)
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute
  ERROR oslo_messaging.rpc.server     result = self.session.execute(
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter
  ERROR oslo_messaging.rpc.server     return self._iter().one()
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one
  ERROR oslo_messaging.rpc.server     return (context.session.query(models.DistributedPortBinding).
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding
  ERROR oslo_messaging.rpc.server     binding = db.ensure_distributed_port_binding(
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding
  ERROR oslo_messaging.rpc.server     return f(*dup_args, **dup_kwargs)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped
  ERROR oslo_messaging.rpc.server     raise self.value
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  ERROR oslo_messaging.rpc.server     self.force_reraise()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  ERROR oslo_messaging.rpc.server     LOG.debug("Retry wrapper got retriable exception: %s", e)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper
  ERROR oslo_messaging.rpc.server     raise self.value
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  ERROR oslo_messaging.rpc.server     self.force_reraise()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  ERROR oslo_messaging.rpc.server     ectxt.value = e.inner_exc
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper
  ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped
  ERROR oslo_messaging.rpc.server     raise self.value
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  ERROR oslo_messaging.rpc.server     self.force_reraise()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  ERROR oslo_messaging.rpc.server     setattr(e, '_RETRY_EXCEEDED', True)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped
  ERROR oslo_messaging.rpc.server     return method(*args, **kwargs)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped
  ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner
  ERROR oslo_messaging.rpc.server     self.plugin.update_distributed_port_binding(
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port
  ERROR oslo_messaging.rpc.server     self._ensure_host_set_on_port(
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports
  ERROR oslo_messaging.rpc.server     self._ensure_host_set_on_ports(context, host, routers)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers
  ERROR oslo_messaging.rpc.server     return f(*dup_args, **dup_kwargs)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped
  ERROR oslo_messaging.rpc.server     raise self.value
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  ERROR oslo_messaging.rpc.server     self.force_reraise()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  ERROR oslo_messaging.rpc.server     LOG.debug("Retry wrapper got retriable exception: %s", e)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper
  ERROR oslo_messaging.rpc.server     raise self.value
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  ERROR oslo_messaging.rpc.server     self.force_reraise()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  ERROR oslo_messaging.rpc.server     ectxt.value = e.inner_exc
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper
  ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped
  ERROR oslo_messaging.rpc.server     raise self.value
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  ERROR oslo_messaging.rpc.server     self.force_reraise()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  ERROR oslo_messaging.rpc.server     setattr(e, '_RETRY_EXCEEDED', True)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped
  ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
  ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
  ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
  ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  ERROR oslo_messaging.rpc.server
  ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
  ERROR oslo_messaging.rpc.server
  ERROR oslo_messaging.rpc.server (Background on this error at: https://sqlalche.me/e/14/gkpj)
  ERROR oslo_messaging.rpc.server [parameters: {'port_id': 'f269cf3d-8a00-4b5b-99c2-63cae0968f42', 'host': 'gm-os-hv-017', 'router_id': 'e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}]
  ERROR oslo_messaging.rpc.server [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]
  ERROR oslo_messaging.rpc.server oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'f269cf3d-8a00-4b5b-99c2-63cae0968f42-gm-os-hv-017' for key 'PRIMARY'")
  ERROR oslo_messaging.rpc.server     raise errorclass(errno, errval)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
  ERROR oslo_messaging.rpc.server     err.raise_mysql_exception(self._data)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error
  ERROR oslo_messaging.rpc.server     packet.raise_for_error()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet
  ERROR oslo_messaging.rpc.server     first_packet = self.connection._read_packet()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read
  ERROR oslo_messaging.rpc.server     result.read()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result
  ERROR oslo_messaging.rpc.server     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query
  ERROR oslo_messaging.rpc.server     conn.query(q)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query
  ERROR oslo_messaging.rpc.server     result = self._query(query)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute
  ERROR oslo_messaging.rpc.server     cursor.execute(statement, parameters)
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
  ERROR oslo_messaging.rpc.server     self.dialect.do_execute(
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
  ERROR oslo_messaging.rpc.server     raise exception
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
  ERROR oslo_messaging.rpc.server     util.raise_(newraise, with_traceback=exc_info[2], from_=e)
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception
  ERROR oslo_messaging.rpc.server     self._handle_dbapi_exception(
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
  ERROR oslo_messaging.rpc.server     ret = self._execute_context(
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
  ERROR oslo_messaging.rpc.server     return connection._execute_clauseelement(
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
  ERROR oslo_messaging.rpc.server     return meth(self, args_10style, kwargs_10style, execution_options)
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20
  ERROR oslo_messaging.rpc.server     c = connection._execute_20(
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements
  ERROR oslo_messaging.rpc.server     _emit_insert_statements(
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
  ERROR oslo_messaging.rpc.server     util.preloaded.orm_persistence.save_obj(
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute
  ERROR oslo_messaging.rpc.server     rec.execute(self)
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
  ERROR oslo_messaging.rpc.server     flush_context.execute()
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush
  ERROR oslo_messaging.rpc.server     raise exception
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
  ERROR oslo_messaging.rpc.server     compat.raise_(
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
  ERROR oslo_messaging.rpc.server     transaction.rollback(_capture_exception=True)
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush
  ERROR oslo_messaging.rpc.server     self._flush(objects)
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush
  ERROR oslo_messaging.rpc.server     self.session.flush()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session
  ERROR oslo_messaging.rpc.server     next(self.gen)
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__
  ERROR oslo_messaging.rpc.server     yield resource
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope
  ERROR oslo_messaging.rpc.server     next(self.gen)
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__
  ERROR oslo_messaging.rpc.server     return record
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding
  ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  ERROR oslo_messaging.rpc.server
  ERROR oslo_messaging.rpc.server The above exception was the direct cause of the following exception:
  ERROR oslo_messaging.rpc.server
  ERROR oslo_messaging.rpc.server pymysql.err.IntegrityError: (1062, "Duplicate entry 'f269cf3d-8a00-4b5b-99c2-63cae0968f42-gm-os-hv-017' for key 'PRIMARY'")
  ERROR oslo_messaging.rpc.server     raise errorclass(errno, errval)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
  ERROR oslo_messaging.rpc.server     err.raise_mysql_exception(self._data)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error
  ERROR oslo_messaging.rpc.server     packet.raise_for_error()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet
  ERROR oslo_messaging.rpc.server     first_packet = self.connection._read_packet()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read
  ERROR oslo_messaging.rpc.server     result.read()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result
  ERROR oslo_messaging.rpc.server     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query
  ERROR oslo_messaging.rpc.server     conn.query(q)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query
  ERROR oslo_messaging.rpc.server     result = self._query(query)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute
  ERROR oslo_messaging.rpc.server     cursor.execute(statement, parameters)
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
  ERROR oslo_messaging.rpc.server     self.dialect.do_execute(
  ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
  ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  ERROR oslo_messaging.rpc.server [req-1112fb3c-05bc-4754-b4d9-e0813bba902c - - - - -] Exception during message handling: sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'f269cf3d-8a00-4b5b-99c2-63cae0968f42-gm-os-hv-017' for key 'PRIMARY'")
  ```

  As you can see neutron-server should handle exception for `DuplicateEntry` and return existing record:
  https://github.com/openstack/neutron/blob/20.3.1/neutron/plugins/ml2/db.py#L111

  But this code is not doing it and failed with:
  ```
  sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry ..."
  ```

  It seems that this part of the code is not covered by tests and not
  work correctly.

  It is reproduced on our side only in one production system and we can
  not reproduce it on our lab env.

  Environment
  ===========
  AlmaLinux 9.5
  ```
  openstack-neutron-20.3.1-1.el9s
  python-oslo-db-11.2.0-1.el9s
  python-oslo-messaging-12.13.3-1.el9s
  python-sqlalchemy-1.4.45-3.el9
  ```

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2106374/+subscriptions
  Thread Previous • Date Previous • • Thread Next