yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2035573] [NEW] Insertion of a duplicated ProviderResourceAssociation entry while creating a HA router

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Jimin Shin <2035573@xxxxxxxxxxxxxxxxxx>
Date: Thu, 14 Sep 2023 05:02:26 -0000
Reply-to: Bug 2035573 <2035573@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

[SUMMARY]
While creating multiple HA routers in a row, some router creations are failed with the log below.

2023-07-21 02:28:19.968 12 DEBUG neutron.wsgi [-] (12) accepted ('10.2.41.158', 57438) server /var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/wsgi.py:1004
2023-07-21 02:28:19.979 12 DEBUG neutron.api.v2.base [None req-726ab30e-b9c9-4e38-8c9d-216752ab2aea 439ae1ccaa9a494284cee3fdb6227208 97895007888245c3acdfc41146d2e151 - - default default] Request body: {'router': {'name': 'test6', 'admin_state_up': True, 'tenant_id': '97895007888245c3acdfc41146d2e151'}} prepare_request_body /var/lib/kolla/venv/lib/python3.10/site-packages/neutron/api/v2/base.py:731
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db [None req-726ab30e-b9c9-4e38-8c9d-216752ab2aea 439ae1ccaa9a494284cee3fdb6227208 97895007888245c3acdfc41146d2e151 - - default default] Failed to schedule HA router 5a599ade-7b6a-4b3e-b635-ca00e37f2657.: neutron_lib.objects.exceptions.NeutronDbObjectDuplicateEntry: Failed to create a duplicate ProviderResourceAssociation: for attribute(s) ['PRIMARY'] with value(s) ha-5a599ade-7b6a-4b3e-b635-ca00e37f2657
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db Traceback (most recent call last):
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     self.dialect.do_execute(
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     cursor.execute(statement, parameters)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 148, in execute
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     result = self._query(query)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 310, in _query
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     conn.query(q)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 548, in query
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 775, in _read_query_result
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     result.read()
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 1156, in read
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     first_packet = self.connection._read_packet()
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 725, in _read_packet
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     packet.raise_for_error()
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/protocol.py", line 221, in raise_for_error
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     err.raise_mysql_exception(self._data)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     raise errorclass(errno, errval)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db pymysql.err.IntegrityError: (1062, "Duplicate entry 'ha-5a599ade-7b6a-4b3e-b635-ca00e37f2657' for key 'PRIMARY'")
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db The above exception was the direct cause of the following exception:
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db Traceback (most recent call last):
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/objects/base.py", line 903, in create
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     db_obj = obj_db_api.create_object(
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/objects/db/api.py", line 72, in create_object
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     with obj_cls.db_context_writer(context):
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     next(self.gen)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1066, in _transaction_scope
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     with current._produce_block(
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     next(self.gen)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 688, in _session
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     self.session.flush()
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3386, in flush
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     self._flush(objects)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3525, in _flush
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     with util.safe_reraise():
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     compat.raise_(
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     raise exception
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3486, in _flush
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     flush_context.execute()
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     rec.execute(self)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     util.preloaded.orm_persistence.save_obj(
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     _emit_insert_statements(
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     c = connection._execute_20(
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     return meth(self, args_10style, kwargs_10style, execution_options)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     return connection._execute_clauseelement(
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     ret = self._execute_context(
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     self._handle_dbapi_exception(
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     util.raise_(newraise, with_traceback=exc_info[2], from_=e)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     raise exception
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     self.dialect.do_execute(
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     cursor.execute(statement, parameters)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 148, in execute
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     result = self._query(query)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 310, in _query
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     conn.query(q)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 548, in query
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 775, in _read_query_result
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     result.read()
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 1156, in read
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     first_packet = self.connection._read_packet()
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 725, in _read_packet
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     packet.raise_for_error()
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/protocol.py", line 221, in raise_for_error
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     err.raise_mysql_exception(self._data)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     raise errorclass(errno, errval)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'ha-5a599ade-7b6a-4b3e-b635-ca00e37f2657' for key 'PRIMARY'")
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db [SQL: INSERT INTO providerresourceassociations (provider_name, resource_id) VALUES (%(provider_name)s, %(resource_id)s)]
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db [parameters: {'provider_name': 'ha', 'resource_id': '5a599ade-7b6a-4b3e-b635-ca00e37f2657'}]
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db (Background on this error at: https://sqlalche.me/e/14/gkpj)

[REPRODUCTION STEPS]
Create HA routers in a row using the script below. More than 10 times of script execution would be needed.

#!/bin/bash

openstack router create 1
openstack router create 2
openstack router create 3
openstack router create 4
openstack router create 5
openstack router delete 1 2 3 4 5

[VERSION]
OpenStack version: stable/2023.1
OS: Ubuntu 22.04
Others: openvswitch 2.17.7

[ENVIRONMENT]
Installed on K8S nodes using openstack-helm-infra chart version 0.3.14. Currently running pods are list below.

$ kubectl get pod -A -l application=neutron
NAMESPACE   NAME                                       READY   STATUS      RESTARTS   AGE
openstack   neutron-db-init-qvc5r                      0/1     Completed   0          43h
openstack   neutron-db-sync-xcsw8                      0/1     Completed   0          43h
openstack   neutron-dhcp-agent-default-6nzgw           1/1     Running     0          43h
openstack   neutron-dhcp-agent-default-rbxk7           1/1     Running     0          43h
openstack   neutron-dhcp-agent-default-s7jrb           1/1     Running     0          43h
openstack   neutron-ks-endpoints-qqrn9                 0/3     Completed   0          43h
openstack   neutron-ks-service-2v9ls                   0/1     Completed   0          43h
openstack   neutron-ks-user-d74c5                      0/1     Completed   0          43h
openstack   neutron-l3-agent-default-49lht             1/1     Running     0          43h
openstack   neutron-l3-agent-default-7s7bh             1/1     Running     0          43h
openstack   neutron-l3-agent-default-l48mv             1/1     Running     0          43h
openstack   neutron-metadata-agent-default-6r8xf       1/1     Running     0          43h
openstack   neutron-metadata-agent-default-gzf29       1/1     Running     0          43h
openstack   neutron-metadata-agent-default-mrlmd       1/1     Running     0          43h
openstack   neutron-netns-cleanup-cron-default-45cbq   1/1     Running     0          43h
openstack   neutron-netns-cleanup-cron-default-bktfq   1/1     Running     0          43h
openstack   neutron-netns-cleanup-cron-default-pf9wg   1/1     Running     0          43h
openstack   neutron-ovs-agent-default-8c87c            1/1     Running     0          43h
openstack   neutron-ovs-agent-default-gg7pc            1/1     Running     0          43h
openstack   neutron-ovs-agent-default-jll4x            1/1     Running     0          43h
openstack   neutron-ovs-agent-default-mxr5g            1/1     Running     0          43h
openstack   neutron-ovs-agent-default-tjp59            1/1     Running     0          43h
openstack   neutron-ovs-agent-default-vqz44            1/1     Running     0          43h
openstack   neutron-ovs-agent-default-wbcwx            1/1     Running     0          43h
openstack   neutron-rabbit-init-gzg62                  0/1     Completed   0          43h
openstack   neutron-server-c77b7bdcd-4jq5m             1/1     Running     0          43h
openstack   neutron-server-c77b7bdcd-4tzc9             1/1     Running     0          43h
openstack   neutron-server-c77b7bdcd-5fwnw             1/1     Running     0          43h
openstack   neutron-server-c77b7bdcd-6txp2             1/1     Running     0          43h
openstack   neutron-server-c77b7bdcd-clm65             1/1     Running     0          43h
openstack   neutron-server-c77b7bdcd-j95n2             1/1     Running     0          43h
openstack   neutron-server-c77b7bdcd-khkkg             1/1     Running     0          43h
openstack   neutron-server-c77b7bdcd-l8qdm             1/1     Running     0          43h
openstack   neutron-server-c77b7bdcd-mv7nk             1/1     Running     0          43h
openstack   neutron-server-c77b7bdcd-ncszs             1/1     Running     0          43h
openstack   neutron-server-c77b7bdcd-p6m4w             1/1     Running     0          43h
openstack   neutron-server-c77b7bdcd-qxqbs             1/1     Running     0          43h

** Affects: neutron
     Importance: Undecided
         Status: New

** Description changed:

  [SUMMARY]
  While creating multiple HA routers in a row, some router creations are failed with the log below.
  
  2023-07-21 02:28:19.968 12 DEBUG neutron.wsgi [-] (12) accepted ('10.2.41.158', 57438) server /var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/wsgi.py:1004
  2023-07-21 02:28:19.979 12 DEBUG neutron.api.v2.base [None req-726ab30e-b9c9-4e38-8c9d-216752ab2aea 439ae1ccaa9a494284cee3fdb6227208 97895007888245c3acdfc41146d2e151 - - default default] Request body: {'router': {'name': 'test6', 'admin_state_up': True, 'tenant_id': '97895007888245c3acdfc41146d2e151'}} prepare_request_body /var/lib/kolla/venv/lib/python3.10/site-packages/neutron/api/v2/base.py:731
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db [None req-726ab30e-b9c9-4e38-8c9d-216752ab2aea 439ae1ccaa9a494284cee3fdb6227208 97895007888245c3acdfc41146d2e151 - - default default] Failed to schedule HA router 5a599ade-7b6a-4b3e-b635-ca00e37f2657.: neutron_lib.objects.exceptions.NeutronDbObjectDuplicateEntry: Failed to create a duplicate ProviderResourceAssociation: for attribute(s) ['PRIMARY'] with value(s) ha-5a599ade-7b6a-4b3e-b635-ca00e37f2657
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db Traceback (most recent call last):
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     self.dialect.do_execute(
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     cursor.execute(statement, parameters)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 148, in execute
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     result = self._query(query)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 310, in _query
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     conn.query(q)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 548, in query
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 775, in _read_query_result
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     result.read()
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 1156, in read
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     first_packet = self.connection._read_packet()
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 725, in _read_packet
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     packet.raise_for_error()
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/protocol.py", line 221, in raise_for_error
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     err.raise_mysql_exception(self._data)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     raise errorclass(errno, errval)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db pymysql.err.IntegrityError: (1062, "Duplicate entry 'ha-5a599ade-7b6a-4b3e-b635-ca00e37f2657' for key 'PRIMARY'")
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db The above exception was the direct cause of the following exception:
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db Traceback (most recent call last):
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/objects/base.py", line 903, in create
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     db_obj = obj_db_api.create_object(
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/objects/db/api.py", line 72, in create_object
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     with obj_cls.db_context_writer(context):
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     next(self.gen)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1066, in _transaction_scope
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     with current._produce_block(
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     next(self.gen)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 688, in _session
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     self.session.flush()
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3386, in flush
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     self._flush(objects)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3525, in _flush
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     with util.safe_reraise():
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     compat.raise_(
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     raise exception
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3486, in _flush
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     flush_context.execute()
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     rec.execute(self)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     util.preloaded.orm_persistence.save_obj(
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     _emit_insert_statements(
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     c = connection._execute_20(
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     return meth(self, args_10style, kwargs_10style, execution_options)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     return connection._execute_clauseelement(
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     ret = self._execute_context(
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     self._handle_dbapi_exception(
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     util.raise_(newraise, with_traceback=exc_info[2], from_=e)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     raise exception
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     self.dialect.do_execute(
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     cursor.execute(statement, parameters)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 148, in execute
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     result = self._query(query)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 310, in _query
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     conn.query(q)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 548, in query
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 775, in _read_query_result
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     result.read()
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 1156, in read
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     first_packet = self.connection._read_packet()
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 725, in _read_packet
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     packet.raise_for_error()
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/protocol.py", line 221, in raise_for_error
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     err.raise_mysql_exception(self._data)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     raise errorclass(errno, errval)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'ha-5a599ade-7b6a-4b3e-b635-ca00e37f2657' for key 'PRIMARY'")
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db [SQL: INSERT INTO providerresourceassociations (provider_name, resource_id) VALUES (%(provider_name)s, %(resource_id)s)]
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db [parameters: {'provider_name': 'ha', 'resource_id': '5a599ade-7b6a-4b3e-b635-ca00e37f2657'}]
- 2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db (Background on this error at: https://sqlalche.me/e/14/gkpj) 
- 
+ 2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db (Background on this error at: https://sqlalche.me/e/14/gkpj)
  
  [REPRODUCTION STEPS]
- Create HA routers in a row using the script below.
+ Create HA routers in a row using the script below. More than 10 times of script execution would be needed.
  
  #!/bin/bash
  
  openstack router create 1
  openstack router create 2
  openstack router create 3
  openstack router create 4
  openstack router create 5
  openstack router delete 1 2 3 4 5
  
- 
  [VERSION]
  OpenStack version: stable/2023.1
  OS: Ubuntu 22.04
  Others: openvswitch 2.17.7
- 
  
  [ENVIRONMENT]
  Installed on K8S nodes using openstack-helm-infra chart version 0.3.14. Currently running pods are list below.
  
  $ kubectl get pod -A -l application=neutron
  NAMESPACE   NAME                                       READY   STATUS      RESTARTS   AGE
  openstack   neutron-db-init-qvc5r                      0/1     Completed   0          43h
  openstack   neutron-db-sync-xcsw8                      0/1     Completed   0          43h
  openstack   neutron-dhcp-agent-default-6nzgw           1/1     Running     0          43h
  openstack   neutron-dhcp-agent-default-rbxk7           1/1     Running     0          43h
  openstack   neutron-dhcp-agent-default-s7jrb           1/1     Running     0          43h
  openstack   neutron-ks-endpoints-qqrn9                 0/3     Completed   0          43h
  openstack   neutron-ks-service-2v9ls                   0/1     Completed   0          43h
  openstack   neutron-ks-user-d74c5                      0/1     Completed   0          43h
  openstack   neutron-l3-agent-default-49lht             1/1     Running     0          43h
  openstack   neutron-l3-agent-default-7s7bh             1/1     Running     0          43h
  openstack   neutron-l3-agent-default-l48mv             1/1     Running     0          43h
  openstack   neutron-metadata-agent-default-6r8xf       1/1     Running     0          43h
  openstack   neutron-metadata-agent-default-gzf29       1/1     Running     0          43h
  openstack   neutron-metadata-agent-default-mrlmd       1/1     Running     0          43h
  openstack   neutron-netns-cleanup-cron-default-45cbq   1/1     Running     0          43h
  openstack   neutron-netns-cleanup-cron-default-bktfq   1/1     Running     0          43h
  openstack   neutron-netns-cleanup-cron-default-pf9wg   1/1     Running     0          43h
  openstack   neutron-ovs-agent-default-8c87c            1/1     Running     0          43h
  openstack   neutron-ovs-agent-default-gg7pc            1/1     Running     0          43h
  openstack   neutron-ovs-agent-default-jll4x            1/1     Running     0          43h
  openstack   neutron-ovs-agent-default-mxr5g            1/1     Running     0          43h
  openstack   neutron-ovs-agent-default-tjp59            1/1     Running     0          43h
  openstack   neutron-ovs-agent-default-vqz44            1/1     Running     0          43h
  openstack   neutron-ovs-agent-default-wbcwx            1/1     Running     0          43h
  openstack   neutron-rabbit-init-gzg62                  0/1     Completed   0          43h
  openstack   neutron-server-c77b7bdcd-4jq5m             1/1     Running     0          43h
  openstack   neutron-server-c77b7bdcd-4tzc9             1/1     Running     0          43h
  openstack   neutron-server-c77b7bdcd-5fwnw             1/1     Running     0          43h
  openstack   neutron-server-c77b7bdcd-6txp2             1/1     Running     0          43h
  openstack   neutron-server-c77b7bdcd-clm65             1/1     Running     0          43h
  openstack   neutron-server-c77b7bdcd-j95n2             1/1     Running     0          43h
  openstack   neutron-server-c77b7bdcd-khkkg             1/1     Running     0          43h
  openstack   neutron-server-c77b7bdcd-l8qdm             1/1     Running     0          43h
  openstack   neutron-server-c77b7bdcd-mv7nk             1/1     Running     0          43h
  openstack   neutron-server-c77b7bdcd-ncszs             1/1     Running     0          43h
  openstack   neutron-server-c77b7bdcd-p6m4w             1/1     Running     0          43h
  openstack   neutron-server-c77b7bdcd-qxqbs             1/1     Running     0          43h

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2035573

Title:
  Insertion of a duplicated ProviderResourceAssociation entry while
  creating a HA router

Status in neutron:
  New

Bug description:
  [SUMMARY]
  While creating multiple HA routers in a row, some router creations are failed with the log below.

  2023-07-21 02:28:19.968 12 DEBUG neutron.wsgi [-] (12) accepted ('10.2.41.158', 57438) server /var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/wsgi.py:1004
  2023-07-21 02:28:19.979 12 DEBUG neutron.api.v2.base [None req-726ab30e-b9c9-4e38-8c9d-216752ab2aea 439ae1ccaa9a494284cee3fdb6227208 97895007888245c3acdfc41146d2e151 - - default default] Request body: {'router': {'name': 'test6', 'admin_state_up': True, 'tenant_id': '97895007888245c3acdfc41146d2e151'}} prepare_request_body /var/lib/kolla/venv/lib/python3.10/site-packages/neutron/api/v2/base.py:731
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db [None req-726ab30e-b9c9-4e38-8c9d-216752ab2aea 439ae1ccaa9a494284cee3fdb6227208 97895007888245c3acdfc41146d2e151 - - default default] Failed to schedule HA router 5a599ade-7b6a-4b3e-b635-ca00e37f2657.: neutron_lib.objects.exceptions.NeutronDbObjectDuplicateEntry: Failed to create a duplicate ProviderResourceAssociation: for attribute(s) ['PRIMARY'] with value(s) ha-5a599ade-7b6a-4b3e-b635-ca00e37f2657
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db Traceback (most recent call last):
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     self.dialect.do_execute(
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     cursor.execute(statement, parameters)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 148, in execute
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     result = self._query(query)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 310, in _query
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     conn.query(q)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 548, in query
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 775, in _read_query_result
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     result.read()
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 1156, in read
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     first_packet = self.connection._read_packet()
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 725, in _read_packet
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     packet.raise_for_error()
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/protocol.py", line 221, in raise_for_error
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     err.raise_mysql_exception(self._data)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     raise errorclass(errno, errval)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db pymysql.err.IntegrityError: (1062, "Duplicate entry 'ha-5a599ade-7b6a-4b3e-b635-ca00e37f2657' for key 'PRIMARY'")
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db The above exception was the direct cause of the following exception:
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db Traceback (most recent call last):
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/objects/base.py", line 903, in create
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     db_obj = obj_db_api.create_object(
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/objects/db/api.py", line 72, in create_object
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     with obj_cls.db_context_writer(context):
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     next(self.gen)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1066, in _transaction_scope
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     with current._produce_block(
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     next(self.gen)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 688, in _session
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     self.session.flush()
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3386, in flush
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     self._flush(objects)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3525, in _flush
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     with util.safe_reraise():
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     compat.raise_(
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     raise exception
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3486, in _flush
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     flush_context.execute()
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     rec.execute(self)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     util.preloaded.orm_persistence.save_obj(
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     _emit_insert_statements(
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     c = connection._execute_20(
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     return meth(self, args_10style, kwargs_10style, execution_options)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     return connection._execute_clauseelement(
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     ret = self._execute_context(
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     self._handle_dbapi_exception(
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     util.raise_(newraise, with_traceback=exc_info[2], from_=e)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     raise exception
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     self.dialect.do_execute(
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     cursor.execute(statement, parameters)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 148, in execute
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     result = self._query(query)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 310, in _query
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     conn.query(q)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 548, in query
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 775, in _read_query_result
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     result.read()
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 1156, in read
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     first_packet = self.connection._read_packet()
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 725, in _read_packet
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     packet.raise_for_error()
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/protocol.py", line 221, in raise_for_error
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     err.raise_mysql_exception(self._data)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db     raise errorclass(errno, errval)
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'ha-5a599ade-7b6a-4b3e-b635-ca00e37f2657' for key 'PRIMARY'")
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db [SQL: INSERT INTO providerresourceassociations (provider_name, resource_id) VALUES (%(provider_name)s, %(resource_id)s)]
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db [parameters: {'provider_name': 'ha', 'resource_id': '5a599ade-7b6a-4b3e-b635-ca00e37f2657'}]
  2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db (Background on this error at: https://sqlalche.me/e/14/gkpj)

  [REPRODUCTION STEPS]
  Create HA routers in a row using the script below. More than 10 times of script execution would be needed.

  #!/bin/bash

  openstack router create 1
  openstack router create 2
  openstack router create 3
  openstack router create 4
  openstack router create 5
  openstack router delete 1 2 3 4 5

  [VERSION]
  OpenStack version: stable/2023.1
  OS: Ubuntu 22.04
  Others: openvswitch 2.17.7

  [ENVIRONMENT]
  Installed on K8S nodes using openstack-helm-infra chart version 0.3.14. Currently running pods are list below.

  $ kubectl get pod -A -l application=neutron
  NAMESPACE   NAME                                       READY   STATUS      RESTARTS   AGE
  openstack   neutron-db-init-qvc5r                      0/1     Completed   0          43h
  openstack   neutron-db-sync-xcsw8                      0/1     Completed   0          43h
  openstack   neutron-dhcp-agent-default-6nzgw           1/1     Running     0          43h
  openstack   neutron-dhcp-agent-default-rbxk7           1/1     Running     0          43h
  openstack   neutron-dhcp-agent-default-s7jrb           1/1     Running     0          43h
  openstack   neutron-ks-endpoints-qqrn9                 0/3     Completed   0          43h
  openstack   neutron-ks-service-2v9ls                   0/1     Completed   0          43h
  openstack   neutron-ks-user-d74c5                      0/1     Completed   0          43h
  openstack   neutron-l3-agent-default-49lht             1/1     Running     0          43h
  openstack   neutron-l3-agent-default-7s7bh             1/1     Running     0          43h
  openstack   neutron-l3-agent-default-l48mv             1/1     Running     0          43h
  openstack   neutron-metadata-agent-default-6r8xf       1/1     Running     0          43h
  openstack   neutron-metadata-agent-default-gzf29       1/1     Running     0          43h
  openstack   neutron-metadata-agent-default-mrlmd       1/1     Running     0          43h
  openstack   neutron-netns-cleanup-cron-default-45cbq   1/1     Running     0          43h
  openstack   neutron-netns-cleanup-cron-default-bktfq   1/1     Running     0          43h
  openstack   neutron-netns-cleanup-cron-default-pf9wg   1/1     Running     0          43h
  openstack   neutron-ovs-agent-default-8c87c            1/1     Running     0          43h
  openstack   neutron-ovs-agent-default-gg7pc            1/1     Running     0          43h
  openstack   neutron-ovs-agent-default-jll4x            1/1     Running     0          43h
  openstack   neutron-ovs-agent-default-mxr5g            1/1     Running     0          43h
  openstack   neutron-ovs-agent-default-tjp59            1/1     Running     0          43h
  openstack   neutron-ovs-agent-default-vqz44            1/1     Running     0          43h
  openstack   neutron-ovs-agent-default-wbcwx            1/1     Running     0          43h
  openstack   neutron-rabbit-init-gzg62                  0/1     Completed   0          43h
  openstack   neutron-server-c77b7bdcd-4jq5m             1/1     Running     0          43h
  openstack   neutron-server-c77b7bdcd-4tzc9             1/1     Running     0          43h
  openstack   neutron-server-c77b7bdcd-5fwnw             1/1     Running     0          43h
  openstack   neutron-server-c77b7bdcd-6txp2             1/1     Running     0          43h
  openstack   neutron-server-c77b7bdcd-clm65             1/1     Running     0          43h
  openstack   neutron-server-c77b7bdcd-j95n2             1/1     Running     0          43h
  openstack   neutron-server-c77b7bdcd-khkkg             1/1     Running     0          43h
  openstack   neutron-server-c77b7bdcd-l8qdm             1/1     Running     0          43h
  openstack   neutron-server-c77b7bdcd-mv7nk             1/1     Running     0          43h
  openstack   neutron-server-c77b7bdcd-ncszs             1/1     Running     0          43h
  openstack   neutron-server-c77b7bdcd-p6m4w             1/1     Running     0          43h
  openstack   neutron-server-c77b7bdcd-qxqbs             1/1     Running     0          43h

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2035573/+subscriptions
Follow ups

[Bug 2035573] Re: Insertion of a duplicated ProviderResourceAssociation entry while creating a HA router
From: Rodolfo Alonso, 2023-09-19