yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #92819
[Bug 2035573] Re: Insertion of a duplicated ProviderResourceAssociation entry while creating a HA router
*** This bug is a duplicate of bug 2016198 ***
https://bugs.launchpad.net/bugs/2016198
Hello:
As Brian commented, this issue was solved in [1]. The problem is that
this fix includes a DB migration and cannot be backported. Please
upgrade to stable/2023.2, if possible, and check again.
Regards.
[1]https://bugs.launchpad.net/neutron/+bug/2016198
** This bug has been marked a duplicate of bug 2016198
[L3][HA] race condition between first two router creations when tenant has no HA network
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2035573
Title:
Insertion of a duplicated ProviderResourceAssociation entry while
creating a HA router
Status in neutron:
Incomplete
Bug description:
[SUMMARY]
While creating multiple HA routers in a row, some router creations are failed with the log below.
2023-07-21 02:28:19.968 12 DEBUG neutron.wsgi [-] (12) accepted ('10.2.41.158', 57438) server /var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/wsgi.py:1004
2023-07-21 02:28:19.979 12 DEBUG neutron.api.v2.base [None req-726ab30e-b9c9-4e38-8c9d-216752ab2aea 439ae1ccaa9a494284cee3fdb6227208 97895007888245c3acdfc41146d2e151 - - default default] Request body: {'router': {'name': 'test6', 'admin_state_up': True, 'tenant_id': '97895007888245c3acdfc41146d2e151'}} prepare_request_body /var/lib/kolla/venv/lib/python3.10/site-packages/neutron/api/v2/base.py:731
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db [None req-726ab30e-b9c9-4e38-8c9d-216752ab2aea 439ae1ccaa9a494284cee3fdb6227208 97895007888245c3acdfc41146d2e151 - - default default] Failed to schedule HA router 5a599ade-7b6a-4b3e-b635-ca00e37f2657.: neutron_lib.objects.exceptions.NeutronDbObjectDuplicateEntry: Failed to create a duplicate ProviderResourceAssociation: for attribute(s) ['PRIMARY'] with value(s) ha-5a599ade-7b6a-4b3e-b635-ca00e37f2657
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db Traceback (most recent call last):
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db self.dialect.do_execute(
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db cursor.execute(statement, parameters)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 148, in execute
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db result = self._query(query)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 310, in _query
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db conn.query(q)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 548, in query
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 775, in _read_query_result
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db result.read()
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 1156, in read
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db first_packet = self.connection._read_packet()
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 725, in _read_packet
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db packet.raise_for_error()
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/protocol.py", line 221, in raise_for_error
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db err.raise_mysql_exception(self._data)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db raise errorclass(errno, errval)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db pymysql.err.IntegrityError: (1062, "Duplicate entry 'ha-5a599ade-7b6a-4b3e-b635-ca00e37f2657' for key 'PRIMARY'")
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db The above exception was the direct cause of the following exception:
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db Traceback (most recent call last):
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/objects/base.py", line 903, in create
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db db_obj = obj_db_api.create_object(
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/objects/db/api.py", line 72, in create_object
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db with obj_cls.db_context_writer(context):
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db next(self.gen)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1066, in _transaction_scope
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db with current._produce_block(
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db next(self.gen)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 688, in _session
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db self.session.flush()
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3386, in flush
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db self._flush(objects)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3525, in _flush
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db with util.safe_reraise():
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db compat.raise_(
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db raise exception
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3486, in _flush
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db flush_context.execute()
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db rec.execute(self)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db util.preloaded.orm_persistence.save_obj(
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db _emit_insert_statements(
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db c = connection._execute_20(
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db return meth(self, args_10style, kwargs_10style, execution_options)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db return connection._execute_clauseelement(
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db ret = self._execute_context(
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db self._handle_dbapi_exception(
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db util.raise_(newraise, with_traceback=exc_info[2], from_=e)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db raise exception
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db self.dialect.do_execute(
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db cursor.execute(statement, parameters)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 148, in execute
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db result = self._query(query)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 310, in _query
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db conn.query(q)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 548, in query
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 775, in _read_query_result
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db result.read()
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 1156, in read
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db first_packet = self.connection._read_packet()
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 725, in _read_packet
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db packet.raise_for_error()
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/protocol.py", line 221, in raise_for_error
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db err.raise_mysql_exception(self._data)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db raise errorclass(errno, errval)
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'ha-5a599ade-7b6a-4b3e-b635-ca00e37f2657' for key 'PRIMARY'")
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db [SQL: INSERT INTO providerresourceassociations (provider_name, resource_id) VALUES (%(provider_name)s, %(resource_id)s)]
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db [parameters: {'provider_name': 'ha', 'resource_id': '5a599ade-7b6a-4b3e-b635-ca00e37f2657'}]
2023-07-21 02:28:20.178 12 ERROR neutron.db.l3_hamode_db (Background on this error at: https://sqlalche.me/e/14/gkpj)
[REPRODUCTION STEPS]
Create HA routers in a row using the script below. More than 10 times of script execution would be needed.
#!/bin/bash
openstack router create 1
openstack router create 2
openstack router create 3
openstack router create 4
openstack router create 5
openstack router delete 1 2 3 4 5
[VERSION]
OpenStack version: stable/2023.1
OS: Ubuntu 22.04
Others: openvswitch 2.17.7
[ENVIRONMENT]
Installed on K8S nodes using openstack-helm-infra chart version 0.3.14. Currently running pods are list below.
$ kubectl get pod -A -l application=neutron
NAMESPACE NAME READY STATUS RESTARTS AGE
openstack neutron-db-init-qvc5r 0/1 Completed 0 43h
openstack neutron-db-sync-xcsw8 0/1 Completed 0 43h
openstack neutron-dhcp-agent-default-6nzgw 1/1 Running 0 43h
openstack neutron-dhcp-agent-default-rbxk7 1/1 Running 0 43h
openstack neutron-dhcp-agent-default-s7jrb 1/1 Running 0 43h
openstack neutron-ks-endpoints-qqrn9 0/3 Completed 0 43h
openstack neutron-ks-service-2v9ls 0/1 Completed 0 43h
openstack neutron-ks-user-d74c5 0/1 Completed 0 43h
openstack neutron-l3-agent-default-49lht 1/1 Running 0 43h
openstack neutron-l3-agent-default-7s7bh 1/1 Running 0 43h
openstack neutron-l3-agent-default-l48mv 1/1 Running 0 43h
openstack neutron-metadata-agent-default-6r8xf 1/1 Running 0 43h
openstack neutron-metadata-agent-default-gzf29 1/1 Running 0 43h
openstack neutron-metadata-agent-default-mrlmd 1/1 Running 0 43h
openstack neutron-netns-cleanup-cron-default-45cbq 1/1 Running 0 43h
openstack neutron-netns-cleanup-cron-default-bktfq 1/1 Running 0 43h
openstack neutron-netns-cleanup-cron-default-pf9wg 1/1 Running 0 43h
openstack neutron-ovs-agent-default-8c87c 1/1 Running 0 43h
openstack neutron-ovs-agent-default-gg7pc 1/1 Running 0 43h
openstack neutron-ovs-agent-default-jll4x 1/1 Running 0 43h
openstack neutron-ovs-agent-default-mxr5g 1/1 Running 0 43h
openstack neutron-ovs-agent-default-tjp59 1/1 Running 0 43h
openstack neutron-ovs-agent-default-vqz44 1/1 Running 0 43h
openstack neutron-ovs-agent-default-wbcwx 1/1 Running 0 43h
openstack neutron-rabbit-init-gzg62 0/1 Completed 0 43h
openstack neutron-server-c77b7bdcd-4jq5m 1/1 Running 0 43h
openstack neutron-server-c77b7bdcd-4tzc9 1/1 Running 0 43h
openstack neutron-server-c77b7bdcd-5fwnw 1/1 Running 0 43h
openstack neutron-server-c77b7bdcd-6txp2 1/1 Running 0 43h
openstack neutron-server-c77b7bdcd-clm65 1/1 Running 0 43h
openstack neutron-server-c77b7bdcd-j95n2 1/1 Running 0 43h
openstack neutron-server-c77b7bdcd-khkkg 1/1 Running 0 43h
openstack neutron-server-c77b7bdcd-l8qdm 1/1 Running 0 43h
openstack neutron-server-c77b7bdcd-mv7nk 1/1 Running 0 43h
openstack neutron-server-c77b7bdcd-ncszs 1/1 Running 0 43h
openstack neutron-server-c77b7bdcd-p6m4w 1/1 Running 0 43h
openstack neutron-server-c77b7bdcd-qxqbs 1/1 Running 0 43h
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2035573/+subscriptions
References
