yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2111108] [NEW] DBDeadlock in nova.instance_extra

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Stanislav Dmitriev <2111108@xxxxxxxxxxxxxxxxxx>
Date: Fri, 16 May 2025 14:10:43 -0000
Reply-to: Bug 2111108 <2111108@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Our environment has a high amount of ephemeral instances (spawn -> do a
task -> deleted) therefore to keep the Nova DB clean we run periodically
"nova-manage db archive_deleted_rows" to archive and purge old DB
entries, even with a small "--max_rows" it causes occasional DB
deadlocks and leave us with instances stuck forever with BUILD status
and "scheduling" task state.

The nova-conductor error related to such instances:
{'code': 500, 'created': '2025-05-15T05:10:15Z', 'message': 'RemoteError', 'details': 'Traceback (most recent call last):\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 207, in decorated_function\n    return function(self, context, *args, **kwargs)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 2145, in _do_build_and_run_instance\n    (task_states.SCHEDULING, None))\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_versionedobjects/base.py", line 210, in wrapper\n    ctxt, self, fn.__name__, args, kwargs)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/conductor/rpcapi.py", line 246, in object_action\n    objmethod=objmethod, args=args, kwargs=kwargs)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 181, in call\n    transport_options=self.transport_options)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/transport.py", line 129, in _send\n    transport_options=transport_options)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 674, in send\n    transport_options=transport_options)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 664, in _send\n    raise result\noslo_messaging.rpc.client.RemoteError: Remote error: DBDeadlock (pymysql.err.OperationalError) (1213, \'Deadlock found when trying to get lock; try restarting transaction\')\n[SQL: UPDATE instance_extra SET updated_at=%(updated_at)s, numa_topology=%(numa_topology)s, pci_requests=%(pci_requests)s, flavor=%(flavor)s, vcpu_model=%(vcpu_model)s WHERE instance_extra.deleted = %(deleted_1)s AND instance_extra.instance_uuid = %(instance_uuid_1)s]\n[parameters: {\'updated_at\': datetime.datetime(2025, 5, 15, 5, 10, 15, 769294), \'numa_topology\': \'{"nova_object.name":                      "InstanceNUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.3", "nova_object.data": {"cells": [{"nova_obj ... (731 characters truncated) ... set_reserved", "id", "cpu_pinning_raw", "pagesize"]}], "emulator_threads_policy": null}, "nova_object.changes": ["cells", "emulator_threads_policy"]}\', \'pci_requests\': \'[{"count": 1, "spec": [{"vendor_id": "10de", "product_id": "1e07", "dev_type": "type-PCI"}], "alias_name": "RTX_2080ti_revA", "is_new": false, "numa_ ... (538 characters truncated) ... "1ad7", "dev_type": "type-PCI"}], "alias_name": "RTX_2080tiex2", "is_new": false, "numa_policy": "legacy", "request_id": null, "requester_id": null}]\', \'flavor\': \'{"cur": {"nova_object.name": "Flavor", "nova_object.namespace": "nova",    "nova_object.version": "1.2", "nova_object.data": {"id": 651, "name": "ux4-g1 ... (547 characters truncated) ... "2025-02-18T20:01:16Z", "updated_at": null, "deleted_at": null, "deleted": false}, "nova_object.changes": ["extra_specs"]}, "old": null, "new": null}\', \'vcpu_model\': \'{"nova_object.name": "VirtCPUModel", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"arch": null, "vendor": null ... (462 characters truncated) ... ": "host-passthrough", "model": null, "match": "exact"}, "nova_object.changes": ["features", "topology", "match", "model", "mode", "arch", "vendor"]}\', \'deleted_1\': 0, \'instance_uuid_1\': \'42288ebb-279c-483e-ad00-993ed453f823\'}]\n(Background on this error at: http://sqlalche.me/e/e3q8)\n[\'Traceback (most recent call last):\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context\\n    cursor, statement, parameters, context\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute\\n    cursor.execute(statement, parameters)\\n\', \'  File  "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 170, in execute\\n    result = self._query(query)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 328, in _query\\n    conn.query(q)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 517, in query\\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 732, in _read_query_result\\n result.read()\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 1075, in read\\n    first_packet = self.connection._read_packet()\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 684, in _read_packet\\n    packet.check_error()\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/protocol.py", line 220, in check_error\\n    err.raise_mysql_exception(self._data)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/err.py", line 109, in raise_mysql_exception\\n    raise errorclass(errno, errval)\\n\', "pymysql.err.OperationalError: (1213, \'Deadlock found when trying to get lock; try restarting transaction\')\\n", \'\\nThe above exception was the direct cause of the following exception:\\n\\n\', \'Traceback (most recent call last):\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/conductor/manager.py", line 135, in _object_dispatch\\n    return getattr(target, method)(*args, **kwargs)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper\\n return fn(self, *args, **kwargs)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py", line 801, in save\\n    self._extra_values_to_save)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/api.py", line 979, in instance_extra_update_by_uuid\\n    updates)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 224, in wrapped\\n    return f(context, *args, **kwargs)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 3072, in instance_extra_update_by_uuid\\n    update(values)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3835, in update\\n    update_op.exec_()\\n\', \' File "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1694, in exec_\\n    self._do_exec()\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-            packages/sqlalchemy/orm/persistence.py", line 1887, in _do_exec\\n    self._execute_stmt(update_stmt)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line  1699, in _execute_stmt\\n    self.result = self.query._execute_crud(stmt, self.mapper)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3367, in _execute_crud\\n    return conn.execute(stmt, self._params)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute\\n    return meth(self,  multiparams, params)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection\\n    return connection._execute_clauseelement(self, multiparams, params)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement\\n    distilled_params,\\n\', \'  File  "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1253, in _execute_context\\n    e, statement, parameters, cursor, context\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1471, in _handle_dbapi_exception\\n    util.raise_from_cause(newraise, exc_info)\\n\', \'  File  "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause\\n    reraise(type(exception), exception, tb=exc_tb, cause=cause)\\n\', \'  File  "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 152, in reraise\\n    raise value.with_traceback(tb)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site- packages/sqlalchemy/engine/base.py", line 1249, in _execute_context\\n    cursor, statement, parameters, context\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute\\n    cursor.execute(statement, parameters)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 170, in execute\\n    result = self._query(query)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 328, in _query\\n    conn.query(q)\\n\', \'  File  "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 517, in query\\n    self._affected_rows = self._read_query_result(unbuffered=unbuffered)\\n\', \'  File           "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 732, in _read_query_result\\n    result.read()\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 1075, in read\\n    first_packet = self.connection._read_packet()\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 684, in _read_packet\\n    packet.check_error()\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/protocol.py", line 220, in check_error\\n err.raise_mysql_exception(self._data)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/err.py", line 109, in raise_mysql_exception\\n    raise errorclass(errno, errval)\\n\',\'oslo_db.exception.DBDeadlock: (pymysql.err.OperationalError) (1213, \\\'Deadlock found when trying to get lock; try restarting transaction\\\')\\n[SQL: UPDATE instance_extra SET updated_at=%(updated_at)s, numa_topology=%(numa_topology)s, pci_requests=%(pci_requests)s, flavor=%(flavor)s, vcpu_model=%(vcpu_model)s WHERE instance_extra.deleted = %(deleted_1)s AND instance_extra.instance_uuid = %(instance_uuid_1)s]\\n[parameters: {\\\'updated_at\\\': datetime.datetime(2025, 5, 15, 5, 10, 15, 769294), \\\'numa_topology\\\': \\\'{"nova_object.name": "InstanceNUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.3", "nova_object.data": {"cells": [{"nova_obj ... (731 characters truncated) ... set_reserved", "id", "cpu_pinning_raw", "pagesize"]}], "emulator_threads_policy": null}, "nova_object.changes": ["cells", "emulator_threads_policy"]}\\\', \\\'pci_requests\\\': \\\'[{"count": 1, "spec": [{"vendor_id":  "10de", "product_id": "1e07", "dev_type": "type-PCI"}], "alias_name": "RTX_2080ti_revA", "is_new": false, "numa_ ... (538 characters truncated) ... "1ad7", "dev_type": "type-PCI"}], "alias_name":  "RTX_2080tiex2", "is_new": false, "numa_policy": "legacy", "request_id": null, "requester_id": null}]\\\', \\\'flavor\\\': \\\'{"cur": {"nova_object.name": "Flavor", "nova_object.namespace": "nova",  "nova_object.version": "1.2", "nova_object.data": {"id": 651, "name": "ux4-g1 ... (547 characters truncated) ... "2025-02-18T20:01:16Z", "updated_at": null, "deleted_at": null, "deleted": false},  "nova_object.changes": ["extra_specs"]}, "old": null, "new": null}\\\', \\\'vcpu_model\\\': \\\'{"nova_object.name": "VirtCPUModel", "nova_object.namespace": "nova", "nova_object.version": "1.0",  "nova_object.data": {"arch": null, "vendor": null ... (462 characters truncated) ... ": "host-passthrough", "model": null, "match": "exact"}, "nova_object.changes": ["features", "topology", "match", "model", "mode", "arch", "vendor"]}\\\', \\\'deleted_1\\\': 0, \\\'instance_uuid_1\\\': \\\'42288ebb-279c-483e-ad00-993ed453f823\\\'}]\\n(Background on this error at: http://sqlalche.me/e/e3q8)\\n\'].\n'}


ENGINE INNODB STATUS:

------------------------
LATEST DETECTED DEADLOCK
------------------------
2025-05-15 13:02:46 0x7fda0b6c7700
*** (1) TRANSACTION:
TRANSACTION 33097244744, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 1136, 2 row lock(s)
MySQL thread id 56380264, OS thread handle 140573857470208, query id 3909877498 10.9.64.13 nova_v1 Updating
UPDATE instance_extra SET updated_at='2025-05-15 13:02:46.460570', numa_topology='{\"nova_object.name\": \"InstanceNUMATopology\", \"nova_object.namespace\": \"nova\", \"nova_object.version\": \"1.3\", \"nova_object.data\": {\"cells\": [{\"nova
_object.name\": \"InstanceNUMACell\", \"nova_object.namespace\": \"nova\", \"nova_object.version\": \"1.4\", \"nova_object.data\": {\"id\": 0, \"cpuset\": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24
, 25, 26, 27, 28, 29, 30, 31], \"memory\": 122880, \"pagesize\": null, \"cpu_topology\": {\"nova_object.name\": \"VirtCPUTopology\", \"nova_object.namespace\": \"nova\", \"nova_object.version\": \"1.0\", \"nova_object.data\": {\"sockets\": 1, \
"cores\": 16, \"threads\": 2}, \"nova_object.changes\": [\"cores\", \"sockets\", \"threads\"]}, \"cpu_pinning_raw\": {\"0\": 40, \"1\": 88, \"2\": 18, \"3\": 66, \"4\": 42, \"5\": 90,
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1178 page no 51132 n bits 72 index PRIMARY of table `nova`.`instance_extra` trx id 33097244744 lock_mode X locks rec but not gap waiting
Record lock, heap no 2 PHYSICAL RECORD: n_fields 18; compact format; info bits 0
...

*** (2) TRANSACTION:
TRANSACTION 33097244739, ACTIVE 0 sec fetching rows
mysql tables in use 3, locked 3
4450 lock struct(s), heap size 532688, 11991 row lock(s)
MySQL thread id 56380887, OS thread handle 140574471255808, query id 3909877431 10.9.64.13 nova_v1 Sending data
DELETE FROM instance_extra WHERE instance_extra.id in (SELECT T1.id FROM (SELECT instance_extra.created_at, instance_extra.updated_at, instance_extra.deleted_at, instance_extra.deleted, instance_extra.id, instance_extra.instance_uuid, instance_
extra.device_metadata, instance_extra.numa_topology, instance_extra.pci_requests, instance_extra.flavor, instance_extra.vcpu_model, instance_extra.migration_context, instance_extra.keypairs, instance_extra.trusted_certs, instance_extra.resource
s
FROM instance_extra, instances
WHERE instances.deleted != 0 AND instances.uuid = instance_extra.instance_uuid ORDER BY instance_extra.id
 LIMIT 200) as T1)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 1178 page no 51132 n bits 72 index PRIMARY of table `nova`.`instance_extra` trx id 33097244739 lock mode S


The deadlocks happen in nova.instance_extra and triggered by colliding "nova-manage db archive_deleted_rows" and "instance_extra_update_by_uuid" function from nova.db.main.api which does not have any retry logic implemented

** Affects: nova
     Importance: Undecided
     Assignee: Stanislav Dmitriev (sdmitriev1)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => Stanislav Dmitriev (sdmitriev1)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2111108

Title:
  DBDeadlock in nova.instance_extra

Status in OpenStack Compute (nova):
  New

Bug description:
  Our environment has a high amount of ephemeral instances (spawn -> do
  a task -> deleted) therefore to keep the Nova DB clean we run
  periodically "nova-manage db archive_deleted_rows" to archive and
  purge old DB entries, even with a small "--max_rows" it causes
  occasional DB deadlocks and leave us with instances stuck forever with
  BUILD status and "scheduling" task state.

  The nova-conductor error related to such instances:
  {'code': 500, 'created': '2025-05-15T05:10:15Z', 'message': 'RemoteError', 'details': 'Traceback (most recent call last):\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 207, in decorated_function\n    return function(self, context, *args, **kwargs)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 2145, in _do_build_and_run_instance\n    (task_states.SCHEDULING, None))\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_versionedobjects/base.py", line 210, in wrapper\n    ctxt, self, fn.__name__, args, kwargs)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/conductor/rpcapi.py", line 246, in object_action\n    objmethod=objmethod, args=args, kwargs=kwargs)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 181, in call\n    transport_options=self.transport_options)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/transport.py", line 129, in _send\n    transport_options=transport_options)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 674, in send\n    transport_options=transport_options)\n  File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 664, in _send\n    raise result\noslo_messaging.rpc.client.RemoteError: Remote error: DBDeadlock (pymysql.err.OperationalError) (1213, \'Deadlock found when trying to get lock; try restarting transaction\')\n[SQL: UPDATE instance_extra SET updated_at=%(updated_at)s, numa_topology=%(numa_topology)s, pci_requests=%(pci_requests)s, flavor=%(flavor)s, vcpu_model=%(vcpu_model)s WHERE instance_extra.deleted = %(deleted_1)s AND instance_extra.instance_uuid = %(instance_uuid_1)s]\n[parameters: {\'updated_at\': datetime.datetime(2025, 5, 15, 5, 10, 15, 769294), \'numa_topology\': \'{"nova_object.name":                      "InstanceNUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.3", "nova_object.data": {"cells": [{"nova_obj ... (731 characters truncated) ... set_reserved", "id", "cpu_pinning_raw", "pagesize"]}], "emulator_threads_policy": null}, "nova_object.changes": ["cells", "emulator_threads_policy"]}\', \'pci_requests\': \'[{"count": 1, "spec": [{"vendor_id": "10de", "product_id": "1e07", "dev_type": "type-PCI"}], "alias_name": "RTX_2080ti_revA", "is_new": false, "numa_ ... (538 characters truncated) ... "1ad7", "dev_type": "type-PCI"}], "alias_name": "RTX_2080tiex2", "is_new": false, "numa_policy": "legacy", "request_id": null, "requester_id": null}]\', \'flavor\': \'{"cur": {"nova_object.name": "Flavor", "nova_object.namespace": "nova",    "nova_object.version": "1.2", "nova_object.data": {"id": 651, "name": "ux4-g1 ... (547 characters truncated) ... "2025-02-18T20:01:16Z", "updated_at": null, "deleted_at": null, "deleted": false}, "nova_object.changes": ["extra_specs"]}, "old": null, "new": null}\', \'vcpu_model\': \'{"nova_object.name": "VirtCPUModel", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"arch": null, "vendor": null ... (462 characters truncated) ... ": "host-passthrough", "model": null, "match": "exact"}, "nova_object.changes": ["features", "topology", "match", "model", "mode", "arch", "vendor"]}\', \'deleted_1\': 0, \'instance_uuid_1\': \'42288ebb-279c-483e-ad00-993ed453f823\'}]\n(Background on this error at: http://sqlalche.me/e/e3q8)\n[\'Traceback (most recent call last):\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context\\n    cursor, statement, parameters, context\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute\\n    cursor.execute(statement, parameters)\\n\', \'  File  "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 170, in execute\\n    result = self._query(query)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 328, in _query\\n    conn.query(q)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 517, in query\\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 732, in _read_query_result\\n result.read()\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 1075, in read\\n    first_packet = self.connection._read_packet()\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 684, in _read_packet\\n    packet.check_error()\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/protocol.py", line 220, in check_error\\n    err.raise_mysql_exception(self._data)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/err.py", line 109, in raise_mysql_exception\\n    raise errorclass(errno, errval)\\n\', "pymysql.err.OperationalError: (1213, \'Deadlock found when trying to get lock; try restarting transaction\')\\n", \'\\nThe above exception was the direct cause of the following exception:\\n\\n\', \'Traceback (most recent call last):\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/conductor/manager.py", line 135, in _object_dispatch\\n    return getattr(target, method)(*args, **kwargs)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper\\n return fn(self, *args, **kwargs)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py", line 801, in save\\n    self._extra_values_to_save)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/api.py", line 979, in instance_extra_update_by_uuid\\n    updates)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 224, in wrapped\\n    return f(context, *args, **kwargs)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 3072, in instance_extra_update_by_uuid\\n    update(values)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3835, in update\\n    update_op.exec_()\\n\', \' File "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1694, in exec_\\n    self._do_exec()\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-            packages/sqlalchemy/orm/persistence.py", line 1887, in _do_exec\\n    self._execute_stmt(update_stmt)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line  1699, in _execute_stmt\\n    self.result = self.query._execute_crud(stmt, self.mapper)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3367, in _execute_crud\\n    return conn.execute(stmt, self._params)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute\\n    return meth(self,  multiparams, params)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection\\n    return connection._execute_clauseelement(self, multiparams, params)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement\\n    distilled_params,\\n\', \'  File  "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1253, in _execute_context\\n    e, statement, parameters, cursor, context\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1471, in _handle_dbapi_exception\\n    util.raise_from_cause(newraise, exc_info)\\n\', \'  File  "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause\\n    reraise(type(exception), exception, tb=exc_tb, cause=cause)\\n\', \'  File  "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 152, in reraise\\n    raise value.with_traceback(tb)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site- packages/sqlalchemy/engine/base.py", line 1249, in _execute_context\\n    cursor, statement, parameters, context\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute\\n    cursor.execute(statement, parameters)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 170, in execute\\n    result = self._query(query)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 328, in _query\\n    conn.query(q)\\n\', \'  File  "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 517, in query\\n    self._affected_rows = self._read_query_result(unbuffered=unbuffered)\\n\', \'  File           "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 732, in _read_query_result\\n    result.read()\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 1075, in read\\n    first_packet = self.connection._read_packet()\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 684, in _read_packet\\n    packet.check_error()\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/protocol.py", line 220, in check_error\\n err.raise_mysql_exception(self._data)\\n\', \'  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/err.py", line 109, in raise_mysql_exception\\n    raise errorclass(errno, errval)\\n\',\'oslo_db.exception.DBDeadlock: (pymysql.err.OperationalError) (1213, \\\'Deadlock found when trying to get lock; try restarting transaction\\\')\\n[SQL: UPDATE instance_extra SET updated_at=%(updated_at)s, numa_topology=%(numa_topology)s, pci_requests=%(pci_requests)s, flavor=%(flavor)s, vcpu_model=%(vcpu_model)s WHERE instance_extra.deleted = %(deleted_1)s AND instance_extra.instance_uuid = %(instance_uuid_1)s]\\n[parameters: {\\\'updated_at\\\': datetime.datetime(2025, 5, 15, 5, 10, 15, 769294), \\\'numa_topology\\\': \\\'{"nova_object.name": "InstanceNUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.3", "nova_object.data": {"cells": [{"nova_obj ... (731 characters truncated) ... set_reserved", "id", "cpu_pinning_raw", "pagesize"]}], "emulator_threads_policy": null}, "nova_object.changes": ["cells", "emulator_threads_policy"]}\\\', \\\'pci_requests\\\': \\\'[{"count": 1, "spec": [{"vendor_id":  "10de", "product_id": "1e07", "dev_type": "type-PCI"}], "alias_name": "RTX_2080ti_revA", "is_new": false, "numa_ ... (538 characters truncated) ... "1ad7", "dev_type": "type-PCI"}], "alias_name":  "RTX_2080tiex2", "is_new": false, "numa_policy": "legacy", "request_id": null, "requester_id": null}]\\\', \\\'flavor\\\': \\\'{"cur": {"nova_object.name": "Flavor", "nova_object.namespace": "nova",  "nova_object.version": "1.2", "nova_object.data": {"id": 651, "name": "ux4-g1 ... (547 characters truncated) ... "2025-02-18T20:01:16Z", "updated_at": null, "deleted_at": null, "deleted": false},  "nova_object.changes": ["extra_specs"]}, "old": null, "new": null}\\\', \\\'vcpu_model\\\': \\\'{"nova_object.name": "VirtCPUModel", "nova_object.namespace": "nova", "nova_object.version": "1.0",  "nova_object.data": {"arch": null, "vendor": null ... (462 characters truncated) ... ": "host-passthrough", "model": null, "match": "exact"}, "nova_object.changes": ["features", "topology", "match", "model", "mode", "arch", "vendor"]}\\\', \\\'deleted_1\\\': 0, \\\'instance_uuid_1\\\': \\\'42288ebb-279c-483e-ad00-993ed453f823\\\'}]\\n(Background on this error at: http://sqlalche.me/e/e3q8)\\n\'].\n'}

  
  ENGINE INNODB STATUS:

  ------------------------
  LATEST DETECTED DEADLOCK
  ------------------------
  2025-05-15 13:02:46 0x7fda0b6c7700
  *** (1) TRANSACTION:
  TRANSACTION 33097244744, ACTIVE 0 sec starting index read
  mysql tables in use 1, locked 1
  LOCK WAIT 3 lock struct(s), heap size 1136, 2 row lock(s)
  MySQL thread id 56380264, OS thread handle 140573857470208, query id 3909877498 10.9.64.13 nova_v1 Updating
  UPDATE instance_extra SET updated_at='2025-05-15 13:02:46.460570', numa_topology='{\"nova_object.name\": \"InstanceNUMATopology\", \"nova_object.namespace\": \"nova\", \"nova_object.version\": \"1.3\", \"nova_object.data\": {\"cells\": [{\"nova
  _object.name\": \"InstanceNUMACell\", \"nova_object.namespace\": \"nova\", \"nova_object.version\": \"1.4\", \"nova_object.data\": {\"id\": 0, \"cpuset\": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24
  , 25, 26, 27, 28, 29, 30, 31], \"memory\": 122880, \"pagesize\": null, \"cpu_topology\": {\"nova_object.name\": \"VirtCPUTopology\", \"nova_object.namespace\": \"nova\", \"nova_object.version\": \"1.0\", \"nova_object.data\": {\"sockets\": 1, \
  "cores\": 16, \"threads\": 2}, \"nova_object.changes\": [\"cores\", \"sockets\", \"threads\"]}, \"cpu_pinning_raw\": {\"0\": 40, \"1\": 88, \"2\": 18, \"3\": 66, \"4\": 42, \"5\": 90,
  *** (1) WAITING FOR THIS LOCK TO BE GRANTED:
  RECORD LOCKS space id 1178 page no 51132 n bits 72 index PRIMARY of table `nova`.`instance_extra` trx id 33097244744 lock_mode X locks rec but not gap waiting
  Record lock, heap no 2 PHYSICAL RECORD: n_fields 18; compact format; info bits 0
  ...

  *** (2) TRANSACTION:
  TRANSACTION 33097244739, ACTIVE 0 sec fetching rows
  mysql tables in use 3, locked 3
  4450 lock struct(s), heap size 532688, 11991 row lock(s)
  MySQL thread id 56380887, OS thread handle 140574471255808, query id 3909877431 10.9.64.13 nova_v1 Sending data
  DELETE FROM instance_extra WHERE instance_extra.id in (SELECT T1.id FROM (SELECT instance_extra.created_at, instance_extra.updated_at, instance_extra.deleted_at, instance_extra.deleted, instance_extra.id, instance_extra.instance_uuid, instance_
  extra.device_metadata, instance_extra.numa_topology, instance_extra.pci_requests, instance_extra.flavor, instance_extra.vcpu_model, instance_extra.migration_context, instance_extra.keypairs, instance_extra.trusted_certs, instance_extra.resource
  s
  FROM instance_extra, instances
  WHERE instances.deleted != 0 AND instances.uuid = instance_extra.instance_uuid ORDER BY instance_extra.id
   LIMIT 200) as T1)
  *** (2) HOLDS THE LOCK(S):
  RECORD LOCKS space id 1178 page no 51132 n bits 72 index PRIMARY of table `nova`.`instance_extra` trx id 33097244739 lock mode S

  
  The deadlocks happen in nova.instance_extra and triggered by colliding "nova-manage db archive_deleted_rows" and "instance_extra_update_by_uuid" function from nova.db.main.api which does not have any retry logic implemented

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2111108/+subscriptions