yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1944787] [NEW] Failed storing info cache: ... DBReferenceError (pymysql.err.IntegrityError) ... foreign key constraint fails (`nova_cell1`.`instance_info_caches` ...

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: melanie witt <1944787@xxxxxxxxxxxxxxxxxx>
Date: Fri, 24 Sep 2021 01:39:20 -0000
Reply-to: Bug 1944787 <1944787@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Noticed in gate runs in the nova-multi-cell job, tracebacks like the
following during cross cell resize:

Sep 23 06:10:18.393560 ubuntu-focal-rax-iad-0026631025 nova-
compute[110797]: DEBUG nova.compute.manager [None
req-782bdf2c-3bee-47ab-8e75-d9c5438638f0 tempest-
ServerActionsTestJSON-1400567331 tempest-
ServerActionsTestJSON-1400567331-project] [instance:
e9254211-66b3-422d-a486-9c7dc92d0d2f] Checking if we can cross-cell
migrate instance to this host (ubuntu-focal-rax-iad-0026631025).
{{(pid=110797) prep_snapshot_based_resize_at_dest
/opt/stack/nova/nova/compute/manager.py:5397}}

[...]

Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [req-da56c934-c427-4c21-bfa5-b26da09b4687 req-08c8d69d-3330-4a23-9ce0-f399d9f61ab0 service nova] [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] Failed storing info cache: oslo_messaging.rpc.client.RemoteError: Remote error: DBReferenceError (pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT `instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))')
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: [SQL: INSERT INTO instance_info_caches (created_at, updated_at, deleted_at, deleted, network_info, instance_uuid) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(network_info)s, %(instance_uuid)s)]
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: [parameters: {'created_at': datetime.datetime(2021, 9, 23, 6, 10, 45, 736541), 'updated_at': None, 'deleted_at': None, 'deleted': 0, 'network_info': '[{"id": "6cda3ae0-0087-4c9d-9aa4-307aaac11229", "address": "fa:16:3e:42:88:b5", "network": {"id": "15d8321e-4160-46c0-aacc-fdfa0e57061e", "bridge": " ... (720 characters truncated) ... : null, "qbg_params": null, "active": false, "vnic_type": "normal", "profile": {}, "preserve_on_delete": false, "delegate_create": true, "meta": {}}]', 'instance_uuid': 'e9254211-66b3-422d-a486-9c7dc92d0d2f'}]
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: (Background on this error at: https://sqlalche.me/e/14/gkpj)
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ['Traceback (most recent call last):\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1771, in _execute_context\n    self.dialect.do_execute(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 717, in do_execute\n    cursor.execute(statement, parameters)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute\n    result = self._query(query)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query\n    conn.query(q)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query\n    self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result\n    result.read()\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read\n    first_packet = self.connection._read_packet()\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet\n    packet.raise_for_error()\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error\n    err.raise_mysql_exception(self._data)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception\n    raise errorclass(errno, errval)\n', "pymysql.err.IntegrityError: (1452, 'Cannot add or update a child row: a foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT `instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))')\n", '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', '  File "/opt/stack/nova/nova/conductor/manager.py", line 139, in _object_dispatch\n    return getattr(target, method)(*args, **kwargs)\n', '  File "/usr/local/lib/python3.8/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper\n    return fn(self, *args, **kwargs)\n', '  File "/opt/stack/nova/nova/objects/instance_info_cache.py", line 78, in save\n    rv = db.instance_info_cache_update(self._context,\n', '  File "/opt/stack/nova/nova/db/utils.py", line 35, in wrapper\n    return f(*args, **kwargs)\n', '  File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 154, in wrapper\n    ectxt.value = e.inner_exc\n', '  File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__\n    self.force_reraise()\n', '  File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise\n    raise self.value\n', '  File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 142, in wrapper\n    return f(*args, **kwargs)\n', '  File "/opt/stack/nova/nova/db/main/api.py", line 190, in wrapper\n    return f(context, *args, **kwargs)\n', '  File "/opt/stack/nova/nova/db/main/api.py", line 2447, in instance_info_cache_update\n    info_cache.save(context.session)\n', '  File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/models.py", line 48, in save\n    session.flush()\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3298, in flush\n    self._flush(objects)\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3438, in _flush\n    transaction.rollback(_capture_exception=True)\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n    compat.raise_(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_\n    raise exception\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3398, in _flush\n    flush_context.execute()\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n    rec.execute(self)\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n    util.preloaded.orm_persistence.save_obj(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 242, in save_obj\n    _emit_insert_statements(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1219, in _emit_insert_statements\n    result = connection._execute_20(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1583, in _execute_20\n    return meth(self, args_10style, kwargs_10style, execution_options)\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection\n    return connection._execute_clauseelement(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1452, in _execute_clauseelement\n    ret = self._execute_context(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1814, in _execute_context\n    self._handle_dbapi_exception(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1993, in _handle_dbapi_exception\n    util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_\n    raise exception\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1771, in _execute_context\n    self.dialect.do_execute(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 717, in do_execute\n    cursor.execute(statement, parameters)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute\n    result = self._query(query)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query\n    conn.query(q)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query\n    self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result\n    result.read()\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read\n    first_packet = self.connection._read_packet()\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet\n    packet.raise_for_error()\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error\n    err.raise_mysql_exception(self._data)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception\n    raise errorclass(errno, errval)\n', 'oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1452, \'Cannot add or update a child row: a foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT `instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))\')\n[SQL: INSERT INTO instance_info_caches (created_at, updated_at, deleted_at, deleted, network_info, instance_uuid) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(network_info)s, %(instance_uuid)s)]\n[parameters: {\'created_at\': datetime.datetime(2021, 9, 23, 6, 10, 45, 736541), \'updated_at\': None, \'deleted_at\': None, \'deleted\': 0, \'network_info\': \'[{"id": "6cda3ae0-0087-4c9d-9aa4-307aaac11229", "address": "fa:16:3e:42:88:b5", "network": {"id": "15d8321e-4160-46c0-aacc-fdfa0e57061e", "bridge": " ... (720 characters truncated) ... : null, "qbg_params": null, "active": false, "vnic_type": "normal", "profile": {}, "preserve_on_delete": false, "delegate_create": true, "meta": {}}]\', \'instance_uuid\': \'e9254211-66b3-422d-a486-9c7dc92d0d2f\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n'].
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] Traceback (most recent call last):
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]   File "/opt/stack/nova/nova/network/neutron.py", line 125, in update_instance_cache_with_nw_info
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]     ic.save()
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]   File "/usr/local/lib/python3.8/dist-packages/oslo_versionedobjects/base.py", line 209, in wrapper
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]     updates, result = self.indirection_api.object_action(
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]   File "/opt/stack/nova/nova/conductor/rpcapi.py", line 247, in object_action
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]     return cctxt.call(context, 'object_action', objinst=objinst,
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/client.py", line 189, in call
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]     result = self.transport._send(
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/transport.py", line 123, in _send
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]     return self._driver.send(target, ctxt, message,
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 680, in send
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]     return self._send(target, ctxt, message, wait_for_reply, timeout,
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 672, in _send
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]     raise result
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] oslo_messaging.rpc.client.RemoteError: Remote error: DBReferenceError (pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT `instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))')
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] [SQL: INSERT INTO instance_info_caches (created_at, updated_at, deleted_at, deleted, network_info, instance_uuid) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(network_info)s, %(instance_uuid)s)]
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] [parameters: {'created_at': datetime.datetime(2021, 9, 23, 6, 10, 45, 736541), 'updated_at': None, 'deleted_at': None, 'deleted': 0, 'network_info': '[{"id": "6cda3ae0-0087-4c9d-9aa4-307aaac11229", "address": "fa:16:3e:42:88:b5", "network": {"id": "15d8321e-4160-46c0-aacc-fdfa0e57061e", "bridge": " ... (720 characters truncated) ... : null, "qbg_params": null, "active": false, "vnic_type": "normal", "profile": {}, "preserve_on_delete": false, "delegate_create": true, "meta": {}}]', 'instance_uuid': 'e9254211-66b3-422d-a486-9c7dc92d0d2f'}]
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] (Background on this error at: https://sqlalche.me/e/14/gkpj)
Sep 23 06:10:45.747562 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] ['Traceback (most recent call last):\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1771, in _execute_context\n    self.dialect.do_execute(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 717, in do_execute\n    cursor.execute(statement, parameters)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute\n    result = self._query(query)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query\n    conn.query(q)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query\n    self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result\n    result.read()\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read\n    first_packet = self.connection._read_packet()\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet\n    packet.raise_for_error()\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error\n    err.raise_mysql_exception(self._data)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception\n    raise errorclass(errno, errval)\n', "pymysql.err.IntegrityError: (1452, 'Cannot add or update a child row: a foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT `instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))')\n", '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', '  File "/opt/stack/nova/nova/conductor/manager.py", line 139, in _object_dispatch\n    return getattr(target, method)(*args, **kwargs)\n', '  File "/usr/local/lib/python3.8/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper\n    return fn(self, *args, **kwargs)\n', '  File "/opt/stack/nova/nova/objects/instance_info_cache.py", line 78, in save\n    rv = db.instance_info_cache_update(self._context,\n', '  File "/opt/stack/nova/nova/db/utils.py", line 35, in wrapper\n    return f(*args, **kwargs)\n', '  File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 154, in wrapper\n    ectxt.value = e.inner_exc\n', '  File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__\n    self.force_reraise()\n', '  File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise\n    raise self.value\n', '  File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 142, in wrapper\n    return f(*args, **kwargs)\n', '  File "/opt/stack/nova/nova/db/main/api.py", line 190, in wrapper\n    return f(context, *args, **kwargs)\n', '  File "/opt/stack/nova/nova/db/main/api.py", line 2447, in instance_info_cache_update\n    info_cache.save(context.session)\n', '  File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/models.py", line 48, in save\n    session.flush()\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3298, in flush\n    self._flush(objects)\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3438, in _flush\n    transaction.rollback(_capture_exception=True)\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n    compat.raise_(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_\n    raise exception\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3398, in _flush\n    flush_context.execute()\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n    rec.execute(self)\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n    util.preloaded.orm_persistence.save_obj(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 242, in save_obj\n    _emit_insert_statements(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1219, in _emit_insert_statements\n    result = connection._execute_20(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1583, in _execute_20\n    return meth(self, args_10style, kwargs_10style, execution_options)\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection\n    return connection._execute_clauseelement(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1452, in _execute_clauseelement\n    ret = self._execute_context(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1814, in _execute_context\n    self._handle_dbapi_exception(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1993, in _handle_dbapi_exception\n    util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_\n    raise exception\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1771, in _execute_context\n    self.dialect.do_execute(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 717, in do_execute\n    cursor.execute(statement, parameters)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute\n    result = self._query(query)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query\n    conn.query(q)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query\n    self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result\n    result.read()\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read\n    first_packet = self.connection._read_packet()\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet\n    packet.raise_for_error()\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error\n    err.raise_mysql_exception(self._data)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception\n    raise errorclass(errno, errval)\n', 'oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1452, \'Cannot add or update a child row: a foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT `instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))\')\n[SQL: INSERT INTO instance_info_caches (created_at, updated_at, deleted_at, deleted, network_info, instance_uuid) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(network_info)s, %(instance_uuid)s)]\n[parameters: {\'created_at\': datetime.datetime(2021, 9, 23, 6, 10, 45, 736541), \'updated_at\': None, \'deleted_at\': None, \'deleted\': 0, \'network_info\': \'[{"id": "6cda3ae0-0087-4c9d-9aa4-307aaac11229", "address": "fa:16:3e:42:88:b5", "network": {"id": "15d8321e-4160-46c0-aacc-fdfa0e57061e", "bridge": " ... (720 characters truncated) ... : null, "qbg_params": null, "active": false, "vnic_type": "normal", "profile": {}, "preserve_on_delete": false, "delegate_create": true, "meta": {}}]\', \'instance_uuid\': \'e9254211-66b3-422d-a486-9c7dc92d0d2f\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n'].
Sep 23 06:10:45.747562 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]


A cursory look through the code it looks like this is an external neutron event coming in after the instance has moved to another cell, so it is not found in the expected cell that was looked up initially when the external event was received.

This doesn't necessarily cause a job failure, so it appears mostly be a
traceback we need to clean up. Maybe we could DEBUG or INFO log that we
possibly received an event for an instance that has moved across cells.

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: cells

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1944787

Title:
  Failed storing info cache: ... DBReferenceError
  (pymysql.err.IntegrityError) ... foreign key constraint fails
  (`nova_cell1`.`instance_info_caches` ...

Status in OpenStack Compute (nova):
  New

Bug description:
  Noticed in gate runs in the nova-multi-cell job, tracebacks like the
  following during cross cell resize:

  Sep 23 06:10:18.393560 ubuntu-focal-rax-iad-0026631025 nova-
  compute[110797]: DEBUG nova.compute.manager [None
  req-782bdf2c-3bee-47ab-8e75-d9c5438638f0 tempest-
  ServerActionsTestJSON-1400567331 tempest-
  ServerActionsTestJSON-1400567331-project] [instance:
  e9254211-66b3-422d-a486-9c7dc92d0d2f] Checking if we can cross-cell
  migrate instance to this host (ubuntu-focal-rax-iad-0026631025).
  {{(pid=110797) prep_snapshot_based_resize_at_dest
  /opt/stack/nova/nova/compute/manager.py:5397}}

  [...]

  Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [req-da56c934-c427-4c21-bfa5-b26da09b4687 req-08c8d69d-3330-4a23-9ce0-f399d9f61ab0 service nova] [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] Failed storing info cache: oslo_messaging.rpc.client.RemoteError: Remote error: DBReferenceError (pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT `instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))')
  Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: [SQL: INSERT INTO instance_info_caches (created_at, updated_at, deleted_at, deleted, network_info, instance_uuid) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(network_info)s, %(instance_uuid)s)]
  Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: [parameters: {'created_at': datetime.datetime(2021, 9, 23, 6, 10, 45, 736541), 'updated_at': None, 'deleted_at': None, 'deleted': 0, 'network_info': '[{"id": "6cda3ae0-0087-4c9d-9aa4-307aaac11229", "address": "fa:16:3e:42:88:b5", "network": {"id": "15d8321e-4160-46c0-aacc-fdfa0e57061e", "bridge": " ... (720 characters truncated) ... : null, "qbg_params": null, "active": false, "vnic_type": "normal", "profile": {}, "preserve_on_delete": false, "delegate_create": true, "meta": {}}]', 'instance_uuid': 'e9254211-66b3-422d-a486-9c7dc92d0d2f'}]
  Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: (Background on this error at: https://sqlalche.me/e/14/gkpj)
  Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ['Traceback (most recent call last):\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1771, in _execute_context\n    self.dialect.do_execute(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 717, in do_execute\n    cursor.execute(statement, parameters)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute\n    result = self._query(query)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query\n    conn.query(q)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query\n    self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result\n    result.read()\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read\n    first_packet = self.connection._read_packet()\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet\n    packet.raise_for_error()\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error\n    err.raise_mysql_exception(self._data)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception\n    raise errorclass(errno, errval)\n', "pymysql.err.IntegrityError: (1452, 'Cannot add or update a child row: a foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT `instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))')\n", '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', '  File "/opt/stack/nova/nova/conductor/manager.py", line 139, in _object_dispatch\n    return getattr(target, method)(*args, **kwargs)\n', '  File "/usr/local/lib/python3.8/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper\n    return fn(self, *args, **kwargs)\n', '  File "/opt/stack/nova/nova/objects/instance_info_cache.py", line 78, in save\n    rv = db.instance_info_cache_update(self._context,\n', '  File "/opt/stack/nova/nova/db/utils.py", line 35, in wrapper\n    return f(*args, **kwargs)\n', '  File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 154, in wrapper\n    ectxt.value = e.inner_exc\n', '  File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__\n    self.force_reraise()\n', '  File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise\n    raise self.value\n', '  File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 142, in wrapper\n    return f(*args, **kwargs)\n', '  File "/opt/stack/nova/nova/db/main/api.py", line 190, in wrapper\n    return f(context, *args, **kwargs)\n', '  File "/opt/stack/nova/nova/db/main/api.py", line 2447, in instance_info_cache_update\n    info_cache.save(context.session)\n', '  File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/models.py", line 48, in save\n    session.flush()\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3298, in flush\n    self._flush(objects)\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3438, in _flush\n    transaction.rollback(_capture_exception=True)\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n    compat.raise_(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_\n    raise exception\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3398, in _flush\n    flush_context.execute()\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n    rec.execute(self)\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n    util.preloaded.orm_persistence.save_obj(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 242, in save_obj\n    _emit_insert_statements(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1219, in _emit_insert_statements\n    result = connection._execute_20(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1583, in _execute_20\n    return meth(self, args_10style, kwargs_10style, execution_options)\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection\n    return connection._execute_clauseelement(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1452, in _execute_clauseelement\n    ret = self._execute_context(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1814, in _execute_context\n    self._handle_dbapi_exception(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1993, in _handle_dbapi_exception\n    util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_\n    raise exception\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1771, in _execute_context\n    self.dialect.do_execute(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 717, in do_execute\n    cursor.execute(statement, parameters)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute\n    result = self._query(query)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query\n    conn.query(q)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query\n    self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result\n    result.read()\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read\n    first_packet = self.connection._read_packet()\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet\n    packet.raise_for_error()\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error\n    err.raise_mysql_exception(self._data)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception\n    raise errorclass(errno, errval)\n', 'oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1452, \'Cannot add or update a child row: a foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT `instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))\')\n[SQL: INSERT INTO instance_info_caches (created_at, updated_at, deleted_at, deleted, network_info, instance_uuid) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(network_info)s, %(instance_uuid)s)]\n[parameters: {\'created_at\': datetime.datetime(2021, 9, 23, 6, 10, 45, 736541), \'updated_at\': None, \'deleted_at\': None, \'deleted\': 0, \'network_info\': \'[{"id": "6cda3ae0-0087-4c9d-9aa4-307aaac11229", "address": "fa:16:3e:42:88:b5", "network": {"id": "15d8321e-4160-46c0-aacc-fdfa0e57061e", "bridge": " ... (720 characters truncated) ... : null, "qbg_params": null, "active": false, "vnic_type": "normal", "profile": {}, "preserve_on_delete": false, "delegate_create": true, "meta": {}}]\', \'instance_uuid\': \'e9254211-66b3-422d-a486-9c7dc92d0d2f\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n'].
  Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] Traceback (most recent call last):
  Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]   File "/opt/stack/nova/nova/network/neutron.py", line 125, in update_instance_cache_with_nw_info
  Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]     ic.save()
  Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]   File "/usr/local/lib/python3.8/dist-packages/oslo_versionedobjects/base.py", line 209, in wrapper
  Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]     updates, result = self.indirection_api.object_action(
  Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]   File "/opt/stack/nova/nova/conductor/rpcapi.py", line 247, in object_action
  Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]     return cctxt.call(context, 'object_action', objinst=objinst,
  Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/client.py", line 189, in call
  Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]     result = self.transport._send(
  Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/transport.py", line 123, in _send
  Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]     return self._driver.send(target, ctxt, message,
  Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 680, in send
  Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]     return self._send(target, ctxt, message, wait_for_reply, timeout,
  Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 672, in _send
  Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]     raise result
  Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] oslo_messaging.rpc.client.RemoteError: Remote error: DBReferenceError (pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT `instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))')
  Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] [SQL: INSERT INTO instance_info_caches (created_at, updated_at, deleted_at, deleted, network_info, instance_uuid) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(network_info)s, %(instance_uuid)s)]
  Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] [parameters: {'created_at': datetime.datetime(2021, 9, 23, 6, 10, 45, 736541), 'updated_at': None, 'deleted_at': None, 'deleted': 0, 'network_info': '[{"id": "6cda3ae0-0087-4c9d-9aa4-307aaac11229", "address": "fa:16:3e:42:88:b5", "network": {"id": "15d8321e-4160-46c0-aacc-fdfa0e57061e", "bridge": " ... (720 characters truncated) ... : null, "qbg_params": null, "active": false, "vnic_type": "normal", "profile": {}, "preserve_on_delete": false, "delegate_create": true, "meta": {}}]', 'instance_uuid': 'e9254211-66b3-422d-a486-9c7dc92d0d2f'}]
  Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] (Background on this error at: https://sqlalche.me/e/14/gkpj)
  Sep 23 06:10:45.747562 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] ['Traceback (most recent call last):\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1771, in _execute_context\n    self.dialect.do_execute(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 717, in do_execute\n    cursor.execute(statement, parameters)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute\n    result = self._query(query)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query\n    conn.query(q)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query\n    self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result\n    result.read()\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read\n    first_packet = self.connection._read_packet()\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet\n    packet.raise_for_error()\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error\n    err.raise_mysql_exception(self._data)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception\n    raise errorclass(errno, errval)\n', "pymysql.err.IntegrityError: (1452, 'Cannot add or update a child row: a foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT `instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))')\n", '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', '  File "/opt/stack/nova/nova/conductor/manager.py", line 139, in _object_dispatch\n    return getattr(target, method)(*args, **kwargs)\n', '  File "/usr/local/lib/python3.8/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper\n    return fn(self, *args, **kwargs)\n', '  File "/opt/stack/nova/nova/objects/instance_info_cache.py", line 78, in save\n    rv = db.instance_info_cache_update(self._context,\n', '  File "/opt/stack/nova/nova/db/utils.py", line 35, in wrapper\n    return f(*args, **kwargs)\n', '  File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 154, in wrapper\n    ectxt.value = e.inner_exc\n', '  File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__\n    self.force_reraise()\n', '  File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise\n    raise self.value\n', '  File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 142, in wrapper\n    return f(*args, **kwargs)\n', '  File "/opt/stack/nova/nova/db/main/api.py", line 190, in wrapper\n    return f(context, *args, **kwargs)\n', '  File "/opt/stack/nova/nova/db/main/api.py", line 2447, in instance_info_cache_update\n    info_cache.save(context.session)\n', '  File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/models.py", line 48, in save\n    session.flush()\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3298, in flush\n    self._flush(objects)\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3438, in _flush\n    transaction.rollback(_capture_exception=True)\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n    compat.raise_(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_\n    raise exception\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3398, in _flush\n    flush_context.execute()\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n    rec.execute(self)\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n    util.preloaded.orm_persistence.save_obj(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 242, in save_obj\n    _emit_insert_statements(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1219, in _emit_insert_statements\n    result = connection._execute_20(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1583, in _execute_20\n    return meth(self, args_10style, kwargs_10style, execution_options)\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection\n    return connection._execute_clauseelement(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1452, in _execute_clauseelement\n    ret = self._execute_context(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1814, in _execute_context\n    self._handle_dbapi_exception(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1993, in _handle_dbapi_exception\n    util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_\n    raise exception\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1771, in _execute_context\n    self.dialect.do_execute(\n', '  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 717, in do_execute\n    cursor.execute(statement, parameters)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute\n    result = self._query(query)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query\n    conn.query(q)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query\n    self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result\n    result.read()\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read\n    first_packet = self.connection._read_packet()\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet\n    packet.raise_for_error()\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error\n    err.raise_mysql_exception(self._data)\n', '  File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception\n    raise errorclass(errno, errval)\n', 'oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1452, \'Cannot add or update a child row: a foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT `instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))\')\n[SQL: INSERT INTO instance_info_caches (created_at, updated_at, deleted_at, deleted, network_info, instance_uuid) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(network_info)s, %(instance_uuid)s)]\n[parameters: {\'created_at\': datetime.datetime(2021, 9, 23, 6, 10, 45, 736541), \'updated_at\': None, \'deleted_at\': None, \'deleted\': 0, \'network_info\': \'[{"id": "6cda3ae0-0087-4c9d-9aa4-307aaac11229", "address": "fa:16:3e:42:88:b5", "network": {"id": "15d8321e-4160-46c0-aacc-fdfa0e57061e", "bridge": " ... (720 characters truncated) ... : null, "qbg_params": null, "active": false, "vnic_type": "normal", "profile": {}, "preserve_on_delete": false, "delegate_create": true, "meta": {}}]\', \'instance_uuid\': \'e9254211-66b3-422d-a486-9c7dc92d0d2f\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n'].
  Sep 23 06:10:45.747562 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]

  
  A cursory look through the code it looks like this is an external neutron event coming in after the instance has moved to another cell, so it is not found in the expected cell that was looked up initially when the external event was received.

  This doesn't necessarily cause a job failure, so it appears mostly be
  a traceback we need to clean up. Maybe we could DEBUG or INFO log that
  we possibly received an event for an instance that has moved across
  cells.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1944787/+subscriptions
Follow ups

[Bug 1944787] Re: Failed storing info cache: ... DBReferenceError (pymysql.err.IntegrityError) ... foreign key constraint fails (`nova_cell1`.`instance_info_caches` ...
From: OpenStack Infra, 2022-02-21