yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2079205] [NEW] Tempest test fails to delete an instance, MessagingTimeout in n-cpu log when processing "_complete_deletion"

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Ihar Hrachyshka <2079205@xxxxxxxxxxxxxxxxxx>
Date: Thu, 05 Sep 2024 16:54:06 -0000
Reply-to: Bug 2079205 <2079205@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Logs:
https://ae5a8daa4540c7da75ef-0b252c7b71d313ec10c19ced03a4a34b.ssl.cf1.rackcdn.com/927182/1/check/neutron-
tempest-plugin-openvswitch-
iptables_hybrid-2023-2/1047b7f/testr_results.html

Test traceback:

```
Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 87, in call_and_ignore_notfound_exc
    return func(*args, **kwargs)
  File "/opt/stack/tempest/tempest/common/waiters.py", line 136, in wait_for_server_termination
    raise lib_exc.DeleteErrorException(details, server_id=server_id)
tempest.lib.exceptions.DeleteErrorException: Resource %(resource_id)s failed to delete and is in ERROR status
Details: Server f9acc6d9-132d-4ed8-875a-48fedb5b81b9 failed to delete and is in ERROR status. Fault: {'code': 500, 'created': '2024-09-05T13:56:54Z', 'message': 'MessagingTimeout'}.
```

In n-cpu logs,
https://ae5a8daa4540c7da75ef-0b252c7b71d313ec10c19ced03a4a34b.ssl.cf1.rackcdn.com/927182/1/check/neutron-
tempest-plugin-openvswitch-
iptables_hybrid-2023-2/1047b7f/controller/logs/screen-n-cpu.txt:

```
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [None req-99b4aa7a-c470-4313-807c-d22d46b0ae38 tempest-MulticastTestIPv4-1796741707 tempest-MulticastTestIPv4-1796741707-project-member] [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9] Setting instance vm_state to ERROR: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 1bb84ce3208146ae828a98299bf658ea
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9] Traceback (most recent call last):
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 441, in get
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     return self._queues[msg_id].get(block=True, timeout=timeout)
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/queue.py", line 322, in get
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     return waiter.wait()
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/queue.py", line 141, in wait
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     return get_hub().switch()
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 313, in switch
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     return self.greenlet.switch()
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9] _queue.Empty
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9] 
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9] During handling of the above exception, another exception occurred:
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9] 
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9] Traceback (most recent call last):
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/nova/nova/compute/manager.py", line 3315, in do_terminate_instance
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     self._delete_instance(context, instance, bdms)
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/nova/nova/compute/manager.py", line 3279, in _delete_instance
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     self._complete_deletion(context, instance)
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/nova/nova/compute/manager.py", line 926, in _complete_deletion
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     self._update_resource_tracker(context, instance)
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/nova/nova/compute/manager.py", line 693, in _update_resource_tracker
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     self.rt.update_usage(context, instance, instance.node)
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py", line 414, in inner
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     return f(*args, **kwargs)
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/nova/nova/compute/resource_tracker.py", line 702, in update_usage
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     self._update(context.elevated(), self.compute_nodes[nodename])
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1400, in _update
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     with excutils.save_and_reraise_exception(logger=LOG):
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     self.force_reraise()
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     raise self.value
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1395, in _update
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     compute_node.save()
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_versionedobjects/base.py", line 209, in wrapper
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     updates, result = self.indirection_api.object_action(
Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/nova/nova/conductor/rpcapi.py", line 247, in object_action
Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     return cctxt.call(context, 'object_action', objinst=objinst,
Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/rpc/client.py", line 190, in call
Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     result = self.transport._send(
Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send
Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     return self._driver.send(target, ctxt, message,
Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     return self._send(target, ctxt, message, wait_for_reply, timeout,
Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 678, in _send
Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     result = self._waiter.wait(msg_id, timeout,
Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 567, in wait
Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     message = self.waiters.get(msg_id, timeout=timeout)
Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 443, in get
Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     raise oslo_messaging.MessagingTimeout(
Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9] oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 1bb84ce3208146ae828a98299bf658ea
Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]
```

I checked conductor and api logs but could find anything relevant there.
Rabbitmq logs seem fine too.

n-cond log has this but I don't think it's related and it seems to be
retried later anyway:

```
Sep 05 13:56:15.434438 np0038400601 nova-conductor[66392]: DEBUG oslo_db.api [-] DB error: (pymysql.err.OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction')
Sep 05 13:56:15.434438 np0038400601 nova-conductor[66392]: [SQL: UPDATE services SET updated_at=%(updated_at)s, report_count=%(report_count)s, last_seen_up=%(last_seen_up)s WHERE services.id = %(services_id)s]
Sep 05 13:56:15.434438 np0038400601 nova-conductor[66392]: [parameters: {'updated_at': datetime.datetime(2024, 9, 5, 13, 55, 24, 730896), 'report_count': 276, 'last_seen_up': datetime.datetime(2024, 9, 5, 13, 55, 24, 730149), 'services_id': 1}]
Sep 05 13:56:15.434438 np0038400601 nova-conductor[66392]: (Background on this error at: https://sqlalche.me/e/14/e3q8) {{(pid=66392) _is_exception_expected /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/api.py:184}}
Sep 05 13:56:15.435657 np0038400601 nova-conductor[66392]: DEBUG oslo_db.api [-] Performing DB retry for function nova.db.main.api.service_update {{(pid=66392) wrapper /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/api.py:155}}
```

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2079205

Title:
  Tempest test fails to delete an instance, MessagingTimeout in n-cpu
  log when processing "_complete_deletion"

Status in OpenStack Compute (nova):
  New

Bug description:
  Logs:
  https://ae5a8daa4540c7da75ef-0b252c7b71d313ec10c19ced03a4a34b.ssl.cf1.rackcdn.com/927182/1/check/neutron-
  tempest-plugin-openvswitch-
  iptables_hybrid-2023-2/1047b7f/testr_results.html

  Test traceback:

  ```
  Traceback (most recent call last):
    File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 87, in call_and_ignore_notfound_exc
      return func(*args, **kwargs)
    File "/opt/stack/tempest/tempest/common/waiters.py", line 136, in wait_for_server_termination
      raise lib_exc.DeleteErrorException(details, server_id=server_id)
  tempest.lib.exceptions.DeleteErrorException: Resource %(resource_id)s failed to delete and is in ERROR status
  Details: Server f9acc6d9-132d-4ed8-875a-48fedb5b81b9 failed to delete and is in ERROR status. Fault: {'code': 500, 'created': '2024-09-05T13:56:54Z', 'message': 'MessagingTimeout'}.
  ```

  In n-cpu logs,
  https://ae5a8daa4540c7da75ef-0b252c7b71d313ec10c19ced03a4a34b.ssl.cf1.rackcdn.com/927182/1/check/neutron-
  tempest-plugin-openvswitch-
  iptables_hybrid-2023-2/1047b7f/controller/logs/screen-n-cpu.txt:

  ```
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [None req-99b4aa7a-c470-4313-807c-d22d46b0ae38 tempest-MulticastTestIPv4-1796741707 tempest-MulticastTestIPv4-1796741707-project-member] [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9] Setting instance vm_state to ERROR: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 1bb84ce3208146ae828a98299bf658ea
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9] Traceback (most recent call last):
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 441, in get
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     return self._queues[msg_id].get(block=True, timeout=timeout)
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/queue.py", line 322, in get
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     return waiter.wait()
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/queue.py", line 141, in wait
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     return get_hub().switch()
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 313, in switch
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     return self.greenlet.switch()
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9] _queue.Empty
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9] 
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9] During handling of the above exception, another exception occurred:
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9] 
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9] Traceback (most recent call last):
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/nova/nova/compute/manager.py", line 3315, in do_terminate_instance
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     self._delete_instance(context, instance, bdms)
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/nova/nova/compute/manager.py", line 3279, in _delete_instance
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     self._complete_deletion(context, instance)
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/nova/nova/compute/manager.py", line 926, in _complete_deletion
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     self._update_resource_tracker(context, instance)
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/nova/nova/compute/manager.py", line 693, in _update_resource_tracker
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     self.rt.update_usage(context, instance, instance.node)
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py", line 414, in inner
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     return f(*args, **kwargs)
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/nova/nova/compute/resource_tracker.py", line 702, in update_usage
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     self._update(context.elevated(), self.compute_nodes[nodename])
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1400, in _update
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     with excutils.save_and_reraise_exception(logger=LOG):
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     self.force_reraise()
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     raise self.value
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1395, in _update
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     compute_node.save()
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_versionedobjects/base.py", line 209, in wrapper
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     updates, result = self.indirection_api.object_action(
  Sep 05 13:56:23.165930 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/nova/nova/conductor/rpcapi.py", line 247, in object_action
  Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     return cctxt.call(context, 'object_action', objinst=objinst,
  Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/rpc/client.py", line 190, in call
  Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     result = self.transport._send(
  Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send
  Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     return self._driver.send(target, ctxt, message,
  Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
  Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     return self._send(target, ctxt, message, wait_for_reply, timeout,
  Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 678, in _send
  Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     result = self._waiter.wait(msg_id, timeout,
  Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 567, in wait
  Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     message = self.waiters.get(msg_id, timeout=timeout)
  Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 443, in get
  Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]     raise oslo_messaging.MessagingTimeout(
  Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9] oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 1bb84ce3208146ae828a98299bf658ea
  Sep 05 13:56:23.170702 np0038400601 nova-compute[66332]: ERROR nova.compute.manager [instance: f9acc6d9-132d-4ed8-875a-48fedb5b81b9]
  ```

  I checked conductor and api logs but could find anything relevant
  there. Rabbitmq logs seem fine too.

  n-cond log has this but I don't think it's related and it seems to be
  retried later anyway:

  ```
  Sep 05 13:56:15.434438 np0038400601 nova-conductor[66392]: DEBUG oslo_db.api [-] DB error: (pymysql.err.OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction')
  Sep 05 13:56:15.434438 np0038400601 nova-conductor[66392]: [SQL: UPDATE services SET updated_at=%(updated_at)s, report_count=%(report_count)s, last_seen_up=%(last_seen_up)s WHERE services.id = %(services_id)s]
  Sep 05 13:56:15.434438 np0038400601 nova-conductor[66392]: [parameters: {'updated_at': datetime.datetime(2024, 9, 5, 13, 55, 24, 730896), 'report_count': 276, 'last_seen_up': datetime.datetime(2024, 9, 5, 13, 55, 24, 730149), 'services_id': 1}]
  Sep 05 13:56:15.434438 np0038400601 nova-conductor[66392]: (Background on this error at: https://sqlalche.me/e/14/e3q8) {{(pid=66392) _is_exception_expected /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/api.py:184}}
  Sep 05 13:56:15.435657 np0038400601 nova-conductor[66392]: DEBUG oslo_db.api [-] Performing DB retry for function nova.db.main.api.service_update {{(pid=66392) wrapper /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/api.py:155}}
  ```

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2079205/+subscriptions