yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2119106] [NEW] [Caracal][Nova][OFFLINE] - API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Nilesh <2119106@xxxxxxxxxxxxxxxxxx>
Date: Wed, 30 Jul 2025 10:42:40 -0000
Reply-to: Bug 2119106 <2119106@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

++++++++++++
Env Details:
++++++++++++

OSA: 2024.1 <Caracal>
OS: Ubuntu 22.04


* We have a Offline deployed openstack enviroenmmtn, The environemnt is up and running, Couple of test cases we performed, 

* able to create instance from image and volume, evacuation, migration
everything is working.

* We saw one patern whenever we are running ``opensatck hypervisor list
`` , sometime its working sometime its giving an exception, out of 5
times only 2/3 times its working

```
RESP BODY: {"computeFault": {"code": 500, "message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'oslo_messaging.exceptions.MessagingTimeout'>"}}
GET call to compute for http://192.168.131.200:8774/v2.1/os-hypervisors/detail used request id req-ad28b6c6-3277-4e94-9023-0478ac71f33a
HttpException: 500: Server Error for url: http://192.168.131.200:8774/v2.1/os-hypervisors/detail, Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'>
```


```
Jul 30 08:33:57 offline20241.ct.lan nova-api-wsgi[2061714]: 2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi [None req-12ea9d0c-f983-468b-afa3-cb6562be492e cfc72fc0bc3d4ba09ef9756ea2fb6395 2f5a2a06638942cbaaeeb466b2e17e10 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID a48c30c8be8d4765a6bbf58b6916ec0f
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 511, in get
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     return self._queues[msg_id].get(block=True, timeout=timeout)
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/eventlet/queue.py", line 320, in get
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     return waiter.wait()
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/eventlet/queue.py", line 139, in wait
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     return get_hub().switch()
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     return self.greenlet.switch()
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi _queue.Empty
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi 
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi During handling of the above exception, another exception occurred:
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi 
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/nova/api/openstack/wsgi.py", line 658, in wrapped
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/nova/api/openstack/compute/hypervisors.py", line 273, in detail
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     return self._detail(req, limit=limit, marker=marker, links=True)
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/nova/api/openstack/compute/hypervisors.py", line 290, in _detail
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     return self._get_hypervisors(req, detail=True, limit=limit,
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/nova/api/openstack/compute/hypervisors.py", line 215, in _get_hypervisors
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     hypervisor = self._view_hypervisor(
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/nova/api/openstack/compute/hypervisors.py", line 106, in _view_hypervisor
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     hyp_dict['uptime'] = self.host_api.get_host_uptime(
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/nova/compute/api.py", line 6042, in targeted
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     return fn(self, context, host, *args, **kwargs)
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/nova/compute/api.py", line 6157, in get_host_uptime
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     return self.rpcapi.get_host_uptime(context, host=host_name)
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/nova/compute/rpcapi.py", line 1311, in get_host_uptime
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     return cctxt.call(ctxt, 'get_host_uptime')
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/oslo_messaging/rpc/client.py", line 190, in call
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     result = self.transport._send(
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     return self._driver.send(target, ctxt, message,
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 793, in send
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     return self._send(target, ctxt, message, wait_for_reply, timeout,
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 782, in _send
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     result = self._waiter.wait(msg_id, timeout,
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 647, in wait
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     message = self.waiters.get(msg_id, timeout=timeout)
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 513, in get
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     raise oslo_messaging.MessagingTimeout(
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID a48c30c8be8d4765a6bbf58b6916ec0f
                                                            2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi 
Jul 30 08:33:57 offline20241.ct.lan nova-api-wsgi[2061714]: 2025-07-30 08:33:57.521 2061714 INFO nova.api.openstack.wsgi [None req-12ea9d0c-f983-468b-afa3-cb6562be492e cfc72fc0bc3d4ba09ef9756ea2fb6395 2f5a2a06638942cbaaeeb466b2e17e10 - - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
```

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: nova openstack-ansible

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2119106

Title:
  [Caracal][Nova][OFFLINE] - API method:
  oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a
  reply to message ID

Status in OpenStack Compute (nova):
  New

Bug description:
  ++++++++++++
  Env Details:
  ++++++++++++

  OSA: 2024.1 <Caracal>
  OS: Ubuntu 22.04

  
  * We have a Offline deployed openstack enviroenmmtn, The environemnt is up and running, Couple of test cases we performed, 

  * able to create instance from image and volume, evacuation, migration
  everything is working.

  * We saw one patern whenever we are running ``opensatck hypervisor
  list `` , sometime its working sometime its giving an exception, out
  of 5 times only 2/3 times its working

  ```
  RESP BODY: {"computeFault": {"code": 500, "message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'oslo_messaging.exceptions.MessagingTimeout'>"}}
  GET call to compute for http://192.168.131.200:8774/v2.1/os-hypervisors/detail used request id req-ad28b6c6-3277-4e94-9023-0478ac71f33a
  HttpException: 500: Server Error for url: http://192.168.131.200:8774/v2.1/os-hypervisors/detail, Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'oslo_messaging.exceptions.MessagingTimeout'>
  ```


  ```
  Jul 30 08:33:57 offline20241.ct.lan nova-api-wsgi[2061714]: 2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi [None req-12ea9d0c-f983-468b-afa3-cb6562be492e cfc72fc0bc3d4ba09ef9756ea2fb6395 2f5a2a06638942cbaaeeb466b2e17e10 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID a48c30c8be8d4765a6bbf58b6916ec0f
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 511, in get
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     return self._queues[msg_id].get(block=True, timeout=timeout)
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/eventlet/queue.py", line 320, in get
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     return waiter.wait()
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/eventlet/queue.py", line 139, in wait
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     return get_hub().switch()
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     return self.greenlet.switch()
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi _queue.Empty
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi 
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi During handling of the above exception, another exception occurred:
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi 
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/nova/api/openstack/wsgi.py", line 658, in wrapped
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/nova/api/openstack/compute/hypervisors.py", line 273, in detail
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     return self._detail(req, limit=limit, marker=marker, links=True)
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/nova/api/openstack/compute/hypervisors.py", line 290, in _detail
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     return self._get_hypervisors(req, detail=True, limit=limit,
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/nova/api/openstack/compute/hypervisors.py", line 215, in _get_hypervisors
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     hypervisor = self._view_hypervisor(
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/nova/api/openstack/compute/hypervisors.py", line 106, in _view_hypervisor
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     hyp_dict['uptime'] = self.host_api.get_host_uptime(
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/nova/compute/api.py", line 6042, in targeted
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     return fn(self, context, host, *args, **kwargs)
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/nova/compute/api.py", line 6157, in get_host_uptime
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     return self.rpcapi.get_host_uptime(context, host=host_name)
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/nova/compute/rpcapi.py", line 1311, in get_host_uptime
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     return cctxt.call(ctxt, 'get_host_uptime')
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/oslo_messaging/rpc/client.py", line 190, in call
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     result = self.transport._send(
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     return self._driver.send(target, ctxt, message,
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 793, in send
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     return self._send(target, ctxt, message, wait_for_reply, timeout,
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 782, in _send
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     result = self._waiter.wait(msg_id, timeout,
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 647, in wait
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     message = self.waiters.get(msg_id, timeout=timeout)
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-29.2.3.dev1/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 513, in get
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi     raise oslo_messaging.MessagingTimeout(
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID a48c30c8be8d4765a6bbf58b6916ec0f
                                                              2025-07-30 08:33:57.518 2061714 ERROR nova.api.openstack.wsgi 
  Jul 30 08:33:57 offline20241.ct.lan nova-api-wsgi[2061714]: 2025-07-30 08:33:57.521 2061714 INFO nova.api.openstack.wsgi [None req-12ea9d0c-f983-468b-afa3-cb6562be492e cfc72fc0bc3d4ba09ef9756ea2fb6395 2f5a2a06638942cbaaeeb466b2e17e10 - - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  ```

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2119106/+subscriptions