yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2044721] [NEW] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'oslo_messaging.exceptions.MessagingTimeout'> call /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Nabil <2044721@xxxxxxxxxxxxxxxxxx>
Date: Mon, 27 Nov 2023 09:41:28 -0000
Reply-to: Bug 2044721 <2044721@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

We were upgrading our server hardware. our compute server was corrupted
(unable to enter ubuntu os) , so i reflash the same ubuntu os version to
it & with the same IP address/mac address , and redeployed the kolla-
ansible with the same configuration . however , after that nova_compute
& zun_compute service on compute node seems to be restarting by itself
(exit ---> restart ---> exit ---> restart). I have no clue whats the
solution to it


expected result
=========================
all services should be working correctly


actual result
=========================
zun_compute is down
nova_compute is down
horizon compute node is down


environment
=======================
kolla-ansible V2023.1
ubuntu version 22.04 for all 3 servers


<class 'oslo_messaging.exceptions.MessagingTimeout'>
2023-11-27 06:26:18.431 22 ERROR nova.api.openstack.wsgi [None req-0c9d29ab-f230-4b26-828f-b1bce144acc9 d54f447283c843df8c11b038ef198fc0 d865adb13a8d4c15b7a4c07f040efdc5 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 6fc08dc0c65849d98b55f762441ae230
2023-11-27 06:26:18.431 22 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 6fc08dc0c65849d98b55f762441ae230
<class 'oslo_messaging.exceptions.MessagingTimeout'>
<class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
2023-11-27 06:27:18.420 21 ERROR nova.api.openstack.wsgi [None req-c61791f0-c689-4b37-bc73-183586fd45c1 d54f447283c843df8c11b038ef198fc0 d865adb13a8d4c15b7a4c07f040efdc5 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID b3811b5050fb45238d0cce866b8db0d3
2023-11-27 06:27:18.420 21 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID b3811b5050fb45238d0cce866b8db0d3
<class 'oslo_messaging.exceptions.MessagingTimeout'>
<class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
2023-11-27 06:28:18.520 23 ERROR nova.api.openstack.wsgi [None req-25845c63-c886-4878-a976-d427f1fb311f - - - - - -] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID a2f8e06d636f4d96b89c5f5e6b685db0
2023-11-27 06:28:18.520 23 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID a2f8e06d636f4d96b89c5f5e6b685db0
<class 'oslo_messaging.exceptions.MessagingTimeout'>
<class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
2023-11-27 06:29:18.436 19 ERROR nova.api.openstack.wsgi [None req-9eb9d577-c192-4579-a8aa-00d7cabb793b d54f447283c843df8c11b038ef198fc0 d865adb13a8d4c15b7a4c07f040efdc5 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 7523a9908d5341a7a667be6c0a1775ea
2023-11-27 06:29:18.436 19 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 7523a9908d5341a7a667be6c0a1775ea
<class 'oslo_messaging.exceptions.MessagingTimeout'>
<class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
2023-11-27 06:30:18.603 23 ERROR nova.api.openstack.wsgi [None req-25f72773-caee-4551-8ebb-92747f12e77f d54f447283c843df8c11b038ef198fc0 d865adb13a8d4c15b7a4c07f040efdc5 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID ac1f456f77c94a96b6deea5d2c3186a5
2023-11-27 06:30:18.603 23 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID ac1f456f77c94a96b6deea5d2c3186a5
<class 'oslo_messaging.exceptions.MessagingTimeout'>
<class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
2023-11-27 06:30:51.383 21 ERROR nova.api.openstack.wsgi [None req-0ecdaea3-7494-4bfd-9c47-d571595f2356 d54f447283c843df8c11b038ef198fc0 d865adb13a8d4c15b7a4c07f040efdc5 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID cc479a6b25994ae2b1e88b0acdad6fa0
2023-11-27 06:30:51.383 21 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID cc479a6b25994ae2b1e88b0acdad6fa0
<class 'oslo_messaging.exceptions.MessagingTimeout'>
<class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
2023-11-27 06:31:51.444 20 ERROR nova.api.openstack.wsgi [None req-be6e7afb-b3e6-4369-ade3-47ad5323d859 - - - - - -] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID d517cb22d4df45649690a6d5c8b69077
2023-11-27 06:31:51.444 20 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID d517cb22d4df45649690a6d5c8b69077
<class 'oslo_messaging.exceptions.MessagingTimeout'>
<class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
2023-11-27 06:32:51.412 19 ERROR nova.api.openstack.wsgi [None req-eaff062a-22bf-4589-82ce-599937f07984 d54f447283c843df8c11b038ef198fc0 d865adb13a8d4c15b7a4c07f040efdc5 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID ff41269181144be98c9d15745063a10d
2023-11-27 06:32:51.412 19 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID ff41269181144be98c9d15745063a10d
<class 'oslo_messaging.exceptions.MessagingTimeout'>
<class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
2023-11-27 06:33:51.421 20 ERROR nova.api.openstack.wsgi [None req-088e4fb0-058d-410f-bd1b-d4d7788ce15b d54f447283c843df8c11b038ef198fc0 d865adb13a8d4c15b7a4c07f040efdc5 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID cbbb44e1417540fba9b4405146b13dc2
2023-11-27 06:33:51.421 20 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID cbbb44e1417540fba9b4405146b13dc2
<class 'oslo_messaging.exceptions.MessagingTimeout'>
<class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
2023-11-27 06:34:51.506 19 ERROR nova.api.openstack.wsgi [None req-9e415a3e-85c4-4bb3-907e-e7cbb3dd2daf d54f447283c843df8c11b038ef198fc0 d865adb13a8d4c15b7a4c07f040efdc5 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID c44ae10b2f774b428763a1d11368cb16
2023-11-27 06:34:51.506 19 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID c44ae10b2f774b428763a1d11368cb16
<class 'oslo_messaging.exceptions.MessagingTimeout'>
<class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936

** Affects: nova
     Importance: Undecided
         Status: New

** Attachment added: "nova api logs"
   https://bugs.launchpad.net/bugs/2044721/+attachment/5723797/+files/nova-api.txt

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2044721

Title:
          Returning 500 to user: Unexpected API Error. Please report
  this at http://bugs.launchpad.net/nova/ and attach the Nova API log if
  possible. <class 'oslo_messaging.exceptions.MessagingTimeout'>
  __call__ /var/lib/kolla/venv/lib/python3.10/site-
  packages/nova/api/openstack/wsgi.py:936

Status in OpenStack Compute (nova):
  New

Bug description:
  We were upgrading our server hardware. our compute server was
  corrupted (unable to enter ubuntu os) , so i reflash the same ubuntu
  os version to it & with the same IP address/mac address , and
  redeployed the kolla-ansible with the same configuration . however ,
  after that nova_compute & zun_compute service on compute node seems to
  be restarting by itself (exit ---> restart ---> exit ---> restart). I
  have no clue whats the solution to it

  
  expected result
  =========================
  all services should be working correctly

  
  actual result
  =========================
  zun_compute is down
  nova_compute is down
  horizon compute node is down

  
  environment
  =======================
  kolla-ansible V2023.1
  ubuntu version 22.04 for all 3 servers

  
  <class 'oslo_messaging.exceptions.MessagingTimeout'>
  2023-11-27 06:26:18.431 22 ERROR nova.api.openstack.wsgi [None req-0c9d29ab-f230-4b26-828f-b1bce144acc9 d54f447283c843df8c11b038ef198fc0 d865adb13a8d4c15b7a4c07f040efdc5 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 6fc08dc0c65849d98b55f762441ae230
  2023-11-27 06:26:18.431 22 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 6fc08dc0c65849d98b55f762441ae230
  <class 'oslo_messaging.exceptions.MessagingTimeout'>
  <class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
  2023-11-27 06:27:18.420 21 ERROR nova.api.openstack.wsgi [None req-c61791f0-c689-4b37-bc73-183586fd45c1 d54f447283c843df8c11b038ef198fc0 d865adb13a8d4c15b7a4c07f040efdc5 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID b3811b5050fb45238d0cce866b8db0d3
  2023-11-27 06:27:18.420 21 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID b3811b5050fb45238d0cce866b8db0d3
  <class 'oslo_messaging.exceptions.MessagingTimeout'>
  <class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
  2023-11-27 06:28:18.520 23 ERROR nova.api.openstack.wsgi [None req-25845c63-c886-4878-a976-d427f1fb311f - - - - - -] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID a2f8e06d636f4d96b89c5f5e6b685db0
  2023-11-27 06:28:18.520 23 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID a2f8e06d636f4d96b89c5f5e6b685db0
  <class 'oslo_messaging.exceptions.MessagingTimeout'>
  <class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
  2023-11-27 06:29:18.436 19 ERROR nova.api.openstack.wsgi [None req-9eb9d577-c192-4579-a8aa-00d7cabb793b d54f447283c843df8c11b038ef198fc0 d865adb13a8d4c15b7a4c07f040efdc5 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 7523a9908d5341a7a667be6c0a1775ea
  2023-11-27 06:29:18.436 19 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 7523a9908d5341a7a667be6c0a1775ea
  <class 'oslo_messaging.exceptions.MessagingTimeout'>
  <class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
  2023-11-27 06:30:18.603 23 ERROR nova.api.openstack.wsgi [None req-25f72773-caee-4551-8ebb-92747f12e77f d54f447283c843df8c11b038ef198fc0 d865adb13a8d4c15b7a4c07f040efdc5 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID ac1f456f77c94a96b6deea5d2c3186a5
  2023-11-27 06:30:18.603 23 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID ac1f456f77c94a96b6deea5d2c3186a5
  <class 'oslo_messaging.exceptions.MessagingTimeout'>
  <class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
  2023-11-27 06:30:51.383 21 ERROR nova.api.openstack.wsgi [None req-0ecdaea3-7494-4bfd-9c47-d571595f2356 d54f447283c843df8c11b038ef198fc0 d865adb13a8d4c15b7a4c07f040efdc5 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID cc479a6b25994ae2b1e88b0acdad6fa0
  2023-11-27 06:30:51.383 21 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID cc479a6b25994ae2b1e88b0acdad6fa0
  <class 'oslo_messaging.exceptions.MessagingTimeout'>
  <class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
  2023-11-27 06:31:51.444 20 ERROR nova.api.openstack.wsgi [None req-be6e7afb-b3e6-4369-ade3-47ad5323d859 - - - - - -] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID d517cb22d4df45649690a6d5c8b69077
  2023-11-27 06:31:51.444 20 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID d517cb22d4df45649690a6d5c8b69077
  <class 'oslo_messaging.exceptions.MessagingTimeout'>
  <class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
  2023-11-27 06:32:51.412 19 ERROR nova.api.openstack.wsgi [None req-eaff062a-22bf-4589-82ce-599937f07984 d54f447283c843df8c11b038ef198fc0 d865adb13a8d4c15b7a4c07f040efdc5 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID ff41269181144be98c9d15745063a10d
  2023-11-27 06:32:51.412 19 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID ff41269181144be98c9d15745063a10d
  <class 'oslo_messaging.exceptions.MessagingTimeout'>
  <class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
  2023-11-27 06:33:51.421 20 ERROR nova.api.openstack.wsgi [None req-088e4fb0-058d-410f-bd1b-d4d7788ce15b d54f447283c843df8c11b038ef198fc0 d865adb13a8d4c15b7a4c07f040efdc5 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID cbbb44e1417540fba9b4405146b13dc2
  2023-11-27 06:33:51.421 20 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID cbbb44e1417540fba9b4405146b13dc2
  <class 'oslo_messaging.exceptions.MessagingTimeout'>
  <class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
  2023-11-27 06:34:51.506 19 ERROR nova.api.openstack.wsgi [None req-9e415a3e-85c4-4bb3-907e-e7cbb3dd2daf d54f447283c843df8c11b038ef198fc0 d865adb13a8d4c15b7a4c07f040efdc5 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID c44ae10b2f774b428763a1d11368cb16
  2023-11-27 06:34:51.506 19 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID c44ae10b2f774b428763a1d11368cb16
  <class 'oslo_messaging.exceptions.MessagingTimeout'>
  <class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2044721/+subscriptions
Follow ups

[Bug 2044721] Re: Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
From: Sylvain Bauza, 2023-11-28