yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1968054] Re: oslo.messaging._drivers.impl_rabbit Connection failed: timed out

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Isa Fishman <1968054@xxxxxxxxxxxxxxxxxx>
Date: Wed, 26 Apr 2023 18:13:16 -0000
Reply-to: Bug 1968054 <1968054@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
This bug effects me whenever I try to launch an instance. The instance
starts the "Scheduling" process and hangs. If I change the connection
URL to an invalid one it gives an error that does not allow the instance
to enter the "Scheduling" state. The communication seems to be
functional as when I try to create an oversized instance it fails with a
different message.

** Changed in: nova
       Status: Invalid => New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1968054

Title:
  oslo.messaging._drivers.impl_rabbit Connection failed: timed out

Status in OpenStack Compute (nova):
  New
Status in oslo.messaging:
  New

Bug description:
  I am running Wallaby Release on Ubuntu 20.04 (Openstack-Ansible
  deployment tool)

  oslo.messaging=12.7.1
  nova=23.1.1

  since i upgrade to Wallaby i have started noticed following error
  message very frequently in nova-compute and solution is to restart
  nova-compute agent.

  Here is the full logs:
  https://paste.opendev.org/show/bft9znewTxyXHkvIcQO0/

  
  01 19:43:36 compute1.example.net nova-compute[1546242]: AssertionError:
  Apr 01 19:45:35 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:35.059 34090 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 110] Connection timed out
  Apr 01 19:45:40 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:40.063 34090 ERROR oslo.messaging._drivers.impl_rabbit [req-707abbfe-8ee0-4af7-900a-e43dc5dec597 - - - - -] [7d350e59-001f-4203-bd41-369650cd5c5c] AMQP server on 172.28.17.24:5671 is unreachable: . Trying again in 1 seconds.: socket.timeout
  Apr 01 19:45:40 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:40.079 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: timed out (retrying in 0 seconds): socket.timeout: timed out
  Apr 01 19:45:41 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:41.983 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 0 seconds): OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:42.367 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 2.0 seconds): OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]: Traceback (most recent call last):
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]:   File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]:     timer()
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]:   File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/timer.py", line 59, in __call__
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]:     cb(*args, **kw)
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]:   File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/semaphore.py", line 152, in _do_acquire
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]:     waiter.switch()
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]: greenlet.error: cannot switch to a different thread
  Apr 01 19:45:49 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:49.388 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: timed out (retrying in 0 seconds): socket.timeout: timed out
  Apr 01 19:45:50 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:50.303 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] No route to host. Trying again in 1 seconds.: OSError: [Errno 113] No route to host
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:51.199 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 0 seconds): OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:51.583 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]: Traceback (most recent call last):
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]:   File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]:     timer()
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]:   File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/timer.py", line 59, in __call__
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]:     cb(*args, **kw)
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]:   File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/semaphore.py", line 152, in _do_acquire
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]:     waiter.switch()
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]: greenlet.error: cannot switch to a different thread
  Apr 01 19:45:57 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:57.601 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: timed out. Trying again in 1 seconds.: socket.timeout: timed out
  Apr 01 19:46:00 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:00.415 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 21 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:18 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:18.464 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 110] Connection timed out. Trying again in 1 seconds.: TimeoutError: [Errno 110] Connection timed out
  Apr 01 19:46:22 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:22.559 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:24 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:24.384 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error>
  Apr 01 19:46:24 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:24.384 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:25 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:25.631 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:26 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:26.802 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: . Trying again in 1 seconds.: socket.timeout
  Apr 01 19:46:27 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:27.455 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:28 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:28.063 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 24 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:28 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:28.064 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:30 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:30.527 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:31 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:31.135 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:31 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:31.775 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 21 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:35 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:35.199 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 21 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:52 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:52.575 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:53 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:53.599 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:53 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:53.599 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:56 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:56.671 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 27 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:57 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:57.663 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:57 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:57.664 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:59 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:59.743 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 24 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:59 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:59.744 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:02 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:02.816 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 24 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:25 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:25.791 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:26 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:26.783 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:28 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:28.864 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:29 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:29.855 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:29 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:29.856 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:31 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:31.935 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 30 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:31 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:31.937 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 27 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:31 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:31.939 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:35 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:35.007 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 27 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:01 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:01.023 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:05 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:05.055 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:05 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:05.056 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:05 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:05.057 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:07 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:07.167 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 30 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:07 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:07.168 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:08 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:08.159 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:10 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:10.239 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 30 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:11 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:11.231 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 33 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:40 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:40.287 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error>
  Apr 01 19:48:40 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:40.288 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:44 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:44.384 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:44 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:44.385 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:44 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:44.386 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:47 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:47.455 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:47 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:47.456 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 33 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:48 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:48.483 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 36 seconds.: OSError: [Errno 113] EHOSTUNREACH


  My compute nova.conf looks like following

  [oslo_messaging_rabbit]
  rabbit_retry_interval = 20
  rabbit_retry_backoff = 3
  rabbit_interval_max = 60
  rabbit_transient_queues_ttl = 300
  rabbit_qos_prefetch_count = 100
  rpc_conn_pool_size = 300

  
  Feel like something crashed in oslo driver and not able to recover itself. I have other deployment running on stein and i haven't seen this kind of issue in last 3 years so definitely this is something related to wallaby oslo bug.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1968054/+subscriptions
References

[Bug 1968054] [NEW] oslo.messaging._drivers.impl_rabbit Connection failed: timed out
From: Satish Patel, 2022-04-06