yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1968054] [NEW] oslo.messaging._drivers.impl_rabbit Connection failed: timed out

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Satish Patel <1968054@xxxxxxxxxxxxxxxxxx>
Date: Wed, 06 Apr 2022 15:20:54 -0000
Reply-to: Bug 1968054 <1968054@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

I am running Wallaby Release on Ubuntu 20.04 (Openstack-Ansible
deployment tool)

oslo.messaging=12.7.1
nova=23.1.1

since i upgrade to Wallaby i have started noticed following error
message very frequently in nova-compute and solution is to restart nova-
compute agent.

Here is the full logs:
https://paste.opendev.org/show/bft9znewTxyXHkvIcQO0/


01 19:43:36 compute1.example.net nova-compute[1546242]: AssertionError:
Apr 01 19:45:35 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:35.059 34090 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 110] Connection timed out
Apr 01 19:45:40 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:40.063 34090 ERROR oslo.messaging._drivers.impl_rabbit [req-707abbfe-8ee0-4af7-900a-e43dc5dec597 - - - - -] [7d350e59-001f-4203-bd41-369650cd5c5c] AMQP server on 172.28.17.24:5671 is unreachable: . Trying again in 1 seconds.: socket.timeout
Apr 01 19:45:40 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:40.079 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: timed out (retrying in 0 seconds): socket.timeout: timed out
Apr 01 19:45:41 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:41.983 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 0 seconds): OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:45:42 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:42.367 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 2.0 seconds): OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:45:42 compute1.example.net nova-compute[34090]: Traceback (most recent call last):
Apr 01 19:45:42 compute1.example.net nova-compute[34090]:   File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers
Apr 01 19:45:42 compute1.example.net nova-compute[34090]:     timer()
Apr 01 19:45:42 compute1.example.net nova-compute[34090]:   File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/timer.py", line 59, in __call__
Apr 01 19:45:42 compute1.example.net nova-compute[34090]:     cb(*args, **kw)
Apr 01 19:45:42 compute1.example.net nova-compute[34090]:   File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/semaphore.py", line 152, in _do_acquire
Apr 01 19:45:42 compute1.example.net nova-compute[34090]:     waiter.switch()
Apr 01 19:45:42 compute1.example.net nova-compute[34090]: greenlet.error: cannot switch to a different thread
Apr 01 19:45:49 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:49.388 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: timed out (retrying in 0 seconds): socket.timeout: timed out
Apr 01 19:45:50 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:50.303 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] No route to host. Trying again in 1 seconds.: OSError: [Errno 113] No route to host
Apr 01 19:45:51 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:51.199 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 0 seconds): OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:45:51 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:51.583 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:45:51 compute1.example.net nova-compute[34090]: Traceback (most recent call last):
Apr 01 19:45:51 compute1.example.net nova-compute[34090]:   File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers
Apr 01 19:45:51 compute1.example.net nova-compute[34090]:     timer()
Apr 01 19:45:51 compute1.example.net nova-compute[34090]:   File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/timer.py", line 59, in __call__
Apr 01 19:45:51 compute1.example.net nova-compute[34090]:     cb(*args, **kw)
Apr 01 19:45:51 compute1.example.net nova-compute[34090]:   File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/semaphore.py", line 152, in _do_acquire
Apr 01 19:45:51 compute1.example.net nova-compute[34090]:     waiter.switch()
Apr 01 19:45:51 compute1.example.net nova-compute[34090]: greenlet.error: cannot switch to a different thread
Apr 01 19:45:57 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:57.601 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: timed out. Trying again in 1 seconds.: socket.timeout: timed out
Apr 01 19:46:00 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:00.415 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 21 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:18 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:18.464 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 110] Connection timed out. Trying again in 1 seconds.: TimeoutError: [Errno 110] Connection timed out
Apr 01 19:46:22 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:22.559 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:24 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:24.384 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error>
Apr 01 19:46:24 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:24.384 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:25 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:25.631 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:26 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:26.802 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: . Trying again in 1 seconds.: socket.timeout
Apr 01 19:46:27 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:27.455 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:28 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:28.063 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 24 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:28 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:28.064 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:30 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:30.527 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:31 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:31.135 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:31 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:31.775 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 21 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:35 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:35.199 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 21 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:52 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:52.575 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:53 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:53.599 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:53 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:53.599 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:56 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:56.671 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 27 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:57 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:57.663 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:57 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:57.664 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:59 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:59.743 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 24 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:59 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:59.744 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:47:02 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:02.816 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 24 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:47:25 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:25.791 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:47:26 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:26.783 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:47:28 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:28.864 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:47:29 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:29.855 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:47:29 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:29.856 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:47:31 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:31.935 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 30 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:47:31 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:31.937 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 27 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:47:31 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:31.939 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:47:35 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:35.007 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 27 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:01 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:01.023 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:05 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:05.055 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:05 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:05.056 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:05 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:05.057 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:07 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:07.167 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 30 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:07 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:07.168 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:08 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:08.159 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:10 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:10.239 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 30 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:11 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:11.231 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 33 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:40 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:40.287 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error>
Apr 01 19:48:40 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:40.288 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:44 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:44.384 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:44 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:44.385 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:44 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:44.386 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:47 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:47.455 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:47 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:47.456 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 33 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:48 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:48.483 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 36 seconds.: OSError: [Errno 113] EHOSTUNREACH


My compute nova.conf looks like following

[oslo_messaging_rabbit]
rabbit_retry_interval = 20
rabbit_retry_backoff = 3
rabbit_interval_max = 60
rabbit_transient_queues_ttl = 300
rabbit_qos_prefetch_count = 100
rpc_conn_pool_size = 300


Feel like something crashed in oslo driver and not able to recover itself. I have other deployment running on stein and i haven't seen this kind of issue in last 3 years so definitely this is something related to wallaby oslo bug.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1968054

Title:
  oslo.messaging._drivers.impl_rabbit Connection failed: timed out

Status in OpenStack Compute (nova):
  New

Bug description:
  I am running Wallaby Release on Ubuntu 20.04 (Openstack-Ansible
  deployment tool)

  oslo.messaging=12.7.1
  nova=23.1.1

  since i upgrade to Wallaby i have started noticed following error
  message very frequently in nova-compute and solution is to restart
  nova-compute agent.

  Here is the full logs:
  https://paste.opendev.org/show/bft9znewTxyXHkvIcQO0/

  
  01 19:43:36 compute1.example.net nova-compute[1546242]: AssertionError:
  Apr 01 19:45:35 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:35.059 34090 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 110] Connection timed out
  Apr 01 19:45:40 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:40.063 34090 ERROR oslo.messaging._drivers.impl_rabbit [req-707abbfe-8ee0-4af7-900a-e43dc5dec597 - - - - -] [7d350e59-001f-4203-bd41-369650cd5c5c] AMQP server on 172.28.17.24:5671 is unreachable: . Trying again in 1 seconds.: socket.timeout
  Apr 01 19:45:40 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:40.079 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: timed out (retrying in 0 seconds): socket.timeout: timed out
  Apr 01 19:45:41 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:41.983 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 0 seconds): OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:42.367 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 2.0 seconds): OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]: Traceback (most recent call last):
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]:   File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]:     timer()
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]:   File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/timer.py", line 59, in __call__
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]:     cb(*args, **kw)
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]:   File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/semaphore.py", line 152, in _do_acquire
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]:     waiter.switch()
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]: greenlet.error: cannot switch to a different thread
  Apr 01 19:45:49 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:49.388 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: timed out (retrying in 0 seconds): socket.timeout: timed out
  Apr 01 19:45:50 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:50.303 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] No route to host. Trying again in 1 seconds.: OSError: [Errno 113] No route to host
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:51.199 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 0 seconds): OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:51.583 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]: Traceback (most recent call last):
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]:   File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]:     timer()
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]:   File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/timer.py", line 59, in __call__
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]:     cb(*args, **kw)
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]:   File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/semaphore.py", line 152, in _do_acquire
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]:     waiter.switch()
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]: greenlet.error: cannot switch to a different thread
  Apr 01 19:45:57 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:57.601 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: timed out. Trying again in 1 seconds.: socket.timeout: timed out
  Apr 01 19:46:00 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:00.415 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 21 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:18 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:18.464 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 110] Connection timed out. Trying again in 1 seconds.: TimeoutError: [Errno 110] Connection timed out
  Apr 01 19:46:22 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:22.559 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:24 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:24.384 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error>
  Apr 01 19:46:24 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:24.384 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:25 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:25.631 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:26 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:26.802 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: . Trying again in 1 seconds.: socket.timeout
  Apr 01 19:46:27 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:27.455 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:28 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:28.063 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 24 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:28 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:28.064 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:30 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:30.527 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:31 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:31.135 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:31 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:31.775 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 21 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:35 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:35.199 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 21 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:52 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:52.575 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:53 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:53.599 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:53 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:53.599 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:56 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:56.671 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 27 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:57 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:57.663 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:57 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:57.664 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:59 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:59.743 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 24 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:59 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:59.744 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:02 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:02.816 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 24 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:25 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:25.791 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:26 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:26.783 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:28 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:28.864 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:29 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:29.855 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:29 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:29.856 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:31 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:31.935 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 30 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:31 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:31.937 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 27 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:31 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:31.939 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:35 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:35.007 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 27 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:01 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:01.023 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:05 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:05.055 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:05 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:05.056 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:05 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:05.057 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:07 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:07.167 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 30 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:07 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:07.168 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:08 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:08.159 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:10 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:10.239 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 30 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:11 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:11.231 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 33 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:40 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:40.287 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error>
  Apr 01 19:48:40 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:40.288 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:44 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:44.384 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:44 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:44.385 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:44 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:44.386 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:47 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:47.455 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:47 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:47.456 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 33 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:48 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:48.483 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 36 seconds.: OSError: [Errno 113] EHOSTUNREACH


  My compute nova.conf looks like following

  [oslo_messaging_rabbit]
  rabbit_retry_interval = 20
  rabbit_retry_backoff = 3
  rabbit_interval_max = 60
  rabbit_transient_queues_ttl = 300
  rabbit_qos_prefetch_count = 100
  rpc_conn_pool_size = 300

  
  Feel like something crashed in oslo driver and not able to recover itself. I have other deployment running on stein and i haven't seen this kind of issue in last 3 years so definitely this is something related to wallaby oslo bug.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1968054/+subscriptions
Follow ups

[Bug 1968054] Re: oslo.messaging._drivers.impl_rabbit Connection failed: timed out
From: Isa Fishman, 2023-04-26
[Bug 1968054] Re: oslo.messaging._drivers.impl_rabbit Connection failed: timed out
From: Sylvain Bauza, 2022-04-19
[Bug 1968054] Re: oslo.messaging._drivers.impl_rabbit Connection failed: timed out
From: Satish Patel, 2022-04-07