yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1968054] Re: oslo.messaging._drivers.impl_rabbit Connection failed: timed out

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sylvain Bauza <1968054@xxxxxxxxxxxxxxxxxx>
Date: Tue, 19 Apr 2022 09:25:57 -0000
Reply-to: Bug 1968054 <1968054@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Unfortunately, this doesn't look a Nova issue : this is either an oslo.messaging bug or rather a configuration issue.
Closing this bug for Nova.

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1968054

Title:
  oslo.messaging._drivers.impl_rabbit Connection failed: timed out

Status in OpenStack Compute (nova):
  Invalid
Status in oslo.messaging:
  New

Bug description:
  I am running Wallaby Release on Ubuntu 20.04 (Openstack-Ansible
  deployment tool)

  oslo.messaging=12.7.1
  nova=23.1.1

  since i upgrade to Wallaby i have started noticed following error
  message very frequently in nova-compute and solution is to restart
  nova-compute agent.

  Here is the full logs:
  https://paste.opendev.org/show/bft9znewTxyXHkvIcQO0/

  
  01 19:43:36 compute1.example.net nova-compute[1546242]: AssertionError:
  Apr 01 19:45:35 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:35.059 34090 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 110] Connection timed out
  Apr 01 19:45:40 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:40.063 34090 ERROR oslo.messaging._drivers.impl_rabbit [req-707abbfe-8ee0-4af7-900a-e43dc5dec597 - - - - -] [7d350e59-001f-4203-bd41-369650cd5c5c] AMQP server on 172.28.17.24:5671 is unreachable: . Trying again in 1 seconds.: socket.timeout
  Apr 01 19:45:40 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:40.079 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: timed out (retrying in 0 seconds): socket.timeout: timed out
  Apr 01 19:45:41 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:41.983 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 0 seconds): OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:42.367 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 2.0 seconds): OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]: Traceback (most recent call last):
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]:   File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]:     timer()
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]:   File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/timer.py", line 59, in __call__
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]:     cb(*args, **kw)
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]:   File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/semaphore.py", line 152, in _do_acquire
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]:     waiter.switch()
  Apr 01 19:45:42 compute1.example.net nova-compute[34090]: greenlet.error: cannot switch to a different thread
  Apr 01 19:45:49 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:49.388 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: timed out (retrying in 0 seconds): socket.timeout: timed out
  Apr 01 19:45:50 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:50.303 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] No route to host. Trying again in 1 seconds.: OSError: [Errno 113] No route to host
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:51.199 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 0 seconds): OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:51.583 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]: Traceback (most recent call last):
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]:   File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]:     timer()
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]:   File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/timer.py", line 59, in __call__
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]:     cb(*args, **kw)
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]:   File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/semaphore.py", line 152, in _do_acquire
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]:     waiter.switch()
  Apr 01 19:45:51 compute1.example.net nova-compute[34090]: greenlet.error: cannot switch to a different thread
  Apr 01 19:45:57 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:57.601 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: timed out. Trying again in 1 seconds.: socket.timeout: timed out
  Apr 01 19:46:00 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:00.415 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 21 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:18 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:18.464 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 110] Connection timed out. Trying again in 1 seconds.: TimeoutError: [Errno 110] Connection timed out
  Apr 01 19:46:22 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:22.559 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:24 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:24.384 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error>
  Apr 01 19:46:24 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:24.384 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:25 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:25.631 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:26 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:26.802 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: . Trying again in 1 seconds.: socket.timeout
  Apr 01 19:46:27 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:27.455 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:28 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:28.063 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 24 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:28 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:28.064 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:30 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:30.527 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:31 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:31.135 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:31 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:31.775 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 21 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:35 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:35.199 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 21 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:52 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:52.575 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:53 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:53.599 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:53 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:53.599 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:56 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:56.671 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 27 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:57 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:57.663 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:57 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:57.664 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:59 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:59.743 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 24 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:46:59 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:59.744 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:02 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:02.816 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 24 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:25 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:25.791 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:26 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:26.783 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:28 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:28.864 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:29 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:29.855 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:29 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:29.856 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:31 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:31.935 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 30 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:31 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:31.937 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 27 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:31 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:31.939 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:47:35 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:35.007 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 27 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:01 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:01.023 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:05 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:05.055 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:05 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:05.056 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:05 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:05.057 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:07 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:07.167 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 30 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:07 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:07.168 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:08 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:08.159 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:10 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:10.239 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 30 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:11 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:11.231 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 33 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:40 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:40.287 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error>
  Apr 01 19:48:40 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:40.288 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:44 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:44.384 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:44 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:44.385 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:44 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:44.386 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:47 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:47.455 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:47 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:47.456 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 33 seconds.: OSError: [Errno 113] EHOSTUNREACH
  Apr 01 19:48:48 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:48.483 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 36 seconds.: OSError: [Errno 113] EHOSTUNREACH


  My compute nova.conf looks like following

  [oslo_messaging_rabbit]
  rabbit_retry_interval = 20
  rabbit_retry_backoff = 3
  rabbit_interval_max = 60
  rabbit_transient_queues_ttl = 300
  rabbit_qos_prefetch_count = 100
  rpc_conn_pool_size = 300

  
  Feel like something crashed in oslo driver and not able to recover itself. I have other deployment running on stein and i haven't seen this kind of issue in last 3 years so definitely this is something related to wallaby oslo bug.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1968054/+subscriptions
References

[Bug 1968054] [NEW] oslo.messaging._drivers.impl_rabbit Connection failed: timed out
From: Satish Patel, 2022-04-06