← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #96113

[Bug 2115980] [NEW] Nova multi-node jobs are failing as second compute cannot connect to rabbit

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Recent failure:
https://zuul.opendev.org/t/openstack/build/beb7c6f697bf4c1eb9752ee4e6fed1f2

https://zuul.opendev.org/t/openstack/build/beb7c6f697bf4c1eb9752ee4e6fed1f2/log/compute1/logs/screen-
n-cpu.txt#1328

Jul 04 11:53:21.361972 np386776bf5ad74 nova-compute[41485]: ERROR
oslo.messaging._drivers.impl_rabbit [None
req-4d1d00d9-92c7-4ced-86e0-b1d0cc44c8bc None None] Connection failed:
timed out (retrying in 31.0 seconds): TimeoutError: timed out


It is failing since 2025-07-01T11:12:19 and it is always compute1 that fails to connect the compute service on the controller node can connect to rabbit.


❯ logsearch log --job-group nova-devstack --result FAILURE --file controller/logs/screen-n-cpu.txt --file compute1/logs/screen-n-cpu.txt  --days 7 'Connection failed: timed out .retrying in 31.0 seconds.'

Builds with matching logs 36/211:
+----------------------------------+---------------------+----------------+----------+-----------------------------------+--------+------------------------+
| uuid                             | finished            | project        | pipeline | review                            | branch | job                    |
+----------------------------------+---------------------+----------------+----------+-----------------------------------+--------+------------------------+
| beb7c6f697bf4c1eb9752ee4e6fed1f2 | 2025-07-04T11:59:13 | openstack/nova | check    | https://review.opendev.org/952282 | master | nova-next              |
| 01f1c2c124d341d59670da90f5705acd | 2025-07-03T18:02:55 | openstack/nova | check    | https://review.opendev.org/948076 | master | nova-grenade-multinode |
| 5a9064b84094476fa5bcdbd658e85d0f | 2025-07-03T19:37:35 | openstack/nova | gate     | https://review.opendev.org/940508 | master | nova-next              |
| 77e40958cb3041b6ace884ebb3dc9270 | 2025-07-03T19:17:16 | openstack/nova | gate     | https://review.opendev.org/940508 | master | nova-multi-cell        |
| 8cb281a99de14fb98cd7043c8103b997 | 2025-07-03T17:33:48 | openstack/nova | gate     | https://review.opendev.org/952283 | master | nova-grenade-multinode |
| 66557ea92bae4e809c09526eba619c07 | 2025-07-03T16:22:57 | openstack/nova | check    | https://review.opendev.org/948079 | master | nova-next              |
| c3601c4a41b04f0eac25f312fb808b13 | 2025-07-03T16:20:29 | openstack/nova | check    | https://review.opendev.org/951957 | master | nova-next              |
| 860170b59a244a569b1645587f5c499a | 2025-07-02T21:32:11 | openstack/nova | check    | https://review.opendev.org/940770 | master | nova-multi-cell        |
| 8603d4f4ca464477b760412907a6e2b8 | 2025-07-02T21:52:44 | openstack/nova | check    | https://review.opendev.org/950643 | master | nova-multi-cell        |
| 5c6afb98f32842d0b7ff662322b53365 | 2025-07-02T19:00:55 | openstack/nova | check    | https://review.opendev.org/948340 | master | nova-multi-cell        |
| 30c9b97130554a97bb1f0193b9c0c817 | 2025-07-02T17:55:57 | openstack/nova | check    | https://review.opendev.org/948186 | master | nova-grenade-multinode |
| 3e306bc27af0466fa1e2dd7956275862 | 2025-07-02T18:01:50 | openstack/nova | check    | https://review.opendev.org/948187 | master | nova-grenade-multinode |
| 17102220c090463bb4094119f1f0b7ae | 2025-07-02T19:09:52 | openstack/nova | check    | https://review.opendev.org/948311 | master | nova-multi-cell        |
| 44f4b5d5575149c2967e7ebf728f91c5 | 2025-07-02T17:40:44 | openstack/nova | check    | https://review.opendev.org/948079 | master | nova-grenade-multinode |
| 947f3f05eb214692b7bf9951cf04875d | 2025-07-02T19:51:06 | openstack/nova | check    | https://review.opendev.org/948079 | master | nova-next              |
| 406d88ca4b074fa3b0d6dc825d39ae83 | 2025-07-02T18:36:30 | openstack/nova | check    | https://review.opendev.org/948079 | master | nova-multi-cell        |
| 99a1ba90c43a46eb80d24aa6fa42359f | 2025-07-02T19:11:22 | openstack/nova | check    | https://review.opendev.org/949754 | master | nova-next              |
| 0174eb89649346c5be4374c3ab37f76c | 2025-07-02T19:41:38 | openstack/nova | check    | https://review.opendev.org/948311 | master | nova-next              |
| 1e7c62a5b2f54ca884803ea202fa54ac | 2025-07-02T17:35:03 | openstack/nova | check    | https://review.opendev.org/953815 | master | nova-grenade-multinode |
| 8bf4e36445d94abdb7cc3f4d984823d5 | 2025-07-02T17:31:48 | openstack/nova | check    | https://review.opendev.org/948086 | master | nova-grenade-multinode |
| 336b33d030ed4835b7c0c6d0ac3980cb | 2025-07-02T17:29:25 | openstack/nova | check    | https://review.opendev.org/953338 | master | nova-grenade-multinode |
| eacaf25f04a6455eac504957fe6497b8 | 2025-07-02T17:08:47 | openstack/nova | check    | https://review.opendev.org/948087 | master | nova-grenade-multinode |
| e5c2fac9eefa4cc786c4087b45ee3f1c | 2025-07-02T19:25:00 | openstack/nova | check    | https://review.opendev.org/948186 | master | nova-next              |
| 0fc3b1ccc5ac4d168abe9bc657195272 | 2025-07-02T16:27:41 | openstack/nova | check    | https://review.opendev.org/948072 | master | nova-next              |
| 55cba550524f4a5c8b9b724d90f24b30 | 2025-07-02T13:24:46 | openstack/nova | check    | https://review.opendev.org/948187 | master | nova-grenade-multinode |
| 89393b4964944033a191e636ead84783 | 2025-07-01T21:26:32 | openstack/nova | check    | https://review.opendev.org/940508 | master | nova-multi-cell        |
| b5517cb864b446f2912cb12aa59b0cf3 | 2025-07-01T19:19:03 | openstack/nova | check    | https://review.opendev.org/940508 | master | nova-next              |
| 2d7e48fa335a4d8f93a4df394f634e33 | 2025-07-01T17:45:38 | openstack/nova | check    | https://review.opendev.org/940508 | master | nova-multi-cell        |
| 64b4d81acef6427794197dfc5e9d71dc | 2025-07-01T13:35:41 | openstack/nova | check    | https://review.opendev.org/948311 | master | nova-next              |
| 24a9a891c7854df190b0bb8eed67312a | 2025-07-01T12:54:41 | openstack/nova | check    | https://review.opendev.org/950994 | master | nova-multi-cell        |
| 080485a9eb8d45b8ac91f2ec1ac59663 | 2025-07-01T13:22:21 | openstack/nova | check    | https://review.opendev.org/948086 | master | nova-next              |
| 0361c38ab1964ac385727bf311594ac1 | 2025-07-01T12:18:16 | openstack/nova | check    | https://review.opendev.org/948311 | master | nova-multi-cell        |
| c7169554320b4eb2970e2d93774cf75f | 2025-07-01T12:21:18 | openstack/nova | check    | https://review.opendev.org/952666 | master | nova-next              |
| b2f0dc55e7cb4bb6822303a45618873f | 2025-07-01T12:29:05 | openstack/nova | check    | https://review.opendev.org/948087 | master | nova-next              |
| d2ca59dba76549f58768b688d074e7c4 | 2025-07-01T11:56:50 | openstack/nova | check    | https://review.opendev.org/948186 | master | nova-multi-cell        |
| 6890c3bdc3ef400a8c561c03e659047f | 2025-07-01T11:12:19 | openstack/nova | check    | https://review.opendev.org/948076 | master | nova-multi-cell        |
+----------------------------------+---------------------+----------------+----------+-----------------------------------+--------+------------------------+

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: gate-failure

** Tags added: gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2115980

Title:
  Nova multi-node jobs are failing as second compute cannot connect to
  rabbit

Status in OpenStack Compute (nova):
  New

Bug description:
  Recent failure:
  https://zuul.opendev.org/t/openstack/build/beb7c6f697bf4c1eb9752ee4e6fed1f2

  https://zuul.opendev.org/t/openstack/build/beb7c6f697bf4c1eb9752ee4e6fed1f2/log/compute1/logs/screen-
  n-cpu.txt#1328

  Jul 04 11:53:21.361972 np386776bf5ad74 nova-compute[41485]: ERROR
  oslo.messaging._drivers.impl_rabbit [None
  req-4d1d00d9-92c7-4ced-86e0-b1d0cc44c8bc None None] Connection failed:
  timed out (retrying in 31.0 seconds): TimeoutError: timed out

  
  It is failing since 2025-07-01T11:12:19 and it is always compute1 that fails to connect the compute service on the controller node can connect to rabbit.

  
  ❯ logsearch log --job-group nova-devstack --result FAILURE --file controller/logs/screen-n-cpu.txt --file compute1/logs/screen-n-cpu.txt  --days 7 'Connection failed: timed out .retrying in 31.0 seconds.'

  Builds with matching logs 36/211:
  +----------------------------------+---------------------+----------------+----------+-----------------------------------+--------+------------------------+
  | uuid                             | finished            | project        | pipeline | review                            | branch | job                    |
  +----------------------------------+---------------------+----------------+----------+-----------------------------------+--------+------------------------+
  | beb7c6f697bf4c1eb9752ee4e6fed1f2 | 2025-07-04T11:59:13 | openstack/nova | check    | https://review.opendev.org/952282 | master | nova-next              |
  | 01f1c2c124d341d59670da90f5705acd | 2025-07-03T18:02:55 | openstack/nova | check    | https://review.opendev.org/948076 | master | nova-grenade-multinode |
  | 5a9064b84094476fa5bcdbd658e85d0f | 2025-07-03T19:37:35 | openstack/nova | gate     | https://review.opendev.org/940508 | master | nova-next              |
  | 77e40958cb3041b6ace884ebb3dc9270 | 2025-07-03T19:17:16 | openstack/nova | gate     | https://review.opendev.org/940508 | master | nova-multi-cell        |
  | 8cb281a99de14fb98cd7043c8103b997 | 2025-07-03T17:33:48 | openstack/nova | gate     | https://review.opendev.org/952283 | master | nova-grenade-multinode |
  | 66557ea92bae4e809c09526eba619c07 | 2025-07-03T16:22:57 | openstack/nova | check    | https://review.opendev.org/948079 | master | nova-next              |
  | c3601c4a41b04f0eac25f312fb808b13 | 2025-07-03T16:20:29 | openstack/nova | check    | https://review.opendev.org/951957 | master | nova-next              |
  | 860170b59a244a569b1645587f5c499a | 2025-07-02T21:32:11 | openstack/nova | check    | https://review.opendev.org/940770 | master | nova-multi-cell        |
  | 8603d4f4ca464477b760412907a6e2b8 | 2025-07-02T21:52:44 | openstack/nova | check    | https://review.opendev.org/950643 | master | nova-multi-cell        |
  | 5c6afb98f32842d0b7ff662322b53365 | 2025-07-02T19:00:55 | openstack/nova | check    | https://review.opendev.org/948340 | master | nova-multi-cell        |
  | 30c9b97130554a97bb1f0193b9c0c817 | 2025-07-02T17:55:57 | openstack/nova | check    | https://review.opendev.org/948186 | master | nova-grenade-multinode |
  | 3e306bc27af0466fa1e2dd7956275862 | 2025-07-02T18:01:50 | openstack/nova | check    | https://review.opendev.org/948187 | master | nova-grenade-multinode |
  | 17102220c090463bb4094119f1f0b7ae | 2025-07-02T19:09:52 | openstack/nova | check    | https://review.opendev.org/948311 | master | nova-multi-cell        |
  | 44f4b5d5575149c2967e7ebf728f91c5 | 2025-07-02T17:40:44 | openstack/nova | check    | https://review.opendev.org/948079 | master | nova-grenade-multinode |
  | 947f3f05eb214692b7bf9951cf04875d | 2025-07-02T19:51:06 | openstack/nova | check    | https://review.opendev.org/948079 | master | nova-next              |
  | 406d88ca4b074fa3b0d6dc825d39ae83 | 2025-07-02T18:36:30 | openstack/nova | check    | https://review.opendev.org/948079 | master | nova-multi-cell        |
  | 99a1ba90c43a46eb80d24aa6fa42359f | 2025-07-02T19:11:22 | openstack/nova | check    | https://review.opendev.org/949754 | master | nova-next              |
  | 0174eb89649346c5be4374c3ab37f76c | 2025-07-02T19:41:38 | openstack/nova | check    | https://review.opendev.org/948311 | master | nova-next              |
  | 1e7c62a5b2f54ca884803ea202fa54ac | 2025-07-02T17:35:03 | openstack/nova | check    | https://review.opendev.org/953815 | master | nova-grenade-multinode |
  | 8bf4e36445d94abdb7cc3f4d984823d5 | 2025-07-02T17:31:48 | openstack/nova | check    | https://review.opendev.org/948086 | master | nova-grenade-multinode |
  | 336b33d030ed4835b7c0c6d0ac3980cb | 2025-07-02T17:29:25 | openstack/nova | check    | https://review.opendev.org/953338 | master | nova-grenade-multinode |
  | eacaf25f04a6455eac504957fe6497b8 | 2025-07-02T17:08:47 | openstack/nova | check    | https://review.opendev.org/948087 | master | nova-grenade-multinode |
  | e5c2fac9eefa4cc786c4087b45ee3f1c | 2025-07-02T19:25:00 | openstack/nova | check    | https://review.opendev.org/948186 | master | nova-next              |
  | 0fc3b1ccc5ac4d168abe9bc657195272 | 2025-07-02T16:27:41 | openstack/nova | check    | https://review.opendev.org/948072 | master | nova-next              |
  | 55cba550524f4a5c8b9b724d90f24b30 | 2025-07-02T13:24:46 | openstack/nova | check    | https://review.opendev.org/948187 | master | nova-grenade-multinode |
  | 89393b4964944033a191e636ead84783 | 2025-07-01T21:26:32 | openstack/nova | check    | https://review.opendev.org/940508 | master | nova-multi-cell        |
  | b5517cb864b446f2912cb12aa59b0cf3 | 2025-07-01T19:19:03 | openstack/nova | check    | https://review.opendev.org/940508 | master | nova-next              |
  | 2d7e48fa335a4d8f93a4df394f634e33 | 2025-07-01T17:45:38 | openstack/nova | check    | https://review.opendev.org/940508 | master | nova-multi-cell        |
  | 64b4d81acef6427794197dfc5e9d71dc | 2025-07-01T13:35:41 | openstack/nova | check    | https://review.opendev.org/948311 | master | nova-next              |
  | 24a9a891c7854df190b0bb8eed67312a | 2025-07-01T12:54:41 | openstack/nova | check    | https://review.opendev.org/950994 | master | nova-multi-cell        |
  | 080485a9eb8d45b8ac91f2ec1ac59663 | 2025-07-01T13:22:21 | openstack/nova | check    | https://review.opendev.org/948086 | master | nova-next              |
  | 0361c38ab1964ac385727bf311594ac1 | 2025-07-01T12:18:16 | openstack/nova | check    | https://review.opendev.org/948311 | master | nova-multi-cell        |
  | c7169554320b4eb2970e2d93774cf75f | 2025-07-01T12:21:18 | openstack/nova | check    | https://review.opendev.org/952666 | master | nova-next              |
  | b2f0dc55e7cb4bb6822303a45618873f | 2025-07-01T12:29:05 | openstack/nova | check    | https://review.opendev.org/948087 | master | nova-next              |
  | d2ca59dba76549f58768b688d074e7c4 | 2025-07-01T11:56:50 | openstack/nova | check    | https://review.opendev.org/948186 | master | nova-multi-cell        |
  | 6890c3bdc3ef400a8c561c03e659047f | 2025-07-01T11:12:19 | openstack/nova | check    | https://review.opendev.org/948076 | master | nova-multi-cell        |
  +----------------------------------+---------------------+----------------+----------+-----------------------------------+--------+------------------------+

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2115980/+subscriptions
  Thread Previous • Date Previous • • Thread Next