yahoo-eng-team team mailing list archive

[Balazs Gibizer <2006467@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Tests trying to ssh into the guest fails intermittently with timeout as
udhcpc fails in the guest:

2023-02-01 20:46:32.286979 | controller | Starting network: udhcpc:
started, v1.29.3

2023-02-01 20:46:32.286987 | controller | udhcp

2023-02-01 20:46:32.286996 | controller |     c: sending discover

2023-02-01 20:46:32.287004 | controller | udhcpc: sending discover

2023-02-01 20:46:32.287013 | controller | udhcpc: sending discover

2023-02-01 20:46:32.287022 | controller | Usage: /sbin/cirros-dhcpc
<up|down>

2023-02-01 20:46:32.287030 | controller | udhcpc: no lease, failing

2023-02-01 20:46:32.287039 | controller | FAIL


Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 70, in wrapper
    return f(*func_args, **func_kwargs)
  File "/opt/stack/tempest/tempest/api/compute/admin/test_volumes_negative.py", line 128, in test_multiattach_rw_volume_update_failure
    server1 = self.create_test_server(
  File "/opt/stack/tempest/tempest/api/compute/base.py", line 272, in create_test_server
    body, servers = compute.create_test_server(
  File "/opt/stack/tempest/tempest/common/compute.py", line 334, in create_test_server
    with excutils.save_and_reraise_exception():
  File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
  File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
  File "/opt/stack/tempest/tempest/common/compute.py", line 329, in create_test_server
    wait_for_ssh_or_ping(
  File "/opt/stack/tempest/tempest/common/compute.py", line 148, in wait_for_ssh_or_ping
    waiters.wait_for_ssh(
  File "/opt/stack/tempest/tempest/common/waiters.py", line 632, in wait_for_ssh
    raise lib_exc.TimeoutException()
tempest.lib.exceptions.TimeoutException: Request timed out
Details: None

Example failure
https://zuul.opendev.org/t/openstack/build/f1c6b7e54b28415c952de0be833731a9/logs

Signature
$ logsearch log --job-group nova-devstack  --result FAILURE 'udhcpc: no lease, failing' --days 7
[snip]
Builds with matching logs 6/138:
+----------------------------------+---------------------+----------------+----------+-----------------------------------+----------------+--------------------------------------------+
| uuid                             | finished            | project        | pipeline | review                            | branch         | job                                        |
+----------------------------------+---------------------+----------------+----------+-----------------------------------+----------------+--------------------------------------------+
| 9bd5d568bfa84c119470df9fbff2de0b | 2023-02-03T12:36:54 | openstack/nova | check    | https://review.opendev.org/857339 | master         | nova-next                                  |
| 3fae6edffe68483fa2627bc40002f524 | 2023-02-02T13:52:04 | openstack/nova | check    | https://review.opendev.org/860285 | master         | nova-next                                  |
| 70eeeb8eb3184d8d9ee802ee53cb979b | 2023-02-02T13:33:57 | openstack/nova | check    | https://review.opendev.org/860287 | master         | nova-next                                  |
| 492821b715974ae389c5d7f9127bb5c3 | 2023-02-02T05:14:11 | openstack/nova | check    | https://review.opendev.org/871798 | stable/wallaby | tempest-integrated-compute-centos-8-stream |
| f1c6b7e54b28415c952de0be833731a9 | 2023-02-01T21:34:36 | openstack/nova | gate     | https://review.opendev.org/872220 | master         | nova-next                                  |
| cca45d74a56f4204a299ee4bbbaad59d | 2023-02-01T06:17:04 | openstack/nova | check    | https://review.opendev.org/871557 | stable/wallaby | tempest-integrated-compute-centos-8-stream |
+----------------------------------+---------------------+----------------+----------+-----------------------------------+----------------+--------------------------------------------+

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: gate-failure

** Tags added: gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2006467

Title:
  tempest ssh timeout due to udhcpc fails in the cirros guest

Status in OpenStack Compute (nova):
  New

Bug description:
  Tests trying to ssh into the guest fails intermittently with timeout
  as udhcpc fails in the guest:

  2023-02-01 20:46:32.286979 | controller | Starting network: udhcpc:
  started, v1.29.3

  2023-02-01 20:46:32.286987 | controller | udhcp

  2023-02-01 20:46:32.286996 | controller |     c: sending discover

  2023-02-01 20:46:32.287004 | controller | udhcpc: sending discover

  2023-02-01 20:46:32.287013 | controller | udhcpc: sending discover

  2023-02-01 20:46:32.287022 | controller | Usage: /sbin/cirros-dhcpc
  <up|down>

  2023-02-01 20:46:32.287030 | controller | udhcpc: no lease, failing

  2023-02-01 20:46:32.287039 | controller | FAIL

  
  Traceback (most recent call last):
    File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 70, in wrapper
      return f(*func_args, **func_kwargs)
    File "/opt/stack/tempest/tempest/api/compute/admin/test_volumes_negative.py", line 128, in test_multiattach_rw_volume_update_failure
      server1 = self.create_test_server(
    File "/opt/stack/tempest/tempest/api/compute/base.py", line 272, in create_test_server
      body, servers = compute.create_test_server(
    File "/opt/stack/tempest/tempest/common/compute.py", line 334, in create_test_server
      with excutils.save_and_reraise_exception():
    File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/opt/stack/tempest/tempest/common/compute.py", line 329, in create_test_server
      wait_for_ssh_or_ping(
    File "/opt/stack/tempest/tempest/common/compute.py", line 148, in wait_for_ssh_or_ping
      waiters.wait_for_ssh(
    File "/opt/stack/tempest/tempest/common/waiters.py", line 632, in wait_for_ssh
      raise lib_exc.TimeoutException()
  tempest.lib.exceptions.TimeoutException: Request timed out
  Details: None

  Example failure
  https://zuul.opendev.org/t/openstack/build/f1c6b7e54b28415c952de0be833731a9/logs

  Signature
  $ logsearch log --job-group nova-devstack  --result FAILURE 'udhcpc: no lease, failing' --days 7
  [snip]
  Builds with matching logs 6/138:
  +----------------------------------+---------------------+----------------+----------+-----------------------------------+----------------+--------------------------------------------+
  | uuid                             | finished            | project        | pipeline | review                            | branch         | job                                        |
  +----------------------------------+---------------------+----------------+----------+-----------------------------------+----------------+--------------------------------------------+
  | 9bd5d568bfa84c119470df9fbff2de0b | 2023-02-03T12:36:54 | openstack/nova | check    | https://review.opendev.org/857339 | master         | nova-next                                  |
  | 3fae6edffe68483fa2627bc40002f524 | 2023-02-02T13:52:04 | openstack/nova | check    | https://review.opendev.org/860285 | master         | nova-next                                  |
  | 70eeeb8eb3184d8d9ee802ee53cb979b | 2023-02-02T13:33:57 | openstack/nova | check    | https://review.opendev.org/860287 | master         | nova-next                                  |
  | 492821b715974ae389c5d7f9127bb5c3 | 2023-02-02T05:14:11 | openstack/nova | check    | https://review.opendev.org/871798 | stable/wallaby | tempest-integrated-compute-centos-8-stream |
  | f1c6b7e54b28415c952de0be833731a9 | 2023-02-01T21:34:36 | openstack/nova | gate     | https://review.opendev.org/872220 | master         | nova-next                                  |
  | cca45d74a56f4204a299ee4bbbaad59d | 2023-02-01T06:17:04 | openstack/nova | check    | https://review.opendev.org/871557 | stable/wallaby | tempest-integrated-compute-centos-8-stream |
  +----------------------------------+---------------------+----------------+----------+-----------------------------------+----------------+--------------------------------------------+

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2006467/+subscriptions