← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #95255

[Bug 2095598] [NEW] test_async_process_respawns functional test failing in fips functional job

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Fails as:-
ft2.2: neutron.tests.functional.agent.linux.test_async_process.TestAsyncProcess.test_async_process_respawnstesttools.testresult.real._StringException: traceback-1: {{{
Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/linux/test_async_process.py", line 46, in _safe_stop
    proc.stop()
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/common/async_process.py", line 158, in stop
    self._kill(kill_signal, kill_timeout)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/common/async_process.py", line 199, in _kill
    self._kill_process_and_wait(pid, kill_signal, kill_timeout)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/common/async_process.py", line 213, in _kill_process_and_wait
    self._process.wait(kill_timeout)
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.9/site-packages/eventlet/green/subprocess.py", line 89, in wait
    eventlet.sleep(check_interval)
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.9/site-packages/eventlet/greenthread.py", line 38, in sleep
    hub.switch()
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.9/site-packages/eventlet/hubs/hub.py", line 310, in switch
    return self.greenlet.switch()
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.9/site-packages/eventlet/hubs/hub.py", line 362, in run
    self.wait(sleep_time)
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.9/site-packages/eventlet/hubs/poll.py", line 80, in wait
    presult = self.do_poll(seconds)
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.9/site-packages/eventlet/hubs/epolls.py", line 31, in do_poll
    return self.poll.poll(seconds)
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.9/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
    raise TimeoutException()
fixtures._fixtures.timeout.TimeoutException
}}}

Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 741, in wait_until_true
    eventlet.sleep(sleep)
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.9/site-packages/eventlet/greenthread.py", line 38, in sleep
    hub.switch()
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.9/site-packages/eventlet/hubs/hub.py", line 310, in switch
    return self.greenlet.switch()
eventlet.timeout.Timeout: 5 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/linux/test_async_process.py", line 75, in test_async_process_respawns
    common_utils.wait_until_true(
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 745, in wait_until_true
    raise exception
RuntimeError: Async process didn't respawn


First failure seen 15th January:- https://zuul.openstack.org/build/9af3f49c77ea4bef91240393d8ce687e
Since then it's consistently failing and passed just once yesterday. Some external change seems made it,
just checking rpm and pip diff:-

$ diff pass_rpm fail_rpm
$ diff pass_rpm fail_rpm 
88,89c88,89
< device-mapper-1.02.202-3.el9.x86_64
< device-mapper-libs-1.02.202-3.el9.x86_64
---
> device-mapper-1.02.202-4.el9.x86_64
> device-mapper-libs-1.02.202-4.el9.x86_64
196,200c196,200
< grub2-common-2.06-93.el9.noarch
< grub2-pc-2.06-93.el9.x86_64
< grub2-pc-modules-2.06-93.el9.noarch
< grub2-tools-2.06-93.el9.x86_64
< grub2-tools-minimal-2.06-93.el9.x86_64
---
> grub2-common-2.06-94.el9.noarch
> grub2-pc-2.06-94.el9.x86_64
> grub2-pc-modules-2.06-94.el9.noarch
> grub2-tools-2.06-94.el9.x86_64
> grub2-tools-minimal-2.06-94.el9.x86_64
365c365
< libsepol-3.6-1.el9.x86_64
---
> libsepol-3.6-2.el9.x86_64
483,484c483,484
< NetworkManager-1.51.4-1.el9.x86_64
< NetworkManager-libnm-1.51.4-1.el9.x86_64
---
> NetworkManager-1.51.5-1.el9.x86_64
> NetworkManager-libnm-1.51.5-1.el9.x86_64
689,693c689,693
< systemd-252-48.el9.x86_64
< systemd-libs-252-48.el9.x86_64
< systemd-pam-252-48.el9.x86_64
< systemd-rpm-macros-252-48.el9.noarch
< systemd-udev-252-48.el9.x86_64
---
> systemd-252-49.el9.x86_64
> systemd-libs-252-49.el9.x86_64
> systemd-pam-252-49.el9.x86_64
> systemd-rpm-macros-252-49.el9.noarch
> systemd-udev-252-49.el9.x86_64


$ diff pass_pip fail_pip 
22c22
< eventlet==0.36.1
---
> eventlet==0.38.2

Builds:- https://zuul.openstack.org/builds?job_name=neutron-functional-
fips&project=openstack/neutron

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2095598

Title:
  test_async_process_respawns functional test failing in fips functional
  job

Status in neutron:
  New

Bug description:
  Fails as:-
  ft2.2: neutron.tests.functional.agent.linux.test_async_process.TestAsyncProcess.test_async_process_respawnstesttools.testresult.real._StringException: traceback-1: {{{
  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/linux/test_async_process.py", line 46, in _safe_stop
      proc.stop()
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/common/async_process.py", line 158, in stop
      self._kill(kill_signal, kill_timeout)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/common/async_process.py", line 199, in _kill
      self._kill_process_and_wait(pid, kill_signal, kill_timeout)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/common/async_process.py", line 213, in _kill_process_and_wait
      self._process.wait(kill_timeout)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.9/site-packages/eventlet/green/subprocess.py", line 89, in wait
      eventlet.sleep(check_interval)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.9/site-packages/eventlet/greenthread.py", line 38, in sleep
      hub.switch()
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.9/site-packages/eventlet/hubs/hub.py", line 310, in switch
      return self.greenlet.switch()
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.9/site-packages/eventlet/hubs/hub.py", line 362, in run
      self.wait(sleep_time)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.9/site-packages/eventlet/hubs/poll.py", line 80, in wait
      presult = self.do_poll(seconds)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.9/site-packages/eventlet/hubs/epolls.py", line 31, in do_poll
      return self.poll.poll(seconds)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.9/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
      raise TimeoutException()
  fixtures._fixtures.timeout.TimeoutException
  }}}

  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 741, in wait_until_true
      eventlet.sleep(sleep)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.9/site-packages/eventlet/greenthread.py", line 38, in sleep
      hub.switch()
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.9/site-packages/eventlet/hubs/hub.py", line 310, in switch
      return self.greenlet.switch()
  eventlet.timeout.Timeout: 5 seconds

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
      return f(self, *args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/linux/test_async_process.py", line 75, in test_async_process_respawns
      common_utils.wait_until_true(
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 745, in wait_until_true
      raise exception
  RuntimeError: Async process didn't respawn

  
  First failure seen 15th January:- https://zuul.openstack.org/build/9af3f49c77ea4bef91240393d8ce687e
  Since then it's consistently failing and passed just once yesterday. Some external change seems made it,
  just checking rpm and pip diff:-

  $ diff pass_rpm fail_rpm
  $ diff pass_rpm fail_rpm 
  88,89c88,89
  < device-mapper-1.02.202-3.el9.x86_64
  < device-mapper-libs-1.02.202-3.el9.x86_64
  ---
  > device-mapper-1.02.202-4.el9.x86_64
  > device-mapper-libs-1.02.202-4.el9.x86_64
  196,200c196,200
  < grub2-common-2.06-93.el9.noarch
  < grub2-pc-2.06-93.el9.x86_64
  < grub2-pc-modules-2.06-93.el9.noarch
  < grub2-tools-2.06-93.el9.x86_64
  < grub2-tools-minimal-2.06-93.el9.x86_64
  ---
  > grub2-common-2.06-94.el9.noarch
  > grub2-pc-2.06-94.el9.x86_64
  > grub2-pc-modules-2.06-94.el9.noarch
  > grub2-tools-2.06-94.el9.x86_64
  > grub2-tools-minimal-2.06-94.el9.x86_64
  365c365
  < libsepol-3.6-1.el9.x86_64
  ---
  > libsepol-3.6-2.el9.x86_64
  483,484c483,484
  < NetworkManager-1.51.4-1.el9.x86_64
  < NetworkManager-libnm-1.51.4-1.el9.x86_64
  ---
  > NetworkManager-1.51.5-1.el9.x86_64
  > NetworkManager-libnm-1.51.5-1.el9.x86_64
  689,693c689,693
  < systemd-252-48.el9.x86_64
  < systemd-libs-252-48.el9.x86_64
  < systemd-pam-252-48.el9.x86_64
  < systemd-rpm-macros-252-48.el9.noarch
  < systemd-udev-252-48.el9.x86_64
  ---
  > systemd-252-49.el9.x86_64
  > systemd-libs-252-49.el9.x86_64
  > systemd-pam-252-49.el9.x86_64
  > systemd-rpm-macros-252-49.el9.noarch
  > systemd-udev-252-49.el9.x86_64

  
  $ diff pass_pip fail_pip 
  22c22
  < eventlet==0.36.1
  ---
  > eventlet==0.38.2

  Builds:- https://zuul.openstack.org/builds?job_name=neutron-
  functional-fips&project=openstack/neutron

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2095598/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next