← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #86738

[Bug 1938262] [NEW] [stable/ussuri] Functional jobs timeout, many tests with fixtures._fixtures.timeout.TimeoutException

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

This started recently (last backport successfully merged was on July 20th), functional tests now fail 100% on  recent backports with TIMED_OUT. For example:
https://review.opendev.org/c/openstack/neutron/+/801882
https://review.opendev.org/c/openstack/neutron/+/802528

I confirmed it with dummy change:
https://review.opendev.org/c/openstack/neutron/+/802552

Many tests fail with:
2021-07-27 17:34:20.939074 | controller |       File "/usr/lib/python3.6/threading.py", line 295, in wait
2021-07-27 17:34:20.939093 | controller |     waiter.acquire()
2021-07-27 17:34:20.939107 | controller |
2021-07-27 17:34:20.939120 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/semaphore.py", line 115, in acquire
2021-07-27 17:34:20.939134 | controller |     hubs.get_hub().switch()
2021-07-27 17:34:20.939147 | controller |
2021-07-27 17:34:20.939161 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 298, in switch
2021-07-27 17:34:20.939177 | controller |     return self.greenlet.switch()
2021-07-27 17:34:20.939191 | controller |
2021-07-27 17:34:20.939205 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 350, in run
2021-07-27 17:34:20.939219 | controller |     self.wait(sleep_time)
2021-07-27 17:34:20.939233 | controller |
2021-07-27 17:34:20.939246 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 80, in wait
2021-07-27 17:34:20.939260 | controller |     presult = self.do_poll(seconds)
2021-07-27 17:34:20.939316 | controller |
2021-07-27 17:34:20.939338 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/hubs/epolls.py", line 31, in do_poll
2021-07-27 17:34:20.939352 | controller |     return self.poll.poll(seconds)
2021-07-27 17:34:20.939366 | controller |
2021-07-27 17:34:20.939380 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
2021-07-27 17:34:20.939394 | controller |     raise TimeoutException()
2021-07-27 17:34:20.939407 | controller |
2021-07-27 17:34:20.939421 | controller |     fixtures._fixtures.timeout.TimeoutException

Start of backtrace depends on tests (namespace creation, ip_lib, ...) so
it does look like generic issue in related package

Note this is specific to stable/ussuri, the first backport mentioned
passed in newer branches and in stable/train without issue. Functional
tests are passing in train with same OS, same python version 3.6

Nothing suspicious logged in the tests functional output itself

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1938262

Title:
  [stable/ussuri] Functional jobs timeout, many tests with
  fixtures._fixtures.timeout.TimeoutException

Status in neutron:
  New

Bug description:
  This started recently (last backport successfully merged was on July 20th), functional tests now fail 100% on  recent backports with TIMED_OUT. For example:
  https://review.opendev.org/c/openstack/neutron/+/801882
  https://review.opendev.org/c/openstack/neutron/+/802528

  I confirmed it with dummy change:
  https://review.opendev.org/c/openstack/neutron/+/802552

  Many tests fail with:
  2021-07-27 17:34:20.939074 | controller |       File "/usr/lib/python3.6/threading.py", line 295, in wait
  2021-07-27 17:34:20.939093 | controller |     waiter.acquire()
  2021-07-27 17:34:20.939107 | controller |
  2021-07-27 17:34:20.939120 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/semaphore.py", line 115, in acquire
  2021-07-27 17:34:20.939134 | controller |     hubs.get_hub().switch()
  2021-07-27 17:34:20.939147 | controller |
  2021-07-27 17:34:20.939161 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 298, in switch
  2021-07-27 17:34:20.939177 | controller |     return self.greenlet.switch()
  2021-07-27 17:34:20.939191 | controller |
  2021-07-27 17:34:20.939205 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 350, in run
  2021-07-27 17:34:20.939219 | controller |     self.wait(sleep_time)
  2021-07-27 17:34:20.939233 | controller |
  2021-07-27 17:34:20.939246 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 80, in wait
  2021-07-27 17:34:20.939260 | controller |     presult = self.do_poll(seconds)
  2021-07-27 17:34:20.939316 | controller |
  2021-07-27 17:34:20.939338 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/hubs/epolls.py", line 31, in do_poll
  2021-07-27 17:34:20.939352 | controller |     return self.poll.poll(seconds)
  2021-07-27 17:34:20.939366 | controller |
  2021-07-27 17:34:20.939380 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
  2021-07-27 17:34:20.939394 | controller |     raise TimeoutException()
  2021-07-27 17:34:20.939407 | controller |
  2021-07-27 17:34:20.939421 | controller |     fixtures._fixtures.timeout.TimeoutException

  Start of backtrace depends on tests (namespace creation, ip_lib, ...)
  so it does look like generic issue in related package

  Note this is specific to stable/ussuri, the first backport mentioned
  passed in newer branches and in stable/train without issue. Functional
  tests are passing in train with same OS, same python version 3.6

  Nothing suspicious logged in the tests functional output itself

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1938262/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next