← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #74978

[Bug 1795086] [NEW] nova.tests.unit.test_profiler.TestProfiler.test_all_public_methods_are_traced sometimes does not return

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

This was reported by clarkb as seen in a gate run, where the
nova.tests.unit.test_profiler.TestProfiler.test_all_public_methods_are_traced
unit test output that it completed but the test run continued to hang
until the entire job timed out. I don't yet see how this could happen
and whether it's a nova unit test problem or possibly a osprofiler or
stestr problem.

Excerpt from a gate run [1]:

2018-09-27 02:24:50.352889 | ubuntu-xenial | py35 runtests: commands[2] | env TEST_OSPROFILER=1 stestr run --combine --no-discover nova.tests.unit.test_profiler
2018-09-27 02:24:50.353111 | ubuntu-xenial | setting PATH=/home/zuul/src/git.openstack.org/openstack/nova/.tox/py35/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
2018-09-27 02:24:50.356186 | ubuntu-xenial |   /home/zuul/src/git.openstack.org/openstack/nova$ /usr/bin/env TEST_OSPROFILER=1 stestr run --combine --no-discover nova.tests.unit.test_profiler
2018-09-27 02:24:54.683712 | ubuntu-xenial | {0} nova.tests.unit.test_profiler.TestProfiler.test_all_public_methods_are_traced [0.892519s] ... ok
2018-09-27 02:24:54.683784 | ubuntu-xenial |
2018-09-27 02:24:54.683815 | ubuntu-xenial | Captured stderr:
2018-09-27 02:24:54.683842 | ubuntu-xenial | ~~~~~~~~~~~~~~~~
2018-09-27 02:24:54.683990 | ubuntu-xenial |     b"/home/zuul/src/git.openstack.org/openstack/nova/nova/test.py:299: DeprecationWarning: Using class 'MoxStubout' (either directly or via inheritance) is deprecated in version '3.5.0'"
2018-09-27 02:24:54.684049 | ubuntu-xenial |     b'  mox_fixture = self.useFixture(moxstubout.MoxStubout())'
2018-09-27 02:24:54.684071 | ubuntu-xenial |     b''
2018-09-27 02:24:55.014484 | ubuntu-xenial | sys:1: ResourceWarning: unclosed file <_io.FileIO name=1 mode='wb' closefd=True>
2018-09-27 02:25:06.645041 | ubuntu-xenial |
2018-09-27 02:25:06.645131 | ubuntu-xenial | ======
2018-09-27 02:25:06.645155 | ubuntu-xenial | Totals
2018-09-27 02:25:06.645174 | ubuntu-xenial | ======
2018-09-27 02:25:06.645221 | ubuntu-xenial | Ran: 1 tests in 15.5792 sec.
2018-09-27 02:25:06.645246 | ubuntu-xenial |  - Passed: 1
2018-09-27 02:25:06.645278 | ubuntu-xenial |  - Skipped: 0
2018-09-27 02:25:06.645317 | ubuntu-xenial |  - Expected Fail: 0
2018-09-27 02:25:06.645348 | ubuntu-xenial |  - Unexpected Success: 0
2018-09-27 02:25:06.645370 | ubuntu-xenial |  - Failed: 0
2018-09-27 02:25:06.645411 | ubuntu-xenial | Sum of execute time for each test: 0.8925 sec.
2018-09-27 02:25:06.645427 | ubuntu-xenial |
2018-09-27 02:25:06.645450 | ubuntu-xenial | ==============
2018-09-27 02:25:06.645474 | ubuntu-xenial | Worker Balance
2018-09-27 02:25:06.645497 | ubuntu-xenial | ==============
2018-09-27 02:25:06.645548 | ubuntu-xenial |  - Worker 0 (1 tests) => 0:00:00.892519
2018-09-27 02:25:06.887450 | ubuntu-xenial | py35 finish: runtests after 497.14 seconds
2018-09-27 02:25:06.887615 | ubuntu-xenial | py35 start: run-test-post
2018-09-27 02:25:06.888016 | ubuntu-xenial | py35 finish: run-test-post after 0.00 seconds
2018-09-27 02:25:06.888085 | ubuntu-xenial | ___________________________________ summary ____________________________________
2018-09-27 02:25:06.888118 | ubuntu-xenial |   py35: commands succeeded
2018-09-27 02:25:06.888145 | ubuntu-xenial |   congratulations :)
2018-09-27 02:52:32.542929 | RUN END RESULT_TIMED_OUT: [untrusted : git.openstack.org/openstack-infra/zuul-jobs/playbooks/tox/run.yaml@master]

And I only see two hits for it in the last 7 days (not all of these
results are this bug):

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22RESULT_TIMED_OUT%3A%20%5Buntrusted%20%3A%20git.openstack.org
%2Fopenstack-infra%2Fzuul-
jobs%2Fplaybooks%2Ftox%2Frun.yaml%40master%5D%5C%22%20AND%20project%3A%5C%22openstack%2Fnova%5C%22%20AND%20build_name%3A%5C
%22openstack-tox-py35%5C%22&from=7d

so I'm setting the importance as Low.

[1] http://logs.openstack.org/06/604906/5/gate/openstack-tox-
py35/c633c4e/job-output.txt.gz#_2018-09-27_02_52_32_542929

** Affects: nova
     Importance: Low
         Status: Confirmed


** Tags: testing

** Changed in: nova
   Importance: Undecided => Low

** Changed in: nova
       Status: New => Confirmed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1795086

Title:
  nova.tests.unit.test_profiler.TestProfiler.test_all_public_methods_are_traced
  sometimes does not return

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  This was reported by clarkb as seen in a gate run, where the
  nova.tests.unit.test_profiler.TestProfiler.test_all_public_methods_are_traced
  unit test output that it completed but the test run continued to hang
  until the entire job timed out. I don't yet see how this could happen
  and whether it's a nova unit test problem or possibly a osprofiler or
  stestr problem.

  Excerpt from a gate run [1]:

  2018-09-27 02:24:50.352889 | ubuntu-xenial | py35 runtests: commands[2] | env TEST_OSPROFILER=1 stestr run --combine --no-discover nova.tests.unit.test_profiler
  2018-09-27 02:24:50.353111 | ubuntu-xenial | setting PATH=/home/zuul/src/git.openstack.org/openstack/nova/.tox/py35/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
  2018-09-27 02:24:50.356186 | ubuntu-xenial |   /home/zuul/src/git.openstack.org/openstack/nova$ /usr/bin/env TEST_OSPROFILER=1 stestr run --combine --no-discover nova.tests.unit.test_profiler
  2018-09-27 02:24:54.683712 | ubuntu-xenial | {0} nova.tests.unit.test_profiler.TestProfiler.test_all_public_methods_are_traced [0.892519s] ... ok
  2018-09-27 02:24:54.683784 | ubuntu-xenial |
  2018-09-27 02:24:54.683815 | ubuntu-xenial | Captured stderr:
  2018-09-27 02:24:54.683842 | ubuntu-xenial | ~~~~~~~~~~~~~~~~
  2018-09-27 02:24:54.683990 | ubuntu-xenial |     b"/home/zuul/src/git.openstack.org/openstack/nova/nova/test.py:299: DeprecationWarning: Using class 'MoxStubout' (either directly or via inheritance) is deprecated in version '3.5.0'"
  2018-09-27 02:24:54.684049 | ubuntu-xenial |     b'  mox_fixture = self.useFixture(moxstubout.MoxStubout())'
  2018-09-27 02:24:54.684071 | ubuntu-xenial |     b''
  2018-09-27 02:24:55.014484 | ubuntu-xenial | sys:1: ResourceWarning: unclosed file <_io.FileIO name=1 mode='wb' closefd=True>
  2018-09-27 02:25:06.645041 | ubuntu-xenial |
  2018-09-27 02:25:06.645131 | ubuntu-xenial | ======
  2018-09-27 02:25:06.645155 | ubuntu-xenial | Totals
  2018-09-27 02:25:06.645174 | ubuntu-xenial | ======
  2018-09-27 02:25:06.645221 | ubuntu-xenial | Ran: 1 tests in 15.5792 sec.
  2018-09-27 02:25:06.645246 | ubuntu-xenial |  - Passed: 1
  2018-09-27 02:25:06.645278 | ubuntu-xenial |  - Skipped: 0
  2018-09-27 02:25:06.645317 | ubuntu-xenial |  - Expected Fail: 0
  2018-09-27 02:25:06.645348 | ubuntu-xenial |  - Unexpected Success: 0
  2018-09-27 02:25:06.645370 | ubuntu-xenial |  - Failed: 0
  2018-09-27 02:25:06.645411 | ubuntu-xenial | Sum of execute time for each test: 0.8925 sec.
  2018-09-27 02:25:06.645427 | ubuntu-xenial |
  2018-09-27 02:25:06.645450 | ubuntu-xenial | ==============
  2018-09-27 02:25:06.645474 | ubuntu-xenial | Worker Balance
  2018-09-27 02:25:06.645497 | ubuntu-xenial | ==============
  2018-09-27 02:25:06.645548 | ubuntu-xenial |  - Worker 0 (1 tests) => 0:00:00.892519
  2018-09-27 02:25:06.887450 | ubuntu-xenial | py35 finish: runtests after 497.14 seconds
  2018-09-27 02:25:06.887615 | ubuntu-xenial | py35 start: run-test-post
  2018-09-27 02:25:06.888016 | ubuntu-xenial | py35 finish: run-test-post after 0.00 seconds
  2018-09-27 02:25:06.888085 | ubuntu-xenial | ___________________________________ summary ____________________________________
  2018-09-27 02:25:06.888118 | ubuntu-xenial |   py35: commands succeeded
  2018-09-27 02:25:06.888145 | ubuntu-xenial |   congratulations :)
  2018-09-27 02:52:32.542929 | RUN END RESULT_TIMED_OUT: [untrusted : git.openstack.org/openstack-infra/zuul-jobs/playbooks/tox/run.yaml@master]

  And I only see two hits for it in the last 7 days (not all of these
  results are this bug):

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22RESULT_TIMED_OUT%3A%20%5Buntrusted%20%3A%20git.openstack.org
  %2Fopenstack-infra%2Fzuul-
  jobs%2Fplaybooks%2Ftox%2Frun.yaml%40master%5D%5C%22%20AND%20project%3A%5C%22openstack%2Fnova%5C%22%20AND%20build_name%3A%5C
  %22openstack-tox-py35%5C%22&from=7d

  so I'm setting the importance as Low.

  [1] http://logs.openstack.org/06/604906/5/gate/openstack-tox-
  py35/c633c4e/job-output.txt.gz#_2018-09-27_02_52_32_542929

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1795086/+subscriptions
  Thread Previous • Date Previous • • Thread Next