← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #93984

[Bug 2065821] [NEW] cover job started to fail with Killed

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Pipeline here:
https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-
cover&project=openstack/neutron

First failure is May 14:
https://zuul.opendev.org/t/openstack/build/6899085a449248ed8b017eb4e9f231ab

In logs, it looks like this:

2024-05-14 16:33:32.050334 | ubuntu-jammy | Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
2024-05-14 16:33:32.050424 | ubuntu-jammy |   declare_namespace(pkg)
2024-05-14 16:33:32.050451 | ubuntu-jammy | /home/zuul/src/opendev.org/openstack/neutron/.tox/cover/lib/python3.10/site-packages/pkg_resources/__init__.py:2832: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('repoze')`.
2024-05-14 16:33:32.050472 | ubuntu-jammy | Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
2024-05-14 16:33:32.050490 | ubuntu-jammy |   declare_namespace(pkg)
2024-05-14 16:33:32.050516 | ubuntu-jammy | /home/zuul/src/opendev.org/openstack/neutron/.tox/cover/lib/python3.10/site-packages/pkg_resources/__init__.py:2832: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('repoze')`.
2024-05-14 16:59:58.794881 | ubuntu-jammy | Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
2024-05-14 16:59:58.796083 | ubuntu-jammy |   declare_namespace(pkg)
2024-05-14 16:59:58.796171 | ubuntu-jammy | Killed
2024-05-14 17:03:29.030113 | ubuntu-jammy | Ran 20812 tests in 1777.707s
2024-05-14 17:03:29.174365 | ubuntu-jammy | FAILED (id=0, failures=1, skips=1701)

Could it be that the job no longer has enough memory and gets OOM
killed?

I've compared versions of packages updated between older good and newer
bad runs, and I only see these bumped: sqlalchemy 1.4.51 -> 2.0.29 and
alembic 1.9.4 -> 1.13.1.

Different runs have different unit tests reported as failed (all failed
runs claim a single test case failed).

Examples of different failed tests:

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_218/917262/3/check/openstack-
tox-cover/2180cc3/testr_results.html -
neutron.tests.unit.services.network_segment_range.test_plugin.TestNetworkSegmentRange.test_delete_network_segment_range_failed_with_segment_referenced

https://9b86ab5bbc6be76c9905-30f46d6ec556e6b2dd47ea35fedbb1ac.ssl.cf5.rackcdn.com/919699/4/check/openstack-
tox-cover/ce9baa9/testr_results.html -
neutron.tests.unit.services.ovn_l3.test_plugin.OVNL3ExtrarouteTests
.test_floatingip_update_different_port_owner_as_admin

https://6eed35a50c35f284b4d2-bf433abff5f8b85f7f80257b72ac6f67.ssl.cf2.rackcdn.com/919632/1/check/openstack-
tox-cover/3b1c5fa/testr_results.html -
neutron.tests.unit.services.placement_report.test_plugin.PlacementReportPluginTestCases.test__sync_placement_state_legacy

I suspect specific unit test cases are not relevant - the test runner
process dies for some reason and whatever the test it was running at
that moment gets reported as failed.

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: gate-failure

** Tags added: gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2065821

Title:
  cover job started to fail with Killed

Status in neutron:
  New

Bug description:
  Pipeline here:
  https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-
  cover&project=openstack/neutron

  First failure is May 14:
  https://zuul.opendev.org/t/openstack/build/6899085a449248ed8b017eb4e9f231ab

  In logs, it looks like this:

  2024-05-14 16:33:32.050334 | ubuntu-jammy | Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  2024-05-14 16:33:32.050424 | ubuntu-jammy |   declare_namespace(pkg)
  2024-05-14 16:33:32.050451 | ubuntu-jammy | /home/zuul/src/opendev.org/openstack/neutron/.tox/cover/lib/python3.10/site-packages/pkg_resources/__init__.py:2832: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('repoze')`.
  2024-05-14 16:33:32.050472 | ubuntu-jammy | Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  2024-05-14 16:33:32.050490 | ubuntu-jammy |   declare_namespace(pkg)
  2024-05-14 16:33:32.050516 | ubuntu-jammy | /home/zuul/src/opendev.org/openstack/neutron/.tox/cover/lib/python3.10/site-packages/pkg_resources/__init__.py:2832: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('repoze')`.
  2024-05-14 16:59:58.794881 | ubuntu-jammy | Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  2024-05-14 16:59:58.796083 | ubuntu-jammy |   declare_namespace(pkg)
  2024-05-14 16:59:58.796171 | ubuntu-jammy | Killed
  2024-05-14 17:03:29.030113 | ubuntu-jammy | Ran 20812 tests in 1777.707s
  2024-05-14 17:03:29.174365 | ubuntu-jammy | FAILED (id=0, failures=1, skips=1701)

  Could it be that the job no longer has enough memory and gets OOM
  killed?

  I've compared versions of packages updated between older good and
  newer bad runs, and I only see these bumped: sqlalchemy 1.4.51 ->
  2.0.29 and alembic 1.9.4 -> 1.13.1.

  Different runs have different unit tests reported as failed (all
  failed runs claim a single test case failed).

  Examples of different failed tests:

  https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_218/917262/3/check/openstack-
  tox-cover/2180cc3/testr_results.html -
  neutron.tests.unit.services.network_segment_range.test_plugin.TestNetworkSegmentRange.test_delete_network_segment_range_failed_with_segment_referenced

  https://9b86ab5bbc6be76c9905-30f46d6ec556e6b2dd47ea35fedbb1ac.ssl.cf5.rackcdn.com/919699/4/check/openstack-
  tox-cover/ce9baa9/testr_results.html -
  neutron.tests.unit.services.ovn_l3.test_plugin.OVNL3ExtrarouteTests
  .test_floatingip_update_different_port_owner_as_admin

  https://6eed35a50c35f284b4d2-bf433abff5f8b85f7f80257b72ac6f67.ssl.cf2.rackcdn.com/919632/1/check/openstack-
  tox-cover/3b1c5fa/testr_results.html -
  neutron.tests.unit.services.placement_report.test_plugin.PlacementReportPluginTestCases.test__sync_placement_state_legacy

  I suspect specific unit test cases are not relevant - the test runner
  process dies for some reason and whatever the test it was running at
  that moment gets reported as failed.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2065821/+subscriptions
  Thread Previous • Date Previous • • Thread Next