yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1993502] [NEW] failing unit tests when not running them all

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Thomas Goirand <1993502@xxxxxxxxxxxxxxxxxx>
Date: Wed, 19 Oct 2022 09:22:55 -0000
Reply-to: Bug 1993502 <1993502@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Looks like a bunch of OVN unit tests are highly depending on the test
order.

When rebuilding the Debian Zed package of Neutron under Debian Unstable,
I get 200+ unit test failures like what's below. Using tox, if running:

tox -e py3 --
neutron.tests.unit.plugins.ml2.drivers.ovn.mech_driver.test_mech_driver

this works, however, running a single unit test like this:

tox -e py3 --
neutron.tests.unit.plugins.ml2.drivers.ovn.mech_driver.test_mech_driver.TestOVNMechanismDriverSecurityGroup

simply fails. Under Debian, I had to add to the blacklist of unit tests
all of:

- plugins.ml2.drivers.ovn.mech_driver.TestOVNMechanismDriverSecurityGroup.*
- services.ovn_l3.test_plugin.OVNL3ExtrarouteTests.*

Please help me fix this.

Below are example of the 2 types of failure.

======================================================================
FAIL: neutron.tests.unit.plugins.ml2.drivers.ovn.mech_driver.test_mech_driver.TestOVNMechanismDriverSecurityGroup.test_update_sg_duplicate_rule_multi_ports
neutron.tests.unit.plugins.ml2.drivers.ovn.mech_driver.test_mech_driver.TestOVNMechanismDriverSecurityGroup.test_update_sg_duplicate_rule_multi_ports
----------------------------------------------------------------------
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/<<PKGBUILDDIR>>/neutron/tests/unit/plugins/ml2/drivers/ovn/mech_driver/test_mech_driver.py", line 3733, in setUp
    cfg.CONF.set_override('dns_servers', ['8.8.8.8'], group='ovn')
  File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2077, in __inner
    result = f(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2460, in set_override
    opt_info = self._get_opt_info(name, group)
  File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2869, in _get_opt_info
    group = self._get_group(group)
  File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2838, in _get_group
    raise NoSuchGroupError(group_name)
oslo_config.cfg.NoSuchGroupError: no such group [ovn]


======================================================================
FAIL: neutron.tests.unit.services.ovn_l3.test_plugin.OVNL3ExtrarouteTests.test__notify_gateway_port_ip_changed
neutron.tests.unit.services.ovn_l3.test_plugin.OVNL3ExtrarouteTests.test__notify_gateway_port_ip_changed
----------------------------------------------------------------------
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2219, in __getattr__
    return self._get(name)
  File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2653, in _get
    value, loc = self._do_get(name, group, namespace)
  File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2671, in _do_get
    info = self._get_opt_info(name, group)
  File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2876, in _get_opt_info
    raise NoSuchOptError(opt_name, group)
oslo_config.cfg.NoSuchOptError: no such option ovn in group [DEFAULT]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/<<PKGBUILDDIR>>/neutron/tests/unit/services/ovn_l3/test_plugin.py", line 1738, in setUp
    super(test_l3.L3BaseForIntTests, self).setUp(
  File "/<<PKGBUILDDIR>>/neutron/tests/unit/db/test_db_base_plugin_v2.py", line 163, in setUp
    self.api = router.APIRouter()
  File "/<<PKGBUILDDIR>>/neutron/api/v2/router.py", line 21, in APIRouter
    return pecan_app.v2_factory(None, **local_config)
  File "/<<PKGBUILDDIR>>/neutron/pecan_wsgi/app.py", line 47, in v2_factory
    startup.initialize_all()
  File "/<<PKGBUILDDIR>>/neutron/pecan_wsgi/startup.py", line 39, in initialize_all
    manager.init()
  File "/<<PKGBUILDDIR>>/neutron/manager.py", line 301, in init
    NeutronManager.get_instance()
  File "/<<PKGBUILDDIR>>/neutron/manager.py", line 252, in get_instance
    cls._create_instance()
  File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 414, in inner
    return f(*args, **kwargs)
  File "/<<PKGBUILDDIR>>/neutron/manager.py", line 238, in _create_instance
    cls._instance = cls()
  File "/<<PKGBUILDDIR>>/neutron/manager.py", line 132, in __init__
    self._load_service_plugins()
  File "/<<PKGBUILDDIR>>/neutron/manager.py", line 211, in _load_service_plugins
    self._create_and_add_service_plugin(provider)
  File "/<<PKGBUILDDIR>>/neutron/manager.py", line 214, in _create_and_add_service_plugin
    plugin_inst = self._get_plugin_instance('neutron.service_plugins',
  File "/<<PKGBUILDDIR>>/neutron/manager.py", line 162, in _get_plugin_instance
    plugin_inst = plugin_class()
  File "/<<PKGBUILDDIR>>/neutron/quota/resource_registry.py", line 124, in wrapper
    return f(*args, **kwargs)
  File "/<<PKGBUILDDIR>>/neutron/services/ovn_l3/plugin.py", line 92, in __init__
    self.scheduler = l3_ovn_scheduler.get_scheduler()
  File "/<<PKGBUILDDIR>>/neutron/scheduler/l3_ovn_scheduler.py", line 153, in get_scheduler
    return OVN_SCHEDULER_STR_TO_CLASS[ovn_conf.get_ovn_l3_scheduler()]()
  File "/<<PKGBUILDDIR>>/neutron/conf/plugins/ml2/drivers/ovn/ovn_conf.py", line 284, in get_ovn_l3_scheduler
    return cfg.CONF.ovn.ovn_l3_scheduler
  File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2223, in __getattr__
    raise NoSuchOptError(name)
oslo_config.cfg.NoSuchOptError: no such option ovn in group [DEFAULT]

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1993502

Title:
  failing unit tests when not running them all

Status in neutron:
  New

Bug description:
  Looks like a bunch of OVN unit tests are highly depending on the test
  order.

  When rebuilding the Debian Zed package of Neutron under Debian
  Unstable, I get 200+ unit test failures like what's below. Using tox,
  if running:

  tox -e py3 --
  neutron.tests.unit.plugins.ml2.drivers.ovn.mech_driver.test_mech_driver

  this works, however, running a single unit test like this:

  tox -e py3 --
  neutron.tests.unit.plugins.ml2.drivers.ovn.mech_driver.test_mech_driver.TestOVNMechanismDriverSecurityGroup

  simply fails. Under Debian, I had to add to the blacklist of unit
  tests all of:

  - plugins.ml2.drivers.ovn.mech_driver.TestOVNMechanismDriverSecurityGroup.*
  - services.ovn_l3.test_plugin.OVNL3ExtrarouteTests.*

  Please help me fix this.

  Below are example of the 2 types of failure.

  ======================================================================
  FAIL: neutron.tests.unit.plugins.ml2.drivers.ovn.mech_driver.test_mech_driver.TestOVNMechanismDriverSecurityGroup.test_update_sg_duplicate_rule_multi_ports
  neutron.tests.unit.plugins.ml2.drivers.ovn.mech_driver.test_mech_driver.TestOVNMechanismDriverSecurityGroup.test_update_sg_duplicate_rule_multi_ports
  ----------------------------------------------------------------------
  testtools.testresult.real._StringException: Traceback (most recent call last):
    File "/<<PKGBUILDDIR>>/neutron/tests/unit/plugins/ml2/drivers/ovn/mech_driver/test_mech_driver.py", line 3733, in setUp
      cfg.CONF.set_override('dns_servers', ['8.8.8.8'], group='ovn')
    File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2077, in __inner
      result = f(self, *args, **kwargs)
    File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2460, in set_override
      opt_info = self._get_opt_info(name, group)
    File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2869, in _get_opt_info
      group = self._get_group(group)
    File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2838, in _get_group
      raise NoSuchGroupError(group_name)
  oslo_config.cfg.NoSuchGroupError: no such group [ovn]

  
  ======================================================================
  FAIL: neutron.tests.unit.services.ovn_l3.test_plugin.OVNL3ExtrarouteTests.test__notify_gateway_port_ip_changed
  neutron.tests.unit.services.ovn_l3.test_plugin.OVNL3ExtrarouteTests.test__notify_gateway_port_ip_changed
  ----------------------------------------------------------------------
  testtools.testresult.real._StringException: Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2219, in __getattr__
      return self._get(name)
    File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2653, in _get
      value, loc = self._do_get(name, group, namespace)
    File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2671, in _do_get
      info = self._get_opt_info(name, group)
    File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2876, in _get_opt_info
      raise NoSuchOptError(opt_name, group)
  oslo_config.cfg.NoSuchOptError: no such option ovn in group [DEFAULT]

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/<<PKGBUILDDIR>>/neutron/tests/unit/services/ovn_l3/test_plugin.py", line 1738, in setUp
      super(test_l3.L3BaseForIntTests, self).setUp(
    File "/<<PKGBUILDDIR>>/neutron/tests/unit/db/test_db_base_plugin_v2.py", line 163, in setUp
      self.api = router.APIRouter()
    File "/<<PKGBUILDDIR>>/neutron/api/v2/router.py", line 21, in APIRouter
      return pecan_app.v2_factory(None, **local_config)
    File "/<<PKGBUILDDIR>>/neutron/pecan_wsgi/app.py", line 47, in v2_factory
      startup.initialize_all()
    File "/<<PKGBUILDDIR>>/neutron/pecan_wsgi/startup.py", line 39, in initialize_all
      manager.init()
    File "/<<PKGBUILDDIR>>/neutron/manager.py", line 301, in init
      NeutronManager.get_instance()
    File "/<<PKGBUILDDIR>>/neutron/manager.py", line 252, in get_instance
      cls._create_instance()
    File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 414, in inner
      return f(*args, **kwargs)
    File "/<<PKGBUILDDIR>>/neutron/manager.py", line 238, in _create_instance
      cls._instance = cls()
    File "/<<PKGBUILDDIR>>/neutron/manager.py", line 132, in __init__
      self._load_service_plugins()
    File "/<<PKGBUILDDIR>>/neutron/manager.py", line 211, in _load_service_plugins
      self._create_and_add_service_plugin(provider)
    File "/<<PKGBUILDDIR>>/neutron/manager.py", line 214, in _create_and_add_service_plugin
      plugin_inst = self._get_plugin_instance('neutron.service_plugins',
    File "/<<PKGBUILDDIR>>/neutron/manager.py", line 162, in _get_plugin_instance
      plugin_inst = plugin_class()
    File "/<<PKGBUILDDIR>>/neutron/quota/resource_registry.py", line 124, in wrapper
      return f(*args, **kwargs)
    File "/<<PKGBUILDDIR>>/neutron/services/ovn_l3/plugin.py", line 92, in __init__
      self.scheduler = l3_ovn_scheduler.get_scheduler()
    File "/<<PKGBUILDDIR>>/neutron/scheduler/l3_ovn_scheduler.py", line 153, in get_scheduler
      return OVN_SCHEDULER_STR_TO_CLASS[ovn_conf.get_ovn_l3_scheduler()]()
    File "/<<PKGBUILDDIR>>/neutron/conf/plugins/ml2/drivers/ovn/ovn_conf.py", line 284, in get_ovn_l3_scheduler
      return cfg.CONF.ovn.ovn_l3_scheduler
    File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2223, in __getattr__
      raise NoSuchOptError(name)
  oslo_config.cfg.NoSuchOptError: no such option ovn in group [DEFAULT]

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1993502/+subscriptions
Follow ups

[Bug 1993502] Re: failing unit tests when not running them all
From: OpenStack Infra, 2022-10-19