yahoo-eng-team team mailing list archive

[OpenStack Infra <1952004@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/neutron/+/819032
Committed: https://opendev.org/openstack/neutron/commit/af03d133f44cc56b557b8ff7b57999acda7b833f
Submitter: "Zuul (22348)"
Branch:    master

commit af03d133f44cc56b557b8ff7b57999acda7b833f
Author: Jakub Libosvar <libosvar@xxxxxxxxxx>
Date:   Tue Nov 23 22:27:16 2021 +0000

    ovn: Wait for northd in functional tests
    
    Each functional test method create new empty NB and SB DBs. Each DB
    has its own table NB_Global or SB_Global respectively that contains
    exactly one record created by ovn-northd. When functional test spawns
    northd, it populates the DBs however it doesn't guarantee the running
    test in parallel will not attempt to access the record.
    
    This patch makes sure that northd creates the records before it moves on
    and considers northd spawned.
    
    Closes-Bug: #1952004
    
    Signed-off-by: Jakub Libosvar <libosvar@xxxxxxxxxx>
    Change-Id: Ic936864aebcabb811860e17913fbff311c52845f


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1952004

Title:
  ovn: Functional test may fail if northd hasn't initialized DBs yet

Status in neutron:
  Fix Released

Bug description:
  Example:
  https://7bed286b1abc124aef60-716c2febf7f730d66787c22f3ed0da3e.ssl.cf5.rackcdn.com/818067/2/check/neutron-functional-with-uwsgi/48adb0f/testr_results.html

  Traceback:
  2021-11-19 22:15:09.291 52011 INFO neutron.tests.unit.plugins.ml2.drivers.mechanism_logger [req-819d37a7-3659-4d27-8a49-834bc474728f - tenid - - -] create_network_precommit called with network settings {'id': 'da5ba714-bfd7-45f8-99bd-55ff4bee2f89', 'name': 'n1', 'tenant_id': '46f70361-ba71-4bd0-9769-3573fd227c4b', 'admin_state_up': True, 'mtu': 1500, 'status': 'ACTIVE', 'subnets': [], 'standard_attr_id': 6, 'shared': False, 'project_id': '46f70361-ba71-4bd0-9769-3573fd227c4b', 'port_security_enabled': True, 'router:external': False, 'provider:network_type': 'vlan', 'provider:physical_network': 'physnet1', 'provider:segmentation_id': 100, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'vlan_transparent': None, 'description': '', 'revision_number': 1, 'l2_adjacency': True} (original settings None) and network segments [{'id': 'd146263d-2926-40a8-b21a-a6b9a930fbff', 'network_type': 'vlan', 'physical_network': 'physnet1', 'segmentation_id': 100, 'network_id': 'da5ba714-bfd7-45f8-99bd-55ff4bee2f89'}]
  2021-11-19 22:15:09.292 52011 ERROR neutron.plugins.ml2.managers [req-819d37a7-3659-4d27-8a49-834bc474728f - tenid - - -] Mechanism driver 'ovn' failed in create_network_precommit: StopIteration
  2021-11-19 22:15:09.292 52011 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
  2021-11-19 22:15:09.292 52011 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 482, in _call_on_drivers
  2021-11-19 22:15:09.292 52011 ERROR neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
  2021-11-19 22:15:09.292 52011 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 536, in create_network_precommit
  2021-11-19 22:15:09.292 52011 ERROR neutron.plugins.ml2.managers     self._validate_network_segments(context.network_segments)
  2021-11-19 22:15:09.292 52011 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 488, in _validate_network_segments
  2021-11-19 22:15:09.292 52011 ERROR neutron.plugins.ml2.managers     max_tunid = self._get_max_tunid()
  2021-11-19 22:15:09.292 52011 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 482, in _get_max_tunid
  2021-11-19 22:15:09.292 52011 ERROR neutron.plugins.ml2.managers     return int(self.nb_ovn.nb_global.options.get('max_tunid'))
  2021-11-19 22:15:09.292 52011 ERROR neutron.plugins.ml2.managers   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 256, in nb_global
  2021-11-19 22:15:09.292 52011 ERROR neutron.plugins.ml2.managers     return next(iter(self.db_list_rows('NB_Global').execute(
  2021-11-19 22:15:09.292 52011 ERROR neutron.plugins.ml2.managers StopIteration
  2021-11-19 22:15:09.292 52011 ERROR neutron.plugins.ml2.managers 
  2021-11-19 22:15:09.296 52011 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] Hash Ring: Node e6333d61-3ac1-4c6f-9b90-a23e848ed629 (host: ubuntu-focal-inmotion-iad3-0027436269) handling event "create" for row c5936e2f-4eb2-461c-b095-a27908868816 (table: NB_Global) notify /home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:671

  This can happen in cases when northd is slow or local IDL hasn't got
  updates yet when northd populates NB_Global table and at the same time
  we try to access any of the content in NB_Global.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1952004/+subscriptions