yahoo-eng-team team mailing list archive

[OpenStack Infra <2045811@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/neutron/+/902841
Committed: https://opendev.org/openstack/neutron/commit/e4323e1f209ea1c63fe7af5275ea2b96f52b8740
Submitter: "Zuul (22348)"
Branch:    master

commit e4323e1f209ea1c63fe7af5275ea2b96f52b8740
Author: Brian Haley <haleyb.dev@xxxxxxxxx>
Date:   Wed Dec 6 16:37:24 2023 -0500

    Fix KeyError failure in _sync_subnet_dhcp_options()
    
    If the netron-ovn-db-sync-util is run while neutron-server
    is active (which is not recommended), it can randomly fail
    if there are active API calls in flight to create networks
    and/or subnets.
    
    Skip the subnet and log a warning if detected.
    
    Closes-bug: #2045811
    Change-Id: Ic5d9608277dd5c4881b3e4b494e1864be0bed1b4


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2045811

Title:
  neutron-ovn-db-sync-util can fail with KeyError

Status in neutron:
  Fix Released

Bug description:
  If the netron-ovn-db-sync-util is run while neutron-server is active
  (which is not recommended), it can randomly fail if there are active
  API calls in flight to create networks and/or subnets.

  This is an example traceback I've seen many times in a production
  environment:

  WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-7fc12422-6fae-4ec9-98bc-8a114f30c9e3 - - - - -] DHCP options for subnet 0662e4fd-f8b4-4d29-8ba7-5846bd19e45d is present in Neutron but out of sync for OVN
  CRITICAL neutron_ovn_db_sync_util [req-7fc12422-6fae-4ec9-98bc-8a114f30c9e3 - - - - -] Unhandled error: KeyError: 'neutron-93ad1c21-d2cf-448a-8fae-21c71f44dc5c'
  ERROR neutron_ovn_db_sync_util Traceback (most recent call last):
  ERROR neutron_ovn_db_sync_util   File "/usr/bin/neutron-ovn-db-sync-util", line 10, in <module>
  ERROR neutron_ovn_db_sync_util     sys.exit(main())
  ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/cmd/ovn/neutron_ovn_db_sync_util.py", line 219, in main
  ERROR neutron_ovn_db_sync_util     synchronizer.do_sync()
  ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 98, in do_sync
  ERROR neutron_ovn_db_sync_util     self.sync_networks_ports_and_dhcp_opts(ctx)
  ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 871, in sync_networks_ports_and_dhcp_opts
  ERROR neutron_ovn_db_sync_util     self._sync_subnet_dhcp_options(
  ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 645, in _sync_subnet_dhcp_options
  ERROR neutron_ovn_db_sync_util     network = db_networks[utils.ovn_name(subnet['network_id'])]
  ERROR neutron_ovn_db_sync_util KeyError: 'neutron-93ad1c21-d2cf-448a-8fae-21c71f44dc5c'

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2045811/+subscriptions