yahoo-eng-team team mailing list archive

[Brian Haley <2045811@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

If the netron-ovn-db-sync-util is run while neutron-server is active
(which is not recommended), it can randomly fail if there are active API
calls in flight to create networks and/or subnets.

This is an example traceback I've seen many times in a production
environment:

WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-7fc12422-6fae-4ec9-98bc-8a114f30c9e3 - - - - -] DHCP options for subnet 0662e4fd-f8b4-4d29-8ba7-5846bd19e45d is present in Neutron but out of sync for OVN
CRITICAL neutron_ovn_db_sync_util [req-7fc12422-6fae-4ec9-98bc-8a114f30c9e3 - - - - -] Unhandled error: KeyError: 'neutron-93ad1c21-d2cf-448a-8fae-21c71f44dc5c'
ERROR neutron_ovn_db_sync_util Traceback (most recent call last):
ERROR neutron_ovn_db_sync_util   File "/usr/bin/neutron-ovn-db-sync-util", line 10, in <module>
ERROR neutron_ovn_db_sync_util     sys.exit(main())
ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/cmd/ovn/neutron_ovn_db_sync_util.py", line 219, in main
ERROR neutron_ovn_db_sync_util     synchronizer.do_sync()
ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 98, in do_sync
ERROR neutron_ovn_db_sync_util     self.sync_networks_ports_and_dhcp_opts(ctx)
ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 871, in sync_networks_ports_and_dhcp_opts
ERROR neutron_ovn_db_sync_util     self._sync_subnet_dhcp_options(
ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 645, in _sync_subnet_dhcp_options
ERROR neutron_ovn_db_sync_util     network = db_networks[utils.ovn_name(subnet['network_id'])]
ERROR neutron_ovn_db_sync_util KeyError: 'neutron-93ad1c21-d2cf-448a-8fae-21c71f44dc5c'

** Affects: neutron
     Importance: Medium
     Assignee: Brian Haley (brian-haley)
         Status: Confirmed


** Tags: ovn

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2045811

Title:
  neutron-ovn-db-sync-util can fail with KeyError

Status in neutron:
  Confirmed

Bug description:
  If the netron-ovn-db-sync-util is run while neutron-server is active
  (which is not recommended), it can randomly fail if there are active
  API calls in flight to create networks and/or subnets.

  This is an example traceback I've seen many times in a production
  environment:

  WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-7fc12422-6fae-4ec9-98bc-8a114f30c9e3 - - - - -] DHCP options for subnet 0662e4fd-f8b4-4d29-8ba7-5846bd19e45d is present in Neutron but out of sync for OVN
  CRITICAL neutron_ovn_db_sync_util [req-7fc12422-6fae-4ec9-98bc-8a114f30c9e3 - - - - -] Unhandled error: KeyError: 'neutron-93ad1c21-d2cf-448a-8fae-21c71f44dc5c'
  ERROR neutron_ovn_db_sync_util Traceback (most recent call last):
  ERROR neutron_ovn_db_sync_util   File "/usr/bin/neutron-ovn-db-sync-util", line 10, in <module>
  ERROR neutron_ovn_db_sync_util     sys.exit(main())
  ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/cmd/ovn/neutron_ovn_db_sync_util.py", line 219, in main
  ERROR neutron_ovn_db_sync_util     synchronizer.do_sync()
  ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 98, in do_sync
  ERROR neutron_ovn_db_sync_util     self.sync_networks_ports_and_dhcp_opts(ctx)
  ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 871, in sync_networks_ports_and_dhcp_opts
  ERROR neutron_ovn_db_sync_util     self._sync_subnet_dhcp_options(
  ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 645, in _sync_subnet_dhcp_options
  ERROR neutron_ovn_db_sync_util     network = db_networks[utils.ovn_name(subnet['network_id'])]
  ERROR neutron_ovn_db_sync_util KeyError: 'neutron-93ad1c21-d2cf-448a-8fae-21c71f44dc5c'

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2045811/+subscriptions