yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1939726] Re: neutron-ovn-db-sync error during generate OVN DB

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1939726@xxxxxxxxxxxxxxxxxx>
Date: Mon, 04 Oct 2021 08:52:57 -0000
Reply-to: Bug 1939726 <1939726@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Reviewed:  https://review.opendev.org/c/openstack/neutron/+/811996
Committed: https://opendev.org/openstack/neutron/commit/5e32dddc11c6ec73afc83d2bfc7c4beaa252de0c
Submitter: "Zuul (22348)"
Branch:    master

commit 5e32dddc11c6ec73afc83d2bfc7c4beaa252de0c
Author: Rodolfo Alonso Hernandez <ralonsoh@xxxxxxxxxx>
Date:   Thu Sep 30 16:28:19 2021 +0000

    Fix "_sync_metadata_ports" with no DHCP subnets
    
    When a subnet does not have DHCP configured, the metadata port does
    not have an IP address on this CIDR. The method
    "OvnNbSynchronizer.sync_networks_ports_and_dhcp_opts", was always
    setting an IP address for the metadata ports, regardless of the subnet
    configuration (with or without DHCP).
    
    The method "_sync_metadata_ports", in charge of synchronizing the
    metadata ports, now filters the subnets by the parameter "enable_dhcp".
    
    In case of having a subnet with DHCP enabled, if the metadata port is
    missing the subnet IP addresses, the method adds them.
    
    In case of having a subnet without DHCP enabled, if the metadata port
    has an IP address on the subnet, the method removes it.
    
    Closes-Bug: #1939726
    Change-Id: I09cc14dff6933aae63cbd43a29f9221f405ecede


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1939726

Title:
  neutron-ovn-db-sync error during generate OVN DB

Status in neutron:
  Fix Released

Bug description:
  I have deployed Openstack Victoria from kolla-ansible tool and
  Openvswich OVN for networking. neutron-ovn-db-sync tool generate OVN
  DB error when any subnet (disable dhcp) has allocated all available IP
  address. Because neutron-ovn-db-sync tool trying recreate port
  "network:distributed".

  --------------------
  2021-07-23 14:19:47.295 1417 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [req-8fb77747-70c7-459c-bab8-97dbe07a8eba - - - - -] Starting OVNClientQosExtension
  2021-07-23 14:19:47.296 1417 INFO neutron.cmd.ovn.neutron_ovn_db_sync_util [req-8fb77747-70c7-459c-bab8-97dbe07a8eba - - - - -] Sync for Northbound db started with mode : repair
  2021-07-23 14:19:55.231 1417 CRITICAL neutron_ovn_db_sync_util [req-8fb77747-70c7-459c-bab8-97dbe07a8eba - - - - -] Unhandled error: neutron_lib.exceptions.IpAddressGenerationFailure: No more IP addresses available on network 15b718eb-4589-4b9f-8b95-aa5dc71da538.
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util Traceback (most recent call last):
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/db/ipam_pluggable_backend.py", line 138, in _ipam_allocate_ips
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     ip_address, subnet_id = ipam_allocator.allocate(ip_request)
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/ipam/subnet_alloc.py", line 240, in allocate
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     raise ipam_exc.IpAddressGenerationFailureAllSubnets()
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util neutron.ipam.exceptions.IpAddressGenerationFailureAllSubnets: No more IP addresses available.
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util 
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util During handling of the above exception, another exception occurred:
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util 
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util Traceback (most recent call last):
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/bin/neutron-ovn-db-sync-util", line 8, in <module>
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     sys.exit(main())
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/cmd/ovn/neutron_ovn_db_sync_util.py", line 229, in main
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     synchronizer.do_sync()
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 101, in do_sync
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     self.sync_networks_ports_and_dhcp_opts(ctx)
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 987, in sync_networks_ports_and_dhcp_opts
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     self._sync_metadata_ports(ctx, db_ports)
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 912, in _sync_metadata_ports
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     self._ovn_client.update_metadata_port(ctx, net['id'])
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2155, in update_metadata_port
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     update_metadata_port_fixed_ips(metadata_port,
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2122, in update_metadata_port_fixed_ips
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     self._plugin.update_port(n_context.get_admin_context(),
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/common/utils.py", line 687, in inner
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     return f(self, context, *args, **kwargs)
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron_lib/db/api.py", line 233, in wrapped
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     return method(*args, **kwargs)
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron_lib/db/api.py", line 139, in wrapped
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     setattr(e, '_RETRY_EXCEEDED', True)
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     self.force_reraise()
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     six.reraise(self.type_, self.value, self.tb)
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     raise value
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron_lib/db/api.py", line 135, in wrapped
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     return f(*args, **kwargs)
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_db/api.py", line 154, in wrapper
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     ectxt.value = e.inner_exc
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     self.force_reraise()
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     six.reraise(self.type_, self.value, self.tb)
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     raise value
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_db/api.py", line 142, in wrapper
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     return f(*args, **kwargs)
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     LOG.debug("Retry wrapper got retriable exception: %s", e)
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     self.force_reraise()
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     six.reraise(self.type_, self.value, self.tb)
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     raise value
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron_lib/db/api.py", line 179, in wrapped
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     return f(*dup_args, **dup_kwargs)
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/plugins/ml2/plugin.py", line 1653, in update_port
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     updated_port = super(Ml2Plugin, self).update_port(context, id,
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron_lib/db/api.py", line 233, in wrapped
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     return method(*args, **kwargs)
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/db/db_base_plugin_v2.py", line 1482, in update_port
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     self.ipam.update_port(
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/db/ipam_backend_mixin.py", line 711, in update_port
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     changes = self.update_port_with_ips(context,
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/db/ipam_pluggable_backend.py", line 427, in update_port_with_ips
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     changes = self._update_ips_for_port(context,
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/db/ipam_pluggable_backend.py", line 381, in _update_ips_for_port
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     added = self._ipam_allocate_ips(context, ipam_driver,
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/db/ipam_pluggable_backend.py", line 148, in _ipam_allocate_ips
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     return
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     self.force_reraise()
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     six.reraise(self.type_, self.value, self.tb)
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     raise value
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/db/ipam_pluggable_backend.py", line 140, in _ipam_allocate_ips
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util     raise n_exc.IpAddressGenerationFailure(
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util neutron_lib.exceptions.IpAddressGenerationFailure: No more IP addresses available on network 15b718eb-4589-4b9f-8b95-aa5dc71da538.
  2021-07-23 14:19:55.231 1417 ERROR neutron_ovn_db_sync_util 

  --------------------

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1939726/+subscriptions
References

[Bug 1939726] [NEW] neutron-ovn-db-sync error during generate OVN DB
From: Son Do Xuan, 2021-08-12