yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2103697] Re: Existing routers become unmanageable after migration from OVS to OVN

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <2103697@xxxxxxxxxxxxxxxxxx>
Date: Tue, 25 Mar 2025 16:05:49 -0000
Reply-to: Bug 2103697 <2103697@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Reviewed:  https://review.opendev.org/c/openstack/neutron/+/945075
Committed: https://opendev.org/openstack/neutron/commit/90736b35179ceee6ef5457af6021f385f02b4300
Submitter: "Zuul (22348)"
Branch:    master

commit 90736b35179ceee6ef5457af6021f385f02b4300
Author: Pavlo Shchelokovskyy <shchelokovskyy@xxxxxxxxx>
Date:   Tue Mar 18 09:03:50 2025 +0000

    Update provider name when migrating to OVN
    
    otherwise, any router created before migration becomes unmanageable,
    as the only service provider ML2/OVN supports is 'ovn'.
    
    Closes-Bug: #2103697
    Change-Id: I104c188c2f86f4a2929ea00a8f28cf616c1bb64b


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2103697

Title:
  Existing routers become unmanageable after migration from OVS to OVN

Status in neutron:
  Fix Released

Bug description:
  We are investigating the possibilities of migrating from OVS to OVN.
  Current state of the migration flow itself we have can be seen here
  https://github.com/Mirantis/rockoon/blob/aef29880203481fb6383da09acb5046747f51137/rockoon/cli/ovs_ovn_migration.py

  What we found is that after the migration is complete, the pre-
  existing routers (created on OVS before migration) become
  unmanageable. For example, one can't delete them via neutron API, the
  response is HTTP500 with the following error/trace in logs:

  31276:2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource [None req-9e728c4e-349a-4c15-b15b-4cbd0fe84da1 cffbb8f11d5f47d49f5ef63114ca04b6 71b11698a0aa4adb9974f825983b8e1a - - default default] delete failed: No details.: neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.services.ovn_l3.service_providers.driver_controller.DriverController._clear_router_provider-649091 failed with "'ha'"
  31277-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource Traceback (most recent call last):
  31278-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron/api/v2/resource.py", line 98, in resource
  31279-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource     result = method(request=request, **args)
  31280-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron/api/v2/base.py", line 570, in delete
  31281-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource     return self._delete(request, id, **kwargs)
  31282-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped
  31283-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource     with excutils.save_and_reraise_exception():
  31284-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  31285-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource     self.force_reraise()
  31286-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  --
  31342-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  31343-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource     self.force_reraise()
  31344-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  31345-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource     raise self.value
  31346-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/db/utils.py", line 104, in _wrapped
  31347-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource     return function(*args, **kwargs)
  31348-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/callbacks/manager.py", line 168, in publish
  31349-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource     raise exceptions.CallbackFailure(errors=errors)
  31350-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.services.ovn_l3.service_providers.driver_controller.DriverController._clear_router_provider-649091 failed with "'ha'"
  31351-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource 
  31352:2025-02-25 11:39:26.913 28 INFO neutron.wsgi [None req-9e728c4e-349a-4c15-b15b-4cbd0fe84da1 cffbb8f11d5f47d49f5ef63114ca04b6 71b11698a0aa4adb9974f825983b8e1a - - default default] 192.168.238.166 "DELETE /v2.0/routers/bba5be48-2721-4ab2-840a-1cff81148ef2 HTTP/1.1" status: 500  len: 368 time: 5.0718703

  Also, after migration one of ovn maintenance tasks (this is Caracal
  specific, this code is already removed from master) keeps failing with

  025-02-28T10:54:44.153763722Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics [None req-5a85f113-4549-4dca-b768-3e3cc4769f28 - - - - - -] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.add_provider_resource_association_to_routers' (it runs every 5.00 seconds): neutron_lib.objects.exceptions.NeutronDbObjectDuplicateEntry: Failed to create a duplicate ProviderResourceAssociation: for attribute(s) ['resource_id'] with value(s) 4d9eb917-f05a-4590-a569-42cb59c52bab
  2025-02-28T10:54:44.153824676Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics Traceback (most recent call last):
  2025-02-28T10:54:44.153830586Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
  2025-02-28T10:54:44.153834724Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     self.dialect.do_execute(
  2025-02-28T10:54:44.153838399Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
  2025-02-28T10:54:44.153842104Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     cursor.execute(statement, parameters)
  2025-02-28T10:54:44.153847123Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/pymysql/cursors.py", line 153, in execute
  2025-02-28T10:54:44.153850609Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     result = self._query(query)
  2025-02-28T10:54:44.153858566Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/pymysql/cursors.py", line 322, in _query
  2025-02-28T10:54:44.153862137Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     conn.query(q)
  2025-02-28T10:54:44.153865765Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/pymysql/connections.py", line 558, in query
  2025-02-28T10:54:44.153869991Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  2025-02-28T10:54:44.153873710Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/pymysql/connections.py", line 822, in _read_query_result
  2025-02-28T10:54:44.153877267Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     result.read()
  2025-02-28T10:54:44.153880644Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/pymysql/connections.py", line 1200, in read
  2025-02-28T10:54:44.153884719Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     first_packet = self.connection._read_packet()
  2025-02-28T10:54:44.153888281Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/pymysql/connections.py", line 772, in _read_packet
  2025-02-28T10:54:44.153891791Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     packet.raise_for_error()
  2025-02-28T10:54:44.153895244Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/pymysql/protocol.py", line 221, in raise_for_error
  2025-02-28T10:54:44.153898678Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     err.raise_mysql_exception(self._data)
  2025-02-28T10:54:44.153902218Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
  2025-02-28T10:54:44.153905719Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     raise errorclass(errno, errval)
  2025-02-28T10:54:44.153909394Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics pymysql.err.IntegrityError: (1062, "Duplicate entry '4d9eb917-f05a-4590-a569-42cb59c52bab' for key 'resource_id'")

  As I understand it, the culprit is in the 'provideresourceassociation' table.
  OVS supports 4 different providers ('single_node', 'ha', dvr', 'dvrha'), and OVN supports only "ovn" one.
  And nothing during the migration (e.g. in neutron/plugins/ml2/drivers/ovn/db_migration.py) actually changes this table. Which then leads to both of the errors above.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2103697/+subscriptions
References

[Bug 2103697] [NEW] Existing routers become unmanageable after migration from OVS to OVN
From: Pavlo Shchelokovskyy, 2025-03-20