yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2103697] [NEW] Existing routers become unmanageable after migration from OVS to OVN

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Pavlo Shchelokovskyy <2103697@xxxxxxxxxxxxxxxxxx>
Date: Thu, 20 Mar 2025 10:01:08 -0000
Reply-to: Bug 2103697 <2103697@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

We are investigating the possibilities of migrating from OVS to OVN.
Current state of the migration flow itself we have can be seen here
https://github.com/Mirantis/rockoon/blob/aef29880203481fb6383da09acb5046747f51137/rockoon/cli/ovs_ovn_migration.py

What we found is that after the migration is complete, the pre-existing
routers (created on OVS before migration) become unmanageable. For
example, one can't delete them via neutron API, the response is HTTP500
with the following error/trace in logs:

31276:2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource [None req-9e728c4e-349a-4c15-b15b-4cbd0fe84da1 cffbb8f11d5f47d49f5ef63114ca04b6 71b11698a0aa4adb9974f825983b8e1a - - default default] delete failed: No details.: neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.services.ovn_l3.service_providers.driver_controller.DriverController._clear_router_provider-649091 failed with "'ha'"
31277-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource Traceback (most recent call last):
31278-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron/api/v2/resource.py", line 98, in resource
31279-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource     result = method(request=request, **args)
31280-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron/api/v2/base.py", line 570, in delete
31281-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource     return self._delete(request, id, **kwargs)
31282-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped
31283-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource     with excutils.save_and_reraise_exception():
31284-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
31285-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource     self.force_reraise()
31286-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
--
31342-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
31343-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource     self.force_reraise()
31344-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
31345-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource     raise self.value
31346-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/db/utils.py", line 104, in _wrapped
31347-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource     return function(*args, **kwargs)
31348-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/callbacks/manager.py", line 168, in publish
31349-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource     raise exceptions.CallbackFailure(errors=errors)
31350-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.services.ovn_l3.service_providers.driver_controller.DriverController._clear_router_provider-649091 failed with "'ha'"
31351-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource 
31352:2025-02-25 11:39:26.913 28 INFO neutron.wsgi [None req-9e728c4e-349a-4c15-b15b-4cbd0fe84da1 cffbb8f11d5f47d49f5ef63114ca04b6 71b11698a0aa4adb9974f825983b8e1a - - default default] 192.168.238.166 "DELETE /v2.0/routers/bba5be48-2721-4ab2-840a-1cff81148ef2 HTTP/1.1" status: 500  len: 368 time: 5.0718703

Also, after migration one of ovn maintenance tasks (this is Caracal
specific, this code is already removed from master) keeps failing with

025-02-28T10:54:44.153763722Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics [None req-5a85f113-4549-4dca-b768-3e3cc4769f28 - - - - - -] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.add_provider_resource_association_to_routers' (it runs every 5.00 seconds): neutron_lib.objects.exceptions.NeutronDbObjectDuplicateEntry: Failed to create a duplicate ProviderResourceAssociation: for attribute(s) ['resource_id'] with value(s) 4d9eb917-f05a-4590-a569-42cb59c52bab
2025-02-28T10:54:44.153824676Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics Traceback (most recent call last):
2025-02-28T10:54:44.153830586Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
2025-02-28T10:54:44.153834724Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     self.dialect.do_execute(
2025-02-28T10:54:44.153838399Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
2025-02-28T10:54:44.153842104Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     cursor.execute(statement, parameters)
2025-02-28T10:54:44.153847123Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/pymysql/cursors.py", line 153, in execute
2025-02-28T10:54:44.153850609Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     result = self._query(query)
2025-02-28T10:54:44.153858566Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/pymysql/cursors.py", line 322, in _query
2025-02-28T10:54:44.153862137Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     conn.query(q)
2025-02-28T10:54:44.153865765Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/pymysql/connections.py", line 558, in query
2025-02-28T10:54:44.153869991Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2025-02-28T10:54:44.153873710Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/pymysql/connections.py", line 822, in _read_query_result
2025-02-28T10:54:44.153877267Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     result.read()
2025-02-28T10:54:44.153880644Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/pymysql/connections.py", line 1200, in read
2025-02-28T10:54:44.153884719Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     first_packet = self.connection._read_packet()
2025-02-28T10:54:44.153888281Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/pymysql/connections.py", line 772, in _read_packet
2025-02-28T10:54:44.153891791Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     packet.raise_for_error()
2025-02-28T10:54:44.153895244Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/pymysql/protocol.py", line 221, in raise_for_error
2025-02-28T10:54:44.153898678Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     err.raise_mysql_exception(self._data)
2025-02-28T10:54:44.153902218Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
2025-02-28T10:54:44.153905719Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     raise errorclass(errno, errval)
2025-02-28T10:54:44.153909394Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics pymysql.err.IntegrityError: (1062, "Duplicate entry '4d9eb917-f05a-4590-a569-42cb59c52bab' for key 'resource_id'")

As I understand it, the culprit is in the 'provideresourceassociation' table.
OVS supports 4 different providers ('single_node', 'ha', dvr', 'dvrha'), and OVN supports only "ovn" one.
And nothing during the migration (e.g. in neutron/plugins/ml2/drivers/ovn/db_migration.py) actually changes this table. Which then leads to both of the errors above.

** Affects: neutron
     Importance: Undecided
     Assignee: Pavlo Shchelokovskyy (pshchelo)
         Status: In Progress

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2103697

Title:
  Existing routers become unmanageable after migration from OVS to OVN

Status in neutron:
  In Progress

Bug description:
  We are investigating the possibilities of migrating from OVS to OVN.
  Current state of the migration flow itself we have can be seen here
  https://github.com/Mirantis/rockoon/blob/aef29880203481fb6383da09acb5046747f51137/rockoon/cli/ovs_ovn_migration.py

  What we found is that after the migration is complete, the pre-
  existing routers (created on OVS before migration) become
  unmanageable. For example, one can't delete them via neutron API, the
  response is HTTP500 with the following error/trace in logs:

  31276:2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource [None req-9e728c4e-349a-4c15-b15b-4cbd0fe84da1 cffbb8f11d5f47d49f5ef63114ca04b6 71b11698a0aa4adb9974f825983b8e1a - - default default] delete failed: No details.: neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.services.ovn_l3.service_providers.driver_controller.DriverController._clear_router_provider-649091 failed with "'ha'"
  31277-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource Traceback (most recent call last):
  31278-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron/api/v2/resource.py", line 98, in resource
  31279-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource     result = method(request=request, **args)
  31280-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron/api/v2/base.py", line 570, in delete
  31281-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource     return self._delete(request, id, **kwargs)
  31282-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped
  31283-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource     with excutils.save_and_reraise_exception():
  31284-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  31285-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource     self.force_reraise()
  31286-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  --
  31342-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  31343-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource     self.force_reraise()
  31344-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  31345-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource     raise self.value
  31346-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/db/utils.py", line 104, in _wrapped
  31347-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource     return function(*args, **kwargs)
  31348-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/callbacks/manager.py", line 168, in publish
  31349-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource     raise exceptions.CallbackFailure(errors=errors)
  31350-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.services.ovn_l3.service_providers.driver_controller.DriverController._clear_router_provider-649091 failed with "'ha'"
  31351-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource 
  31352:2025-02-25 11:39:26.913 28 INFO neutron.wsgi [None req-9e728c4e-349a-4c15-b15b-4cbd0fe84da1 cffbb8f11d5f47d49f5ef63114ca04b6 71b11698a0aa4adb9974f825983b8e1a - - default default] 192.168.238.166 "DELETE /v2.0/routers/bba5be48-2721-4ab2-840a-1cff81148ef2 HTTP/1.1" status: 500  len: 368 time: 5.0718703

  Also, after migration one of ovn maintenance tasks (this is Caracal
  specific, this code is already removed from master) keeps failing with

  025-02-28T10:54:44.153763722Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics [None req-5a85f113-4549-4dca-b768-3e3cc4769f28 - - - - - -] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.add_provider_resource_association_to_routers' (it runs every 5.00 seconds): neutron_lib.objects.exceptions.NeutronDbObjectDuplicateEntry: Failed to create a duplicate ProviderResourceAssociation: for attribute(s) ['resource_id'] with value(s) 4d9eb917-f05a-4590-a569-42cb59c52bab
  2025-02-28T10:54:44.153824676Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics Traceback (most recent call last):
  2025-02-28T10:54:44.153830586Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
  2025-02-28T10:54:44.153834724Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     self.dialect.do_execute(
  2025-02-28T10:54:44.153838399Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
  2025-02-28T10:54:44.153842104Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     cursor.execute(statement, parameters)
  2025-02-28T10:54:44.153847123Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/pymysql/cursors.py", line 153, in execute
  2025-02-28T10:54:44.153850609Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     result = self._query(query)
  2025-02-28T10:54:44.153858566Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/pymysql/cursors.py", line 322, in _query
  2025-02-28T10:54:44.153862137Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     conn.query(q)
  2025-02-28T10:54:44.153865765Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/pymysql/connections.py", line 558, in query
  2025-02-28T10:54:44.153869991Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  2025-02-28T10:54:44.153873710Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/pymysql/connections.py", line 822, in _read_query_result
  2025-02-28T10:54:44.153877267Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     result.read()
  2025-02-28T10:54:44.153880644Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/pymysql/connections.py", line 1200, in read
  2025-02-28T10:54:44.153884719Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     first_packet = self.connection._read_packet()
  2025-02-28T10:54:44.153888281Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/pymysql/connections.py", line 772, in _read_packet
  2025-02-28T10:54:44.153891791Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     packet.raise_for_error()
  2025-02-28T10:54:44.153895244Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/pymysql/protocol.py", line 221, in raise_for_error
  2025-02-28T10:54:44.153898678Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     err.raise_mysql_exception(self._data)
  2025-02-28T10:54:44.153902218Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics   File "/var/lib/openstack/lib/python3.10/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
  2025-02-28T10:54:44.153905719Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics     raise errorclass(errno, errval)
  2025-02-28T10:54:44.153909394Z 2025-02-28 10:54:44.151 27 ERROR futurist.periodics pymysql.err.IntegrityError: (1062, "Duplicate entry '4d9eb917-f05a-4590-a569-42cb59c52bab' for key 'resource_id'")

  As I understand it, the culprit is in the 'provideresourceassociation' table.
  OVS supports 4 different providers ('single_node', 'ha', dvr', 'dvrha'), and OVN supports only "ovn" one.
  And nothing during the migration (e.g. in neutron/plugins/ml2/drivers/ovn/db_migration.py) actually changes this table. Which then leads to both of the errors above.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2103697/+subscriptions
Follow ups

[Bug 2103697] Re: Existing routers become unmanageable after migration from OVS to OVN
From: OpenStack Infra, 2025-03-25