yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2060163] [NEW] [ovn] race condition with add/remove router interface

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Mohammed Naser <2060163@xxxxxxxxxxxxxxxxxx>
Date: Wed, 03 Apr 2024 22:01:37 -0000
Reply-to: Bug 2060163 <2060163@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

We're running into an issue in our CI with Atmosphere where we
frequently see failures when a router port is removed from an interface,
the traceback is the following:

==============================
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource [None req-e5d08cdd-28e6-4231-a50c-7eafc1b8f942 70fc3b55af8c4386b80207dad11db5da dcec54844db44eedbd9667951a5ceb6b - - - -] remove_router_interface failed: No details.: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router_Port with name=lrp-7e0debbb-893c-420a-8569-d8fb98e6a16e
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource Traceback (most recent call last):
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron/api/v2/resource.py", line 98, in resource
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     result = method(request=request, **args)
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/db/api.py", line 140, in wrapped
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     with excutils.save_and_reraise_exception():
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     self.force_reraise()
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     raise self.value
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/db/api.py", line 138, in wrapped
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     with excutils.save_and_reraise_exception() as ectxt:
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     self.force_reraise()
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     raise self.value
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/db/api.py", line 186, in wrapped
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     with excutils.save_and_reraise_exception():
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     self.force_reraise()
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     raise self.value
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/db/api.py", line 184, in wrapped
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron/api/v2/base.py", line 253, in _handle_action
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     ret_value = getattr(self._plugin, name)(*arg_list, **kwargs)
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron/services/ovn_l3/plugin.py", line 260, in remove_router_interface
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     with excutils.save_and_reraise_exception():
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     self.force_reraise()
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     raise self.value
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron/services/ovn_l3/plugin.py", line 257, in remove_router_interface
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     self._ovn_client.delete_router_port(
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1904, in delete_router_port
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     with self._nb_idl.transaction(check_error=True) as txn:
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     next(self.gen)
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     with super(OvsdbNbOvnIdl, self).transaction(*args, **kwargs) as t:
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     next(self.gen)
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/api.py", line 104, in transaction
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     with self.create_transaction(
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/api.py", line 61, in __exit__
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     self.result = self.commit()
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     raise result.ex
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     txn.results.put(txn.do_commit())
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 92, in do_commit
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     command.run_idl(txn)
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 341, in run_idl
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     entity = self.api.lookup(self.table, self.entity)
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 181, in lookup
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     return self._lookup(table, record)
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 228, in _lookup
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     row = idlutils.row_by_value(self, rl.table, rl.column, record)
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     raise RowNotFound(table=table, col=column, match=match)
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router_Port with name=lrp-7e0debbb-893c-420a-8569-d8fb98e6a16e
2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource [00m
2024-04-03 21:13:09.807 10 INFO neutron.wsgi [None req-e5d08cdd-28e6-4231-a50c-7eafc1b8f942 70fc3b55af8c4386b80207dad11db5da dcec54844db44eedbd9667951a5ceb6b - - - -] 10.0.0.73 "PUT /v2.0/routers/1a3225bc-725d-400f-b9ef-3ffe7d4fb8d8/remove_router_interface HTTP/1.1" status: 500  len: 363 time: 2.0234911[00m
==============================

I can't tell what's going on exactly, but it seems like for some reason
the `lrp` is missing when we try and remove it.  The add seems to happen
just fine though:

==============================
2024-04-03 21:12:58.396 9 INFO neutron.wsgi [None req-0b71c77f-3844-4549-af1e-541f344f302e 70fc3b55af8c4386b80207dad11db5da dcec54844db44eedbd9667951a5ceb6b - - - -] 10.0.0.73 "PUT /v2.0/routers/1a3225bc-725d-400f-b9ef-3ffe7d4fb8d8/add_router_interface HTTP/1.1" status: 200  len: 503 time: 3.1546130[00m
==============================

https://logs.atmosphere.dev/61/1061/46e91a5e83194215e35f7209ff92bf86597b4cd6/check/atmosphere-
molecule-aio-ovn/5f98342/ubuntu-jammy/pod-logs/openstack/neutron-
server-746d84b879-6q8kt/neutron-server.txt

Since this is a CI environment, in order to reduce the resources we
consume, we limit neutron to 2 workers.  Is it possible that it's not
keeping up with the events coming in from OVN and we end up in this
mismatch?

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2060163

Title:
  [ovn] race condition with add/remove router interface

Status in neutron:
  New

Bug description:
  We're running into an issue in our CI with Atmosphere where we
  frequently see failures when a router port is removed from an
  interface, the traceback is the following:

  ==============================
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource [None req-e5d08cdd-28e6-4231-a50c-7eafc1b8f942 70fc3b55af8c4386b80207dad11db5da dcec54844db44eedbd9667951a5ceb6b - - - -] remove_router_interface failed: No details.: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router_Port with name=lrp-7e0debbb-893c-420a-8569-d8fb98e6a16e
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource Traceback (most recent call last):
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron/api/v2/resource.py", line 98, in resource
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     result = method(request=request, **args)
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/db/api.py", line 140, in wrapped
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     with excutils.save_and_reraise_exception():
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     self.force_reraise()
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     raise self.value
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/db/api.py", line 138, in wrapped
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     with excutils.save_and_reraise_exception() as ectxt:
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     self.force_reraise()
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     raise self.value
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/db/api.py", line 186, in wrapped
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     with excutils.save_and_reraise_exception():
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     self.force_reraise()
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     raise self.value
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/db/api.py", line 184, in wrapped
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron/api/v2/base.py", line 253, in _handle_action
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     ret_value = getattr(self._plugin, name)(*arg_list, **kwargs)
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron/services/ovn_l3/plugin.py", line 260, in remove_router_interface
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     with excutils.save_and_reraise_exception():
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     self.force_reraise()
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     raise self.value
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron/services/ovn_l3/plugin.py", line 257, in remove_router_interface
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     self._ovn_client.delete_router_port(
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1904, in delete_router_port
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     with self._nb_idl.transaction(check_error=True) as txn:
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     next(self.gen)
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     with super(OvsdbNbOvnIdl, self).transaction(*args, **kwargs) as t:
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     next(self.gen)
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/api.py", line 104, in transaction
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     with self.create_transaction(
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/api.py", line 61, in __exit__
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     self.result = self.commit()
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     raise result.ex
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     txn.results.put(txn.do_commit())
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 92, in do_commit
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     command.run_idl(txn)
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 341, in run_idl
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     entity = self.api.lookup(self.table, self.entity)
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 181, in lookup
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     return self._lookup(table, record)
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 228, in _lookup
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     row = idlutils.row_by_value(self, rl.table, rl.column, record)
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource     raise RowNotFound(table=table, col=column, match=match)
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router_Port with name=lrp-7e0debbb-893c-420a-8569-d8fb98e6a16e
  2024-04-03 21:13:09.804 10 ERROR neutron.api.v2.resource [00m
  2024-04-03 21:13:09.807 10 INFO neutron.wsgi [None req-e5d08cdd-28e6-4231-a50c-7eafc1b8f942 70fc3b55af8c4386b80207dad11db5da dcec54844db44eedbd9667951a5ceb6b - - - -] 10.0.0.73 "PUT /v2.0/routers/1a3225bc-725d-400f-b9ef-3ffe7d4fb8d8/remove_router_interface HTTP/1.1" status: 500  len: 363 time: 2.0234911[00m
  ==============================

  I can't tell what's going on exactly, but it seems like for some
  reason the `lrp` is missing when we try and remove it.  The add seems
  to happen just fine though:

  ==============================
  2024-04-03 21:12:58.396 9 INFO neutron.wsgi [None req-0b71c77f-3844-4549-af1e-541f344f302e 70fc3b55af8c4386b80207dad11db5da dcec54844db44eedbd9667951a5ceb6b - - - -] 10.0.0.73 "PUT /v2.0/routers/1a3225bc-725d-400f-b9ef-3ffe7d4fb8d8/add_router_interface HTTP/1.1" status: 200  len: 503 time: 3.1546130[00m
  ==============================

  https://logs.atmosphere.dev/61/1061/46e91a5e83194215e35f7209ff92bf86597b4cd6/check/atmosphere-
  molecule-aio-ovn/5f98342/ubuntu-jammy/pod-logs/openstack/neutron-
  server-746d84b879-6q8kt/neutron-server.txt

  Since this is a CI environment, in order to reduce the resources we
  consume, we limit neutron to 2 workers.  Is it possible that it's not
  keeping up with the events coming in from OVN and we end up in this
  mismatch?

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2060163/+subscriptions