yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2124269] [NEW] Can't bind an HA router_gateway port to chassis

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Walid Moghrabi <2124269@xxxxxxxxxxxxxxxxxx>
Date: Wed, 17 Sep 2025 12:28:33 -0000
Reply-to: Bug 2124269 <2124269@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Running Openstack 2024.1 "Caracal" on Ubuntu Jammy deployed with FCE
(MaaS 3.6, Juju 3.6) and OVN 24.03, we're facing an issue when using
private network+FIP based tenants.

When creating such tenant, we need to create a router with a gateway subnet.
When this object is created we see these error messages in neutron logs :


==============================================================================================
2025-09-17 12:04:57.099 4411 INFO neutron.db.ovn_revision_numbers_db [None req-dc2f485e-6938-4c4c-a776-91cb1578809d 3d1d518e2e534286a94a3ad73e9ec00a 900e2c26f77041d19163704cbd8243f9 - - 6b4f95d250fe4dd08b9a4c23e61cce88 6b4f95d250fe4dd08b9a4c23e61cce88] Successfully bumped revision number for resource 0626d085-bfb4-4019-8ff5-b8dd8259bf0d (type: ports) to 2
2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command [None req-541af44e-4c9f-4e19-88ff-b777bf201710 - - - - - -] Error executing command (DbGetCommand): ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router_Port with name=lrp-e9c93a54-539e-4f1e-97b5-be16943c7c6f
2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 47, in execute
2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command     self.run_idl(None)
2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 235, in run_idl
2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command     record = self.api.lookup(self.table, self.record)
2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 184, in lookup
2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command     return self._lookup(table, record)
2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 235, in _lookup
2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command     row = idlutils.row_by_value(self, rl.table, rl.column, record)
2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command     raise RowNotFound(table=table, col=column, match=match)
2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router_Port with name=lrp-e9c93a54-539e-4f1e-97b5-be16943c7c6f
2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command 
2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager [None req-541af44e-4c9f-4e19-88ff-b777bf201710 - - - - - -] Error during notification for neutron.services.ovn_l3.plugin.OVNL3RouterPlugin._port_update-8663864275972 port, after_update: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router_Port with name=lrp-e9c93a54-539e-4f1e-97b5-be16943c7c6f
2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 189, in _notify_loop
2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager     callback.method(resource, event, trigger, payload=payload)
2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron/services/ovn_l3/plugin.py", line 391, in _port_update
2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager     l3plugin._ovn_client.update_router_port(context,
2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1866, in update_router_port
2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager     self._update_lrouter_port(context, port, if_exists=if_exists,
2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1843, in _update_lrouter_port
2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager     external_ids = self._nb_idl.db_get(
2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 47, in execute
2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager     self.run_idl(None)
2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 235, in run_idl
2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager     record = self.api.lookup(self.table, self.record)
2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 184, in lookup
2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager     return self._lookup(table, record)
2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 235, in _lookup
2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager     row = idlutils.row_by_value(self, rl.table, rl.column, record)
2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager     raise RowNotFound(table=table, col=column, match=match)
2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router_Port with name=lrp-e9c93a54-539e-4f1e-97b5-be16943c7c6f
2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager 
2025-09-17 12:04:57.142 4412 WARNING neutron_lib.context [None req-142f99a3-4935-48fc-8286-6d8d19d1fba8 6e3294ba50944dd7b489d144d2f652ec 8933ba1c7be94c51b89d44e2843592c8 - - 6b4f95d250fe4dd08b9a4c23e61cce88 6b4f95d250fe4dd08b9a4c23e61cce88] Method 'is_advsvc' is deprecated since 2023.2 release (neutron-lib 3.8.0) and will be removed once support for the old RBAC API policies will be removed from Neutron. Please use method 'is_service_role' instead.
2025-09-17 12:04:57.206 4411 INFO neutron.db.ovn_revision_numbers_db [None req-dc2f485e-6938-4c4c-a776-91cb1578809d 3d1d518e2e534286a94a3ad73e9ec00a 900e2c26f77041d19163704cbd8243f9 - - 6b4f95d250fe4dd08b9a4c23e61cce88 6b4f95d250fe4dd08b9a4c23e61cce88] Successfully bum
==============================================================================================


VMs are created sucessfully and private network works as expected, VMs talk to each other on the private network, private gateway is also reachable.

However, traffic don't go through FIP gateway, VMs are not accessible
through their FIP and they can't talk outside the private network.


We have a test project with a private network and we had the same issue
and we found out that we were missing a HA_Chassis_Group in the OVN NB
database. As a manual workaround, we needed to apply those commands :

==============================================================================================
# 4cfa62ce-ba4b-4d9d-a1db-4133e962b542 is the port associated to the router gateway

sudo ovn-nbctl lrp-set-ha-chassis-group lrp-4cfa62ce-
ba4b-4d9d-a1db-4133e962b542 hcg-fip

HCG=$(sudo ovn-nbctl --db=unix:/var/run/ovn/ovnnb_db.sock --bare
--columns=_uuid  find HA_Chassis_Group name=hcg-fip)

sudo ovn-nbctl --db=unix:/var/run/ovn/ovnnb_db.sock   set Logical_Router_Port lrp-4cfa62ce-ba4b-4d9d-a1db-4133e962b542 ha_chassis_group=$HCG
==============================================================================================

Once this was applied, our test VMs were reachable through their FIP.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2124269

Title:
  Can't bind an HA router_gateway port to chassis

Status in neutron:
  New

Bug description:
  Running Openstack 2024.1 "Caracal" on Ubuntu Jammy deployed with FCE
  (MaaS 3.6, Juju 3.6) and OVN 24.03, we're facing an issue when using
  private network+FIP based tenants.

  When creating such tenant, we need to create a router with a gateway subnet.
  When this object is created we see these error messages in neutron logs :

  
  ==============================================================================================
  2025-09-17 12:04:57.099 4411 INFO neutron.db.ovn_revision_numbers_db [None req-dc2f485e-6938-4c4c-a776-91cb1578809d 3d1d518e2e534286a94a3ad73e9ec00a 900e2c26f77041d19163704cbd8243f9 - - 6b4f95d250fe4dd08b9a4c23e61cce88 6b4f95d250fe4dd08b9a4c23e61cce88] Successfully bumped revision number for resource 0626d085-bfb4-4019-8ff5-b8dd8259bf0d (type: ports) to 2
  2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command [None req-541af44e-4c9f-4e19-88ff-b777bf201710 - - - - - -] Error executing command (DbGetCommand): ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router_Port with name=lrp-e9c93a54-539e-4f1e-97b5-be16943c7c6f
  2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
  2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 47, in execute
  2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command     self.run_idl(None)
  2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 235, in run_idl
  2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command     record = self.api.lookup(self.table, self.record)
  2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 184, in lookup
  2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command     return self._lookup(table, record)
  2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 235, in _lookup
  2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command     row = idlutils.row_by_value(self, rl.table, rl.column, record)
  2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
  2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command     raise RowNotFound(table=table, col=column, match=match)
  2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router_Port with name=lrp-e9c93a54-539e-4f1e-97b5-be16943c7c6f
  2025-09-17 12:04:57.110 4413 ERROR ovsdbapp.backend.ovs_idl.command 
  2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager [None req-541af44e-4c9f-4e19-88ff-b777bf201710 - - - - - -] Error during notification for neutron.services.ovn_l3.plugin.OVNL3RouterPlugin._port_update-8663864275972 port, after_update: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router_Port with name=lrp-e9c93a54-539e-4f1e-97b5-be16943c7c6f
  2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
  2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 189, in _notify_loop
  2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager     callback.method(resource, event, trigger, payload=payload)
  2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron/services/ovn_l3/plugin.py", line 391, in _port_update
  2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager     l3plugin._ovn_client.update_router_port(context,
  2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1866, in update_router_port
  2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager     self._update_lrouter_port(context, port, if_exists=if_exists,
  2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1843, in _update_lrouter_port
  2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager     external_ids = self._nb_idl.db_get(
  2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 47, in execute
  2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager     self.run_idl(None)
  2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 235, in run_idl
  2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager     record = self.api.lookup(self.table, self.record)
  2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 184, in lookup
  2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager     return self._lookup(table, record)
  2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 235, in _lookup
  2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager     row = idlutils.row_by_value(self, rl.table, rl.column, record)
  2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
  2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager     raise RowNotFound(table=table, col=column, match=match)
  2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router_Port with name=lrp-e9c93a54-539e-4f1e-97b5-be16943c7c6f
  2025-09-17 12:04:57.111 4413 ERROR neutron_lib.callbacks.manager 
  2025-09-17 12:04:57.142 4412 WARNING neutron_lib.context [None req-142f99a3-4935-48fc-8286-6d8d19d1fba8 6e3294ba50944dd7b489d144d2f652ec 8933ba1c7be94c51b89d44e2843592c8 - - 6b4f95d250fe4dd08b9a4c23e61cce88 6b4f95d250fe4dd08b9a4c23e61cce88] Method 'is_advsvc' is deprecated since 2023.2 release (neutron-lib 3.8.0) and will be removed once support for the old RBAC API policies will be removed from Neutron. Please use method 'is_service_role' instead.
  2025-09-17 12:04:57.206 4411 INFO neutron.db.ovn_revision_numbers_db [None req-dc2f485e-6938-4c4c-a776-91cb1578809d 3d1d518e2e534286a94a3ad73e9ec00a 900e2c26f77041d19163704cbd8243f9 - - 6b4f95d250fe4dd08b9a4c23e61cce88 6b4f95d250fe4dd08b9a4c23e61cce88] Successfully bum
  ==============================================================================================

  
  VMs are created sucessfully and private network works as expected, VMs talk to each other on the private network, private gateway is also reachable.

  However, traffic don't go through FIP gateway, VMs are not accessible
  through their FIP and they can't talk outside the private network.


  We have a test project with a private network and we had the same
  issue and we found out that we were missing a HA_Chassis_Group in the
  OVN NB database. As a manual workaround, we needed to apply those
  commands :

  ==============================================================================================
  # 4cfa62ce-ba4b-4d9d-a1db-4133e962b542 is the port associated to the router gateway

  sudo ovn-nbctl lrp-set-ha-chassis-group lrp-4cfa62ce-
  ba4b-4d9d-a1db-4133e962b542 hcg-fip

  HCG=$(sudo ovn-nbctl --db=unix:/var/run/ovn/ovnnb_db.sock --bare
  --columns=_uuid  find HA_Chassis_Group name=hcg-fip)

  sudo ovn-nbctl --db=unix:/var/run/ovn/ovnnb_db.sock   set Logical_Router_Port lrp-4cfa62ce-ba4b-4d9d-a1db-4133e962b542 ha_chassis_group=$HCG
  ==============================================================================================

  Once this was applied, our test VMs were reachable through their FIP.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2124269/+subscriptions