← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 2024251] [NEW] Adding a static route to router returns internal server error

 

Public bug reported:

In a multi-tenant setup, with a shared provider network, adding a static
route to a router returns internal server error.  Note: IP addresses are
obfucated

openstack router set --route destination='300.300.0.0/16',gateway='172.31.96.1' tickets_private_router
HttpException: 500: Server Error for url: https://controller01:9696/v2.0/routers/78c02b65-9819-48d9-9700-f045a28909e0, Request Failed: internal server error while processing your request.

Openstack release: Antelope (2023.1)
Neutron release: 22.0.0-1
OVN: ovn22.09
Distribution: Centos 9 Streams
Manual Installation.

The router:
# openstack router show tickets_private_router
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                   | Value                                                                                                                                                                                     |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up          | UP                                                                                                                                                                                        |
| availability_zone_hints |                                                                                                                                                                                           |
| availability_zones      |                                                                                                                                                                                           |
| created_at              | 2023-06-14T17:49:52Z                                                                                                                                                                      |
| description             |                                                                                                                                                                                           |
| enable_ndp_proxy        | None                                                                                                                                                                                      |
| external_gateway_info   | {"network_id": "b0c5df5c-8d5c-4fc2-babf-829421243017", "external_fixed_ips": [{"subnet_id": "4b9540d4-e3d4-403d-9918-98f111d8fedd", "ip_address": "172.31.97.226"}], "enable_snat": true} |
| flavor_id               | None                                                                                                                                                                                      |
| id                      | 78c02b65-9819-48d9-9700-f045a28909e0                                                                                                                                                      |
| interfaces_info         | [{"port_id": "7f628431-87bf-460e-88a0-b0426040a07c", "ip_address": "172.31.100.1", "subnet_id": "dd636d3d-7dc9-472e-b340-095ff85db1c2"}]                                                  |
| name                    | tickets_private_router                                                                                                                                                                    |
| project_id              | d9e5cee8a9e94a1883376ce48e676ca4                                                                                                                                                          |
| revision_number         | 10                                                                                                                                                                                        |
| routes                  |                                                                                                                                                                                           |
| status                  | ACTIVE                                                                                                                                                                                    |
| tags                    |                                                                                                                                                                                           |
| tenant_id               | d9e5cee8a9e94a1883376ce48e676ca4                                                                                                                                                          |
| updated_at              | 2023-06-16T16:58:52Z                                                                                                                                                                      |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

In the logs for neutron-server:

2023-06-16 13:17:39.769 10 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-beb7e61c-a6da-4055-b41f-2bc728061394 - - - - - -] OVSDB Error: {"details":"Table Logical_Router column static_routes row 3fae901a-c2e3-468b-b8af-4343b6c3b937 references nonexistent row 57183168-288c-4700-b651-a87e3f2bf4eb in table Logical_Router_Static_Route.","error":"referential integrity violation"}
neutron-server_1  | 2023-06-16 13:17:39.769 10 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-79ee6c1d-3c7b-4578-aedb-9af1dbb13e60 13d6e02fd8b74011b0e5d600d5c64300 cbadbe9f0cce4066b722422b3844a09e - - default default] Traceback (most recent call last):
neutron-server_1  |   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
neutron-server_1  |     txn.results.put(txn.do_commit())
neutron-server_1  |   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
neutron-server_1  |     raise RuntimeError(msg)
neutron-server_1  | RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 3fae901a-c2e3-468b-b8af-4343b6c3b937 references nonexistent row 57183168-288c-4700-b651-a87e3f2bf4eb in table Logical_Router_Static_Route.","error":"referential integrity violation"}
neutron-server_1  | 
neutron-server_1  | 2023-06-16 13:17:39.769 10 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-79ee6c1d-3c7b-4578-aedb-9af1dbb13e60 13d6e02fd8b74011b0e5d600d5c64300 cbadbe9f0cce4066b722422b3844a09e - - default default] Unable to update router 78c02b65-9819-48d9-9700-f045a28909e0. Error: OVSDB Error: {"details":"Table Logical_Router column static_routes row 3fae901a-c2e3-468b-b8af-4343b6c3b937 references nonexistent row 57183168-288c-4700-b651-a87e3f2bf4eb in table Logical_Router_Static_Route.","error":"referential integrity violation"}: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 3fae901a-c2e3-468b-b8af-4343b6c3b937 references nonexistent row 57183168-288c-4700-b651-a87e3f2bf4eb in table Logical_Router_Static_Route.","error":"referential integrity violation"}
neutron-server_1  | 2023-06-16 13:17:39.770 10 ERROR neutron.services.ovn_l3.plugin [None req-79ee6c1d-3c7b-4578-aedb-9af1dbb13e60 13d6e02fd8b74011b0e5d600d5c64300 cbadbe9f0cce4066b722422b3844a09e - - default default] Unable to update lrouter for 78c02b65-9819-48d9-9700-f045a28909e0: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 3fae901a-c2e3-468b-b8af-4343b6c3b937 references nonexistent row 57183168-288c-4700-b651-a87e3f2bf4eb in table Logical_Router_Static_Route.","error":"referential integrity violation"}

Stack trace follows.

This will ocassionaly succeed on other routers after a delay of 10+
minutes, but some routers refuse to accept the staic route.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2024251

Title:
  Adding a static route to router returns internal server error

Status in neutron:
  New

Bug description:
  In a multi-tenant setup, with a shared provider network, adding a
  static route to a router returns internal server error.  Note: IP
  addresses are obfucated

  openstack router set --route destination='300.300.0.0/16',gateway='172.31.96.1' tickets_private_router
  HttpException: 500: Server Error for url: https://controller01:9696/v2.0/routers/78c02b65-9819-48d9-9700-f045a28909e0, Request Failed: internal server error while processing your request.

  Openstack release: Antelope (2023.1)
  Neutron release: 22.0.0-1
  OVN: ovn22.09
  Distribution: Centos 9 Streams
  Manual Installation.

  The router:
  # openstack router show tickets_private_router
  +-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  | Field                   | Value                                                                                                                                                                                     |
  +-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  | admin_state_up          | UP                                                                                                                                                                                        |
  | availability_zone_hints |                                                                                                                                                                                           |
  | availability_zones      |                                                                                                                                                                                           |
  | created_at              | 2023-06-14T17:49:52Z                                                                                                                                                                      |
  | description             |                                                                                                                                                                                           |
  | enable_ndp_proxy        | None                                                                                                                                                                                      |
  | external_gateway_info   | {"network_id": "b0c5df5c-8d5c-4fc2-babf-829421243017", "external_fixed_ips": [{"subnet_id": "4b9540d4-e3d4-403d-9918-98f111d8fedd", "ip_address": "172.31.97.226"}], "enable_snat": true} |
  | flavor_id               | None                                                                                                                                                                                      |
  | id                      | 78c02b65-9819-48d9-9700-f045a28909e0                                                                                                                                                      |
  | interfaces_info         | [{"port_id": "7f628431-87bf-460e-88a0-b0426040a07c", "ip_address": "172.31.100.1", "subnet_id": "dd636d3d-7dc9-472e-b340-095ff85db1c2"}]                                                  |
  | name                    | tickets_private_router                                                                                                                                                                    |
  | project_id              | d9e5cee8a9e94a1883376ce48e676ca4                                                                                                                                                          |
  | revision_number         | 10                                                                                                                                                                                        |
  | routes                  |                                                                                                                                                                                           |
  | status                  | ACTIVE                                                                                                                                                                                    |
  | tags                    |                                                                                                                                                                                           |
  | tenant_id               | d9e5cee8a9e94a1883376ce48e676ca4                                                                                                                                                          |
  | updated_at              | 2023-06-16T16:58:52Z                                                                                                                                                                      |
  +-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

  In the logs for neutron-server:

  2023-06-16 13:17:39.769 10 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-beb7e61c-a6da-4055-b41f-2bc728061394 - - - - - -] OVSDB Error: {"details":"Table Logical_Router column static_routes row 3fae901a-c2e3-468b-b8af-4343b6c3b937 references nonexistent row 57183168-288c-4700-b651-a87e3f2bf4eb in table Logical_Router_Static_Route.","error":"referential integrity violation"}
  neutron-server_1  | 2023-06-16 13:17:39.769 10 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-79ee6c1d-3c7b-4578-aedb-9af1dbb13e60 13d6e02fd8b74011b0e5d600d5c64300 cbadbe9f0cce4066b722422b3844a09e - - default default] Traceback (most recent call last):
  neutron-server_1  |   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
  neutron-server_1  |     txn.results.put(txn.do_commit())
  neutron-server_1  |   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
  neutron-server_1  |     raise RuntimeError(msg)
  neutron-server_1  | RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 3fae901a-c2e3-468b-b8af-4343b6c3b937 references nonexistent row 57183168-288c-4700-b651-a87e3f2bf4eb in table Logical_Router_Static_Route.","error":"referential integrity violation"}
  neutron-server_1  | 
  neutron-server_1  | 2023-06-16 13:17:39.769 10 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-79ee6c1d-3c7b-4578-aedb-9af1dbb13e60 13d6e02fd8b74011b0e5d600d5c64300 cbadbe9f0cce4066b722422b3844a09e - - default default] Unable to update router 78c02b65-9819-48d9-9700-f045a28909e0. Error: OVSDB Error: {"details":"Table Logical_Router column static_routes row 3fae901a-c2e3-468b-b8af-4343b6c3b937 references nonexistent row 57183168-288c-4700-b651-a87e3f2bf4eb in table Logical_Router_Static_Route.","error":"referential integrity violation"}: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 3fae901a-c2e3-468b-b8af-4343b6c3b937 references nonexistent row 57183168-288c-4700-b651-a87e3f2bf4eb in table Logical_Router_Static_Route.","error":"referential integrity violation"}
  neutron-server_1  | 2023-06-16 13:17:39.770 10 ERROR neutron.services.ovn_l3.plugin [None req-79ee6c1d-3c7b-4578-aedb-9af1dbb13e60 13d6e02fd8b74011b0e5d600d5c64300 cbadbe9f0cce4066b722422b3844a09e - - default default] Unable to update lrouter for 78c02b65-9819-48d9-9700-f045a28909e0: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 3fae901a-c2e3-468b-b8af-4343b6c3b937 references nonexistent row 57183168-288c-4700-b651-a87e3f2bf4eb in table Logical_Router_Static_Route.","error":"referential integrity violation"}

  Stack trace follows.

  This will ocassionaly succeed on other routers after a delay of 10+
  minutes, but some routers refuse to accept the staic route.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2024251/+subscriptions



Follow ups