yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #96505
[Bug 2027742] Re: [RFE] unmanaged dynamic router resources - OVN
Reviewed: https://review.opendev.org/c/openstack/neutron/+/958679
Committed: https://opendev.org/openstack/neutron/commit/ff57491a976e0a8b4945f7dd55096370c744c76a
Submitter: "Zuul (22348)"
Branch: master
commit ff57491a976e0a8b4945f7dd55096370c744c76a
Author: Martin Kalcok <martin.kalcok@xxxxxxxxxxxxx>
Date: Wed Aug 27 19:17:04 2025 +0200
ovn_db_sync: Improve coexistence support.
neutron-ovn-db-sync-util synchronizes content between neutron's
database and OVN NB/SB databases. As a side-effect, it can sometimes
remove resources from OVN database that were not meant for neutron to
manage. Coexistence support [0] aims to avoid these scenarios.
The ovn_db_sync script already tries to avoid these unwanted removals
by checking for presence of well known neutron external_ids for
resources like "Logical Switch" and "Logical Switch Port" [1].
This change adds similar checks for:
* Logical Router Port
* Static Route
* Port Group
NAT rules are still missing the check because they don't have
"neutron:" external_ids to check.
In addition to the ovn_db_sync script, there is a 'maintenance' process
that periodically updates OVN resources. This change also update its
methods to not alter resources not owned by the neutron.
[0] https://specs.openstack.org/openstack/neutron-specs/specs/2024.1/ml2ovn-coexistence-support-ovn-ext-resources.html
[1] https://opendev.org/openstack/neutron/src/commit/f9067a719084710ee4f46fa31edb6a938e0dbbb0/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py#L308-L316
Closes-bug: #2027742
Change-Id: I1434700928779577073d1369c0a2983a4076cc0e
Signed-off-by: Martin Kalcok <martin.kalcok@xxxxxxxxxxxxx>
** Changed in: neutron
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2027742
Title:
[RFE] unmanaged dynamic router resources - OVN
Status in neutron:
Fix Released
Bug description:
Problem description
--------------------------------------
Regarding the conversation started in March [1] about the use of OVN interconnect with Neutron, we are testing the use of the OVN-IC to interconnect workloads in multiple AZs - with different OpenStacks deployments.
The Neutron default design does not allow to interconnect workloads
between different OpenStacks natively, and this could be a requirement
for a high availability cloud solution (if we are talking about Cloud
Region). Additionally, this OVN-IC solution allows interconnecting
other cloud solutions that use OVN as network backend - ovn-kube case.
We tested an OVN interconnect integrated with 3 OpenStack deployments
and it worked very well !!! in this case, we are considering direct L3
traffic at the router level between different network infrastructures.
To make it work we need to configure the TS and the LRP manually, as
well as examples from the ovn-kube project [2]. The problem with snat
(and FIPs) that was reported in the ovn-kube project has already been
fixed in OVN version 22.09, and in newer OVN versions it is not
necessary to modify anything in Neutron to pass the (--gateway-port)
because OVN finds the gateway port automatically.
At the moment the only issue found in Neutron is related to DB sync,
and it is natural because the LRP connected to the TS does not exist
in the DB. If the operator needs to restore the Neutron DB, the SYNC
repair command will remove the unmanaged externally added resources.
Note: The route learning has been tested with IPv4 and IPv6 addresses
and is working fine. A detail in the case of IPv6 is related to the
filter of routes learned via LLC addresses [3], take care of this
case.
SYNC_REPAIR - problem
* Static Routes (learned OVN-IC routes)
* Router Port -> Transit Switches
Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port found in OVN but not in Neutron, port_id=rt2-admin-tenant1
Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes [{'destination': '10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': '10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but not in Neutron
-------------------------------------
Proposed solution:
--------------------------------------
This RFE intends to implement a filter in the OVN mech_driver to
validate the external_ids key and not remove LRP's and static routes
present in the OVN backend without Neutron "key" in external_ids
register.
sync_routers_and_rports method:
LRPs case:
Filter the port list when iterating over existing OVN LRPs before
checking for existence in the Neutron DB.
LRP created by Neutron - example
_uuid : 1266061f-2a0b-4cb7-bcc4-14cb61a85173
enabled : []
external_ids : {"neutron:network_name"=neutron-d5169427-8fa0-4a25-a679-b33c604dbee1, "neutron:revision_number"="3", "neutron:router_name"="68917dd3-de44-465d-bf85-2722ca742ec0", "neutron:subnet_ids"="1ad00870-1efc-4eb0-b189-d69fbfae626f"}
gateway_chassis : []
ha_chassis_group : []
ipv6_prefix : []
ipv6_ra_configs : {}
mac : "fa:16:3e:d4:63:c5"
name : lrp-3581bee6-ea95-4971-862f-378546244487
networks : ["192.168.1.1/24"]
options : {}
peer : []
LRP externally created - example
ovn-nbctl list logical_router_port rt1-admin-tenant1
_uuid : 6a8bbf7b-4bf6-46a8-b0be-631154b87446
enabled : []
external_ids : {}
gateway_chassis : [b4487769-273e-4ba9-abd5-4743ff987f74]
ha_chassis_group : []
ipv6_prefix : []
ipv6_ra_configs : {}
mac : "aa:aa:aa:aa:ab:01"
name : rt1-admin-tenant1
networks : ["169.254.100.11/24", "fd00::1/64"]
options : {}
peer : []
LRPs created manually do not have Neutron keys in external_ids, the
idea here is to filter the return list and use the resource only when
external_ids contain Neutron keys.
Static routes case:
Additionally, OVN sync_db composes a list of static routes linked to a
router (get_all_logical_routers_with_rports). The proposal is to
extend the Neutron key filter in the external ids when creating the
return list. Similar to the router port case.
ovn-nbctl lr-route-list 078fd69b-f4c7-4469-a900-918d0a229bd1
IPv4 Routes
Route Table <main>:
10.0.1.0/24 169.254.100.12 dst-ip (learned)
10.0.2.0/24 169.254.100.13 dst-ip (learned)
0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes
Route Table <main>:
2001:db8:1::/64 fd00::2 dst-ip (learned)
2001:db8:1::/64 fd00::3 dst-ip (learned)
2801:80:3ea0:822::/64 fd00::2 dst-ip (learned)
2801:80:3ea0:823::/64 fd00::3 dst-ip (learned)
::/0 2001:db8:1:: dst-ip
With these specific changes, the management of these LRPs and learned
routes will be completely disassociated from Neutron perspective, and
the resources can be managed by the operator, creating and removing
links with remote routers on demand.
--------------------------------------
Regards,
Roberto
[1] https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.html
[2] https://github.com/kubeovn/kube-ovn/blob/v1.11.0/docs/cluster-interconnection.md
[3] https://github.com/ovn-
org/ovn/commit/cb0e2b3f44daeafb2f02f07289e3c410ee6ead28
------------------------------------------------------------------
Additional logs - Interconnect use case:
OpenStack 1
root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#
root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#
root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498
IPv4 Routes
Route Table <main>:
20.0.1.0/24 169.254.200.2 dst-ip (learned)
20.0.2.0/24 169.254.200.3 dst-ip (learned)
0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes
Route Table <main>:
::/0 fc00:ca5a:ca5a:8000:: dst-ip
root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07
IPv4 Routes
Route Table <main>:
10.0.1.0/24 169.254.100.2 dst-ip (learned)
10.0.2.0/24 169.254.100.3 dst-ip (learned)
0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes
Route Table <main>:
::/0 fc00:ca5a:ca5a:8000:: dst-ip
root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#
OpenStack 2
root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc
IPv4 Routes
Route Table <main>:
20.0.0.0/24 169.254.200.1 dst-ip (learned)
20.0.2.0/24 169.254.200.3 dst-ip (learned)
0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes
Route Table <main>:
::/0 fc00:ca5a:ca5a:8000:: dst-ip
root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list ce45f681-6454-43fe-974f-81344bb8113a
IPv4 Routes
Route Table <main>:
10.0.0.0/24 169.254.100.1 dst-ip (learned)
10.0.2.0/24 169.254.100.3 dst-ip (learned)
0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes
Route Table <main>:
::/0 fc00:ca5a:ca5a:8000:: dst-ip
OpenStack 3
root@os-infra-1-neutron-ovn-northd-container-f237db97:~#
root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3
IPv4 Routes
Route Table <main>:
20.0.0.0/24 169.254.200.1 dst-ip (learned)
20.0.1.0/24 169.254.200.2 dst-ip (learned)
0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes
Route Table <main>:
::/0 fc00:ca5a:ca5a:8000:: dst-ip
root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0
IPv4 Routes
Route Table <main>:
10.0.0.0/24 169.254.100.1 dst-ip (learned)
10.0.1.0/24 169.254.100.2 dst-ip (learned)
0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes
Route Table <main>:
::/0 fc00:ca5a:ca5a:8000:: dst-ip
OVN-IC Global database
root@ovn-global-db1:~# ovn-ic-sbctl show
availability-zone osp1
gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5
hostname: osp1-gwnode1
type: geneve
ip: 192.168.200.28
port admin-rt1-tenant1
transit switch: admin-tenant1
address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 fe80::1/64"]
port admin-rt1-tenant1_1
transit switch: admin-tenant1_1
address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"]
availability-zone osp2
gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8
hostname: osp2-gwnode1
type: geneve
ip: 192.168.200.128
port admin-rt2-tenant1
transit switch: admin-tenant1
address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 fe80::2/64"]
port admin-rt2-tenant1_1
transit switch: admin-tenant1_1
address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"]
availability-zone osp3
gateway 97595af9-7896-40d0-a883-beadbff1aa5b
hostname: osp3-gwnode1
type: geneve
ip: 192.168.200.228
port admin-rt3-tenant1
transit switch: admin-tenant1
address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 fe80::3/64"]
port admin-rt3-tenant1_1
transit switch: admin-tenant1_1
address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"]
--------------------------------------
Reference design:
# Global database OVN-IC
ovn-ic-nbctl ts-add admin-tenant1
**** OpenStack 1 ***********
# OVN central 1
ovn-nbctl set NB_Global . name=osp1
ovn-nbctl set NB_Global . options:ic-route-adv=true \
options:ic-route-learn=true
ovn-nbctl lrp-add NEUTRON_ROUTER rt1-admin-tenant1 aa:aa:aa:aa:aa:01 169.254.100.1/24
ovn-nbctl lsp-add admin-tenant1 admin-rt1-tenant1 -- \
lsp-set-addresses admin-rt1-tenant1 router -- \
lsp-set-type admin-rt1-tenant1 router -- \
lsp-set-options admin-rt1-tenant1 router-port=rt1-admin-tenant1
ovn-nbctl lrp-set-gateway-chassis rt1-admin-tenant1
832b6c0d-13ce-4600-ab37-78516d8ec4c5 1
ovn-nbctl set NB_Global . options:ic-route-blacklist=200.200.200.0/24
# Gateway node - Openstack 1
ovs-vsctl set open_vswitch . external_ids:ovn-is-interconn=true
**** OpenStack 2 ***********
ovn-nbctl set NB_Global . name=osp2
ovn-nbctl set NB_Global . options:ic-route-adv=true \
options:ic-route-learn=true
ovn-nbctl lrp-add NEUTRON_ROUTER rt2-admin-tenant1 aa:aa:aa:aa:aa:02 169.254.100.2/24
ovn-nbctl lsp-add admin-tenant1 admin-rt2-tenant1 -- \
lsp-set-addresses admin-rt2-tenant1 router -- \
lsp-set-type admin-rt2-tenant1 router -- \
lsp-set-options admin-rt2-tenant1 router-port=rt2-admin-tenant1
ovn-nbctl lrp-set-gateway-chassis rt2-admin-tenant1 17ffabdf-
cf47-41ab-9539-d326c13c4ca8 1
ovn-nbctl set NB_Global . options:ic-route-blacklist=200.200.200.0/24
# Gateway node
ovs-vsctl set open_vswitch . external_ids:ovn-is-interconn=true
**** OpenStack 3 ***********
ovn-nbctl set NB_Global . name=osp3
ovn-nbctl set NB_Global . options:ic-route-adv=true \
options:ic-route-learn=true
ovn-nbctl lrp-add NEUTRON_ROUTER rt3-admin-tenant1 aa:aa:aa:aa:aa:03 169.254.100.3/24
ovn-nbctl lsp-add admin-tenant1 admin-rt3-tenant1 -- \
lsp-set-addresses admin-rt3-tenant1 router -- \
lsp-set-type admin-rt3-tenant1 router -- \
lsp-set-options admin-rt3-tenant1 router-port=rt3-admin-tenant1
ovn-nbctl lrp-set-gateway-chassis rt3-admin-tenant1
97595af9-7896-40d0-a883-beadbff1aa5b 1
ovn-nbctl set NB_Global . options:ic-route-blacklist=200.200.200.0/24
# Gateway node
ovs-vsctl set open_vswitch . external_ids:ovn-is-interconn=true
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2027742/+subscriptions
References
