← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 2027742] [NEW] [RFE] unmanaged dynamic router resources - OVN

 

Public bug reported:


Problem description
--------------------------------------
Regarding the conversation started in March [1] about the use of OVN interconnect with Neutron, we are testing the use of the OVN-IC to interconnect workloads in multiple AZs - with different OpenStacks deployments.    

The Neutron default design does not allow to interconnect workloads
between different OpenStacks natively, and this could be a requirement
for a high availability cloud solution (if we are talking about Cloud
Region). Additionally, this OVN-IC solution allows interconnecting other
cloud solutions that use OVN as network backend - ovn-kube case.

We tested an OVN interconnect integrated with 3 OpenStack deployments
and it worked very well !!! in this case, we are considering direct L3
traffic at the router level between different network infrastructures.

To make it work we need to configure the TS and the LRP manually, as
well as examples from the ovn-kube project [2]. The problem with snat
(and FIPs) that was reported in the ovn-kube project has already been
fixed in OVN version 22.09, and in newer OVN versions it is not
necessary to modify anything in Neutron to pass the (--gateway-port)
because OVN finds the gateway port automatically.

At the moment the only issue found in Neutron is related to DB sync, and
it is natural because the LRP connected to the TS does not exist in the
DB. If the operator needs to restore the Neutron DB, the SYNC repair
command will remove the unmanaged externally added resources.

Note: The route learning has been tested with IPv4 and IPv6 addresses
and is working fine. A detail in the case of IPv6 is related to the
filter of routes learned via LLC addresses [3], take care of this case.

SYNC_REPAIR - problem

* Static Routes (learned OVN-IC routes)
* Router Port -> Transit Switches

Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port found in OVN but not in Neutron, port_id=rt2-admin-tenant1
Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes [{'destination': '10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': '10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but not in Neutron

-------------------------------------


Proposed solution:
--------------------------------------

This RFE intends to implement a filter in the OVN mech_driver to
validate the external_ids key and not remove LRP's and static routes
present in the OVN backend without Neutron "key" in external_ids
register.

sync_routers_and_rports method:

LRPs case:

Filter the port list when iterating over existing OVN LRPs before
checking for existence in the Neutron DB.

ovn-nbctl list logical_router_port rt1-admin-tenant1 
_uuid               : 6a8bbf7b-4bf6-46a8-b0be-631154b87446
enabled             : []
external_ids        : {}
gateway_chassis     : [b4487769-273e-4ba9-abd5-4743ff987f74]
ha_chassis_group    : []
ipv6_prefix         : []
ipv6_ra_configs     : {}
mac                 : "aa:aa:aa:aa:ab:01"
name                : rt1-admin-tenant1
networks            : ["169.254.100.11/24", "fd00::1/64"]
options             : {}
peer                : []


Static routes case:

Additionally, OVN sync_db composes a list of static routes linked to a
router (get_all_logical_routers_with_rports). The proposal is to extend
the Neutron key filter in the external ids when creating the return
list. Similar to the router port case.

ovn-nbctl lr-route-list 078fd69b-f4c7-4469-a900-918d0a229bd1
IPv4 Routes
Route Table <main>:
              10.0.1.0/24            169.254.100.12 dst-ip (learned)
              10.0.2.0/24            169.254.100.13 dst-ip (learned)
                0.0.0.0/0             200.200.200.1 dst-ip

IPv6 Routes
Route Table <main>:
          2001:db8:1::/64                   fd00::2 dst-ip (learned)
          2001:db8:1::/64                   fd00::3 dst-ip (learned)
    2801:80:3ea0:822::/64                   fd00::2 dst-ip (learned)
    2801:80:3ea0:823::/64                   fd00::3 dst-ip (learned)
                     ::/0              2001:db8:1:: dst-ip

With these specific changes, the management of these LRPs and learned
routes will be completely disassociated from Neutron perspective, and
the resources can be managed by the operator, creating and removing
links with remote routers on demand.

--------------------------------------


Regards,
Roberto


[1] https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.html
[2] https://github.com/kubeovn/kube-ovn/blob/v1.11.0/docs/cluster-interconnection.md

[3] https://github.com/ovn-
org/ovn/commit/cb0e2b3f44daeafb2f02f07289e3c410ee6ead28


------------------------------------------------------------------

Additional logs - Interconnect use case:

OpenStack 1

root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#
root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#
root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498
IPv4 Routes
Route Table <main>:
              20.0.1.0/24             169.254.200.2 dst-ip (learned)
              20.0.2.0/24             169.254.200.3 dst-ip (learned)
                0.0.0.0/0             200.200.200.1 dst-ip

IPv6 Routes
Route Table <main>:
                     ::/0     fc00:ca5a:ca5a:8000:: dst-ip
root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07
IPv4 Routes
Route Table <main>:
              10.0.1.0/24             169.254.100.2 dst-ip (learned)
              10.0.2.0/24             169.254.100.3 dst-ip (learned)
                0.0.0.0/0             200.200.200.1 dst-ip

IPv6 Routes
Route Table <main>:
                     ::/0     fc00:ca5a:ca5a:8000:: dst-ip
root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#


OpenStack 2

root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc
IPv4 Routes
Route Table <main>:
              20.0.0.0/24             169.254.200.1 dst-ip (learned)
              20.0.2.0/24             169.254.200.3 dst-ip (learned)
                0.0.0.0/0             200.200.200.1 dst-ip

IPv6 Routes
Route Table <main>:
                     ::/0     fc00:ca5a:ca5a:8000:: dst-ip
root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list ce45f681-6454-43fe-974f-81344bb8113a
IPv4 Routes
Route Table <main>:
              10.0.0.0/24             169.254.100.1 dst-ip (learned)
              10.0.2.0/24             169.254.100.3 dst-ip (learned)
                0.0.0.0/0             200.200.200.1 dst-ip

IPv6 Routes
Route Table <main>:
                     ::/0     fc00:ca5a:ca5a:8000:: dst-ip


OpenStack 3

root@os-infra-1-neutron-ovn-northd-container-f237db97:~#
root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list  cfa259d6-311f-4409-bcf2-79a929835cb3
IPv4 Routes
Route Table <main>:
              20.0.0.0/24             169.254.200.1 dst-ip (learned)
              20.0.1.0/24             169.254.200.2 dst-ip (learned)
                0.0.0.0/0             200.200.200.1 dst-ip

IPv6 Routes
Route Table <main>:
                     ::/0     fc00:ca5a:ca5a:8000:: dst-ip
root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list  c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0
IPv4 Routes
Route Table <main>:
              10.0.0.0/24             169.254.100.1 dst-ip (learned)
              10.0.1.0/24             169.254.100.2 dst-ip (learned)
                0.0.0.0/0             200.200.200.1 dst-ip

IPv6 Routes
Route Table <main>:
                     ::/0     fc00:ca5a:ca5a:8000:: dst-ip


OVN-IC Global database

root@ovn-global-db1:~# ovn-ic-sbctl show
availability-zone osp1
    gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5
        hostname: osp1-gwnode1
        type: geneve
            ip: 192.168.200.28
        port admin-rt1-tenant1
            transit switch: admin-tenant1
            address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 fe80::1/64"]
        port admin-rt1-tenant1_1
            transit switch: admin-tenant1_1
            address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"]
availability-zone osp2
    gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8
        hostname: osp2-gwnode1
        type: geneve
            ip: 192.168.200.128
        port admin-rt2-tenant1
            transit switch: admin-tenant1
            address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 fe80::2/64"]
        port admin-rt2-tenant1_1
            transit switch: admin-tenant1_1
            address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"]
availability-zone osp3
    gateway 97595af9-7896-40d0-a883-beadbff1aa5b
        hostname: osp3-gwnode1
        type: geneve
            ip: 192.168.200.228
        port admin-rt3-tenant1
            transit switch: admin-tenant1
            address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 fe80::3/64"]
        port admin-rt3-tenant1_1
            transit switch: admin-tenant1_1
            address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"]
            
--------------------------------------


Reference design:

# Global database OVN-IC

ovn-ic-nbctl ts-add admin-tenant1


**** OpenStack 1 ***********

# OVN central 1

ovn-nbctl set NB_Global . name=osp1

ovn-nbctl set NB_Global . options:ic-route-adv=true \
                            options:ic-route-learn=true
                            
                            
ovn-nbctl lrp-add NEUTRON_ROUTER rt1-admin-tenant1 aa:aa:aa:aa:aa:01 169.254.100.1/24
ovn-nbctl lsp-add admin-tenant1 admin-rt1-tenant1  -- \
        lsp-set-addresses admin-rt1-tenant1 router -- \
        lsp-set-type admin-rt1-tenant1 router -- \
        lsp-set-options admin-rt1-tenant1 router-port=rt1-admin-tenant1

ovn-nbctl lrp-set-gateway-chassis rt1-admin-tenant1
832b6c0d-13ce-4600-ab37-78516d8ec4c5 1

ovn-nbctl set NB_Global . options:ic-route-blacklist=200.200.200.0/24

# Gateway node - Openstack 1

ovs-vsctl set open_vswitch . external_ids:ovn-is-interconn=true

     
**** OpenStack 2 ***********   
     
ovn-nbctl set NB_Global . name=osp2

ovn-nbctl set NB_Global . options:ic-route-adv=true \
                            options:ic-route-learn=true


ovn-nbctl lrp-add NEUTRON_ROUTER  rt2-admin-tenant1 aa:aa:aa:aa:aa:02 169.254.100.2/24
ovn-nbctl lsp-add admin-tenant1 admin-rt2-tenant1  -- \
        lsp-set-addresses admin-rt2-tenant1 router -- \
        lsp-set-type admin-rt2-tenant1 router -- \
        lsp-set-options admin-rt2-tenant1 router-port=rt2-admin-tenant1

ovn-nbctl lrp-set-gateway-chassis rt2-admin-tenant1 17ffabdf-
cf47-41ab-9539-d326c13c4ca8 1

ovn-nbctl set NB_Global . options:ic-route-blacklist=200.200.200.0/24

# Gateway node

ovs-vsctl set open_vswitch . external_ids:ovn-is-interconn=true

**** OpenStack 3 ***********

ovn-nbctl set NB_Global . name=osp3

ovn-nbctl set NB_Global . options:ic-route-adv=true \
                            options:ic-route-learn=true

ovn-nbctl lrp-add NEUTRON_ROUTER  rt3-admin-tenant1 aa:aa:aa:aa:aa:03 169.254.100.3/24
ovn-nbctl lsp-add admin-tenant1 admin-rt3-tenant1  -- \
        lsp-set-addresses admin-rt3-tenant1 router -- \
        lsp-set-type admin-rt3-tenant1 router -- \
        lsp-set-options admin-rt3-tenant1 router-port=rt3-admin-tenant1

ovn-nbctl lrp-set-gateway-chassis rt3-admin-tenant1
97595af9-7896-40d0-a883-beadbff1aa5b 1

ovn-nbctl set NB_Global . options:ic-route-blacklist=200.200.200.0/24
        
# Gateway node

ovs-vsctl set open_vswitch . external_ids:ovn-is-interconn=true

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2027742

Title:
  [RFE] unmanaged dynamic router resources - OVN

Status in neutron:
  New

Bug description:
  
  Problem description
  --------------------------------------
  Regarding the conversation started in March [1] about the use of OVN interconnect with Neutron, we are testing the use of the OVN-IC to interconnect workloads in multiple AZs - with different OpenStacks deployments.    

  The Neutron default design does not allow to interconnect workloads
  between different OpenStacks natively, and this could be a requirement
  for a high availability cloud solution (if we are talking about Cloud
  Region). Additionally, this OVN-IC solution allows interconnecting
  other cloud solutions that use OVN as network backend - ovn-kube case.

  We tested an OVN interconnect integrated with 3 OpenStack deployments
  and it worked very well !!! in this case, we are considering direct L3
  traffic at the router level between different network infrastructures.

  To make it work we need to configure the TS and the LRP manually, as
  well as examples from the ovn-kube project [2]. The problem with snat
  (and FIPs) that was reported in the ovn-kube project has already been
  fixed in OVN version 22.09, and in newer OVN versions it is not
  necessary to modify anything in Neutron to pass the (--gateway-port)
  because OVN finds the gateway port automatically.

  At the moment the only issue found in Neutron is related to DB sync,
  and it is natural because the LRP connected to the TS does not exist
  in the DB. If the operator needs to restore the Neutron DB, the SYNC
  repair command will remove the unmanaged externally added resources.

  Note: The route learning has been tested with IPv4 and IPv6 addresses
  and is working fine. A detail in the case of IPv6 is related to the
  filter of routes learned via LLC addresses [3], take care of this
  case.

  SYNC_REPAIR - problem

  * Static Routes (learned OVN-IC routes)
  * Router Port -> Transit Switches

  Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port found in OVN but not in Neutron, port_id=rt2-admin-tenant1
  Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes [{'destination': '10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': '10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but not in Neutron

  -------------------------------------

  
  Proposed solution:
  --------------------------------------

  This RFE intends to implement a filter in the OVN mech_driver to
  validate the external_ids key and not remove LRP's and static routes
  present in the OVN backend without Neutron "key" in external_ids
  register.

  sync_routers_and_rports method:

  LRPs case:

  Filter the port list when iterating over existing OVN LRPs before
  checking for existence in the Neutron DB.

  ovn-nbctl list logical_router_port rt1-admin-tenant1 
  _uuid               : 6a8bbf7b-4bf6-46a8-b0be-631154b87446
  enabled             : []
  external_ids        : {}
  gateway_chassis     : [b4487769-273e-4ba9-abd5-4743ff987f74]
  ha_chassis_group    : []
  ipv6_prefix         : []
  ipv6_ra_configs     : {}
  mac                 : "aa:aa:aa:aa:ab:01"
  name                : rt1-admin-tenant1
  networks            : ["169.254.100.11/24", "fd00::1/64"]
  options             : {}
  peer                : []

  
  Static routes case:

  Additionally, OVN sync_db composes a list of static routes linked to a
  router (get_all_logical_routers_with_rports). The proposal is to
  extend the Neutron key filter in the external ids when creating the
  return list. Similar to the router port case.

  ovn-nbctl lr-route-list 078fd69b-f4c7-4469-a900-918d0a229bd1
  IPv4 Routes
  Route Table <main>:
                10.0.1.0/24            169.254.100.12 dst-ip (learned)
                10.0.2.0/24            169.254.100.13 dst-ip (learned)
                  0.0.0.0/0             200.200.200.1 dst-ip

  IPv6 Routes
  Route Table <main>:
            2001:db8:1::/64                   fd00::2 dst-ip (learned)
            2001:db8:1::/64                   fd00::3 dst-ip (learned)
      2801:80:3ea0:822::/64                   fd00::2 dst-ip (learned)
      2801:80:3ea0:823::/64                   fd00::3 dst-ip (learned)
                       ::/0              2001:db8:1:: dst-ip

  With these specific changes, the management of these LRPs and learned
  routes will be completely disassociated from Neutron perspective, and
  the resources can be managed by the operator, creating and removing
  links with remote routers on demand.

  --------------------------------------

  
  Regards,
  Roberto

  
  [1] https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.html
  [2] https://github.com/kubeovn/kube-ovn/blob/v1.11.0/docs/cluster-interconnection.md

  [3] https://github.com/ovn-
  org/ovn/commit/cb0e2b3f44daeafb2f02f07289e3c410ee6ead28


  
  ------------------------------------------------------------------

  Additional logs - Interconnect use case:

  OpenStack 1

  root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#
  root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#
  root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498
  IPv4 Routes
  Route Table <main>:
                20.0.1.0/24             169.254.200.2 dst-ip (learned)
                20.0.2.0/24             169.254.200.3 dst-ip (learned)
                  0.0.0.0/0             200.200.200.1 dst-ip

  IPv6 Routes
  Route Table <main>:
                       ::/0     fc00:ca5a:ca5a:8000:: dst-ip
  root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07
  IPv4 Routes
  Route Table <main>:
                10.0.1.0/24             169.254.100.2 dst-ip (learned)
                10.0.2.0/24             169.254.100.3 dst-ip (learned)
                  0.0.0.0/0             200.200.200.1 dst-ip

  IPv6 Routes
  Route Table <main>:
                       ::/0     fc00:ca5a:ca5a:8000:: dst-ip
  root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#

  
  OpenStack 2

  root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc
  IPv4 Routes
  Route Table <main>:
                20.0.0.0/24             169.254.200.1 dst-ip (learned)
                20.0.2.0/24             169.254.200.3 dst-ip (learned)
                  0.0.0.0/0             200.200.200.1 dst-ip

  IPv6 Routes
  Route Table <main>:
                       ::/0     fc00:ca5a:ca5a:8000:: dst-ip
  root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list ce45f681-6454-43fe-974f-81344bb8113a
  IPv4 Routes
  Route Table <main>:
                10.0.0.0/24             169.254.100.1 dst-ip (learned)
                10.0.2.0/24             169.254.100.3 dst-ip (learned)
                  0.0.0.0/0             200.200.200.1 dst-ip

  IPv6 Routes
  Route Table <main>:
                       ::/0     fc00:ca5a:ca5a:8000:: dst-ip


  OpenStack 3

  root@os-infra-1-neutron-ovn-northd-container-f237db97:~#
  root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list  cfa259d6-311f-4409-bcf2-79a929835cb3
  IPv4 Routes
  Route Table <main>:
                20.0.0.0/24             169.254.200.1 dst-ip (learned)
                20.0.1.0/24             169.254.200.2 dst-ip (learned)
                  0.0.0.0/0             200.200.200.1 dst-ip

  IPv6 Routes
  Route Table <main>:
                       ::/0     fc00:ca5a:ca5a:8000:: dst-ip
  root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list  c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0
  IPv4 Routes
  Route Table <main>:
                10.0.0.0/24             169.254.100.1 dst-ip (learned)
                10.0.1.0/24             169.254.100.2 dst-ip (learned)
                  0.0.0.0/0             200.200.200.1 dst-ip

  IPv6 Routes
  Route Table <main>:
                       ::/0     fc00:ca5a:ca5a:8000:: dst-ip

  
  OVN-IC Global database

  root@ovn-global-db1:~# ovn-ic-sbctl show
  availability-zone osp1
      gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5
          hostname: osp1-gwnode1
          type: geneve
              ip: 192.168.200.28
          port admin-rt1-tenant1
              transit switch: admin-tenant1
              address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 fe80::1/64"]
          port admin-rt1-tenant1_1
              transit switch: admin-tenant1_1
              address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"]
  availability-zone osp2
      gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8
          hostname: osp2-gwnode1
          type: geneve
              ip: 192.168.200.128
          port admin-rt2-tenant1
              transit switch: admin-tenant1
              address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 fe80::2/64"]
          port admin-rt2-tenant1_1
              transit switch: admin-tenant1_1
              address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"]
  availability-zone osp3
      gateway 97595af9-7896-40d0-a883-beadbff1aa5b
          hostname: osp3-gwnode1
          type: geneve
              ip: 192.168.200.228
          port admin-rt3-tenant1
              transit switch: admin-tenant1
              address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 fe80::3/64"]
          port admin-rt3-tenant1_1
              transit switch: admin-tenant1_1
              address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"]
              
  --------------------------------------

  
  Reference design:

  # Global database OVN-IC

  ovn-ic-nbctl ts-add admin-tenant1

  
  **** OpenStack 1 ***********

  # OVN central 1

  ovn-nbctl set NB_Global . name=osp1

  ovn-nbctl set NB_Global . options:ic-route-adv=true \
                              options:ic-route-learn=true
                              
                              
  ovn-nbctl lrp-add NEUTRON_ROUTER rt1-admin-tenant1 aa:aa:aa:aa:aa:01 169.254.100.1/24
  ovn-nbctl lsp-add admin-tenant1 admin-rt1-tenant1  -- \
          lsp-set-addresses admin-rt1-tenant1 router -- \
          lsp-set-type admin-rt1-tenant1 router -- \
          lsp-set-options admin-rt1-tenant1 router-port=rt1-admin-tenant1

  ovn-nbctl lrp-set-gateway-chassis rt1-admin-tenant1
  832b6c0d-13ce-4600-ab37-78516d8ec4c5 1

  ovn-nbctl set NB_Global . options:ic-route-blacklist=200.200.200.0/24

  # Gateway node - Openstack 1

  ovs-vsctl set open_vswitch . external_ids:ovn-is-interconn=true

       
  **** OpenStack 2 ***********   
       
  ovn-nbctl set NB_Global . name=osp2

  ovn-nbctl set NB_Global . options:ic-route-adv=true \
                              options:ic-route-learn=true

  
  ovn-nbctl lrp-add NEUTRON_ROUTER  rt2-admin-tenant1 aa:aa:aa:aa:aa:02 169.254.100.2/24
  ovn-nbctl lsp-add admin-tenant1 admin-rt2-tenant1  -- \
          lsp-set-addresses admin-rt2-tenant1 router -- \
          lsp-set-type admin-rt2-tenant1 router -- \
          lsp-set-options admin-rt2-tenant1 router-port=rt2-admin-tenant1

  ovn-nbctl lrp-set-gateway-chassis rt2-admin-tenant1 17ffabdf-
  cf47-41ab-9539-d326c13c4ca8 1

  ovn-nbctl set NB_Global . options:ic-route-blacklist=200.200.200.0/24

  # Gateway node

  ovs-vsctl set open_vswitch . external_ids:ovn-is-interconn=true

  **** OpenStack 3 ***********

  ovn-nbctl set NB_Global . name=osp3

  ovn-nbctl set NB_Global . options:ic-route-adv=true \
                              options:ic-route-learn=true

  ovn-nbctl lrp-add NEUTRON_ROUTER  rt3-admin-tenant1 aa:aa:aa:aa:aa:03 169.254.100.3/24
  ovn-nbctl lsp-add admin-tenant1 admin-rt3-tenant1  -- \
          lsp-set-addresses admin-rt3-tenant1 router -- \
          lsp-set-type admin-rt3-tenant1 router -- \
          lsp-set-options admin-rt3-tenant1 router-port=rt3-admin-tenant1

  ovn-nbctl lrp-set-gateway-chassis rt3-admin-tenant1
  97595af9-7896-40d0-a883-beadbff1aa5b 1

  ovn-nbctl set NB_Global . options:ic-route-blacklist=200.200.200.0/24
          
  # Gateway node

  ovs-vsctl set open_vswitch . external_ids:ovn-is-interconn=true

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2027742/+subscriptions