← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 2035281] [NEW] [ML2/OVN] DGP/Floating IP issue - no flows for chassis gateway port

 

Public bug reported:

Hello everyone.

I noticed a problem with DGP feature when configured by OpenStack
Neutron using multiple external (provider) subnets.

For example, the OpenStack external provider network has multiple
subnets, such as:

subnet1: 172.16.10.0/24
subnet2: 172.16.20.0/24

When the Logical Router attaches the external gateway port to this
network, only one subnet is configured (static or dinamically), e.g. IP
address = 172.16.10.1/24.

If the Floating IP assigned for some VM uses the same subnet range as
the router's IP network, the dnat_and_snat rule will be created
correctly and inbound/outbound traffic will work. However, when the
Floating IP uses the other one subnet (not on the same network of the
external router port), the dnat_and_snat is not created and we can see
the warning message in the log as below:

2023-09-08T13:29:40.721Z|00202|northd|WARN|Unable to determine
gateway_port for NAT with external_ip: 172.16.20.157 configured on
logical router: neutron-477cf920-21e3-46e5-8c8f-7b8caef7f549 with
multiple distributed gateway ports

This problem occurs because Neutron has not configured the "gateway-
port" param in the OVN NAT rule. In this case, the northd [1]
automatically obtains the gateway port using the external IP from the
NAT rule and the external network configured on the OVN logical router.
This issue was introduced with these commits [1][2] and affect ML2/OVN
backend since OVN version 21.09.

This problem was discussed on the ovs-discuss mailing list [3], but
technically it seems to me that it is a required change in the CMS to
guarantee the creation of FIP flows without having to rely on OVN to
automatically discover the gateway port.


If OVN is using Distributed Gateway Port on the router, the FIP created by Neutron will not work due to lack of openflow flows to the gateway port:

Before set the gateway_port:

ovn-nbctl lr-nat-list 078fd69b-f4c7-4469-a900-918d0a229bd1 
TYPE             GATEWAY_PORT          EXTERNAL_IP        EXTERNAL_PORT    LOGICAL_IP          EXTERNAL_MAC         LOGICAL_PORT
dnat_and_snat                          172.16.20.10                        10.0.0.232
snat                                   172.16.10.41                        10.0.0.0/24

ovn-sbctl lflow-list | grep 172.16.20.10
  table=25(ls_in_l2_lkup      ), priority=80   , match=(flags[1] == 0 && arp.op == 1 && arp.tpa == 172.16.20.10), action=(clone {outport = "admin-rt1-tenant1"; output; }; outport = "_MC_flood_l2"; output;)
  table=25(ls_in_l2_lkup      ), priority=80   , match=(flags[1] == 0 && arp.op == 1 && arp.tpa == 172.16.20.10), action=(clone {outport = "1cda494c-4e86-4941-9680-b949341b12a5"; output; }; outport = "_MC_flood_l2"; output;)
  table=25(ls_in_l2_lkup      ), priority=80   , match=(flags[1] == 0 && arp.op == 1 && arp.tpa == 172.16.20.10), action=(clone {outport = "bdf0ad70-8677-4340-b5ec-f26af6575e5e"; output; }; outport = "_MC_flood_l2"; output;)
  table=25(ls_in_l2_lkup      ), priority=80   , match=(flags[1] == 0 && arp.op == 1 && arp.tpa == 172.16.20.10), action=(clone {outport = "e77e522c-5170-4566-a7b5-1b6ef9f88000"; output; }; outport = "_MC_flood_l2"; output;)
  table=3 (lr_in_ip_input     ), priority=90   , match=(arp.op == 1 && arp.tpa == 172.16.20.10), action=(eth.dst = eth.src; eth.src = xreg0[0..47]; arp.op = 2; /* ARP reply */ arp.tha = arp.sha; arp.sha = xreg0[0..47]; arp.tpa <-> arp.spa; outport = inport; flags.loopback = 1; output;)
  table=25(ls_in_l2_lkup      ), priority=80   , match=(flags[1] == 0 && arp.op == 1 && arp.tpa == 172.16.20.10), action=(clone {outport = "bb89ed8d-a60d-4a9d-8210-205770490180"; output; }; outport = "_MC_flood_l2"; output;)


After set the gateway_port:
ovn-nbctl lr-nat-list 078fd69b-f4c7-4469-a900-918d0a229bd1 
TYPE             GATEWAY_PORT          EXTERNAL_IP        EXTERNAL_PORT    LOGICAL_IP          EXTERNAL_MAC         LOGICAL_PORT
dnat_and_snat    lrp-bb89ed8d-a60d-    172.16.20.10                        10.0.0.232
snat                                   172.16.10.41                        10.0.0.0/24

ovn-sbctl lflow-list | grep 172.16.20.10
  table=25(ls_in_l2_lkup      ), priority=80   , match=(flags[1] == 0 && arp.op == 1 && arp.tpa == 172.16.20.10), action=(clone {outport = "admin-rt1-tenant1"; output; }; outport = "_MC_flood_l2"; output;)
  table=25(ls_in_l2_lkup      ), priority=80   , match=(flags[1] == 0 && arp.op == 1 && arp.tpa == 172.16.20.10), action=(clone {outport = "1cda494c-4e86-4941-9680-b949341b12a5"; output; }; outport = "_MC_flood_l2"; output;)
  table=25(ls_in_l2_lkup      ), priority=80   , match=(flags[1] == 0 && arp.op == 1 && arp.tpa == 172.16.20.10), action=(clone {outport = "bdf0ad70-8677-4340-b5ec-f26af6575e5e"; output; }; outport = "_MC_flood_l2"; output;)
  table=25(ls_in_l2_lkup      ), priority=80   , match=(flags[1] == 0 && arp.op == 1 && arp.tpa == 172.16.20.10), action=(clone {outport = "e77e522c-5170-4566-a7b5-1b6ef9f88000"; output; }; outport = "_MC_flood_l2"; output;)
  table=3 (lr_in_ip_input     ), priority=92   , match=(inport == "lrp-bb89ed8d-a60d-4a9d-8210-205770490180" && arp.op == 1 && arp.tpa == 172.16.20.10 && is_chassis_resident("cr-lrp-bb89ed8d-a60d-4a9d-8210-205770490180")), action=(eth.dst = eth.src; eth.src = xreg0[0..47]; arp.op = 2; /* ARP reply */ arp.tha = arp.sha; arp.sha = xreg0[0..47]; arp.tpa <-> arp.spa; outport = inport; flags.loopback = 1; output;)
  table=3 (lr_in_ip_input     ), priority=91   , match=(inport == "lrp-bb89ed8d-a60d-4a9d-8210-205770490180" && arp.op == 1 && arp.tpa == 172.16.20.10), action=(drop;)
  table=3 (lr_in_ip_input     ), priority=90   , match=(arp.op == 1 && arp.tpa == 172.16.20.10), action=(eth.dst = eth.src; eth.src = xreg0[0..47]; arp.op = 2; /* ARP reply */ arp.tha = arp.sha; arp.sha = xreg0[0..47]; arp.tpa <-> arp.spa; outport = inport; flags.loopback = 1; output;)
  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.16.20.10 && inport == "lrp-bb89ed8d-a60d-4a9d-8210-205770490180" && flags.loopback == 0 && is_chassis_resident("cr-lrp-bb89ed8d-a60d-4a9d-8210-205770490180")), action=(ct_snat_in_czone;)
  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.16.20.10 && inport == "lrp-bb89ed8d-a60d-4a9d-8210-205770490180" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lrp-bb89ed8d-a60d-4a9d-8210-205770490180")), action=(ct_snat;)
  table=7 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.16.20.10 && inport == "lrp-bb89ed8d-a60d-4a9d-8210-205770490180" && is_chassis_resident("cr-lrp-bb89ed8d-a60d-4a9d-8210-205770490180")), action=(ct_dnat_in_czone(10.0.0.232);)
  table=17(lr_in_arp_resolve  ), priority=150  , match=(inport == "lrp-bb89ed8d-a60d-4a9d-8210-205770490180" && outport == "lrp-bb89ed8d-a60d-4a9d-8210-205770490180" && ip4.dst == 172.16.20.10), action=(drop;)
  table=17(lr_in_arp_resolve  ), priority=100  , match=(outport == "lrp-bb89ed8d-a60d-4a9d-8210-205770490180" && reg0 == 172.16.20.10), action=(eth.dst = fa:16:3e:dc:fb:47; next;)
  table=0 (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.16.20.10 && is_chassis_resident("cr-lrp-bb89ed8d-a60d-4a9d-8210-205770490180")), action=(reg9[4] = 1; next;)
  table=3 (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.232 && outport == "lrp-bb89ed8d-a60d-4a9d-8210-205770490180" && is_chassis_resident("cr-lrp-bb89ed8d-a60d-4a9d-8210-205770490180") && reg9[4] == 1), action=(reg9[4] = 0; ct_snat(172.16.20.10);)
  table=3 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.232 && outport == "lrp-bb89ed8d-a60d-4a9d-8210-205770490180" && is_chassis_resident("cr-lrp-bb89ed8d-a60d-4a9d-8210-205770490180")), action=(ct_snat_in_czone(172.16.20.10);)
  table=5 (lr_out_egr_loop    ), priority=100  , match=(ip4.dst == 172.16.20.10 && outport == "lrp-bb89ed8d-a60d-4a9d-8210-205770490180" && is_chassis_resident("cr-lrp-bb89ed8d-a60d-4a9d-8210-205770490180")), action=(clone { ct_clear; inport = outport; outport = ""; eth.dst <-> eth.src; flags = 0; flags.loopback = 1; flags.use_snat_zone = reg9[4]; reg0 = 0; reg1 = 0; reg2 = 0; reg3 = 0; reg4 = 0; reg5 = 0; reg6 = 0; reg7 = 0; reg8 = 0; reg9 = 0; reg9[0] = 1; next(pipeline=ingress, table=0); };)
  table=25(ls_in_l2_lkup      ), priority=80   , match=(flags[1] == 0 && arp.op == 1 && arp.tpa == 172.16.20.10), action=(clone {outport = "bb89ed8d-a60d-4a9d-8210-205770490180"; output; }; outport = "_MC_flood_l2"; output;)


Basically, after the change [2], OVN allows operation without setting the NAT rule gateway_port for a very restricted use cases and Neutron ends up being vulnerable to different OVN architectures (using DGP, for example). If Neutron configured the gateway_port parameter on NAT rule command it would work, but the issue is that Neutron never configured this before and automatically gateway port discovery may break the working of FIPs.


My suggestion is that Neutron adds this column when creating/updating FIP rules (if the OVN backend supports this column in the Northbound DB Schema), and updating the previously existing dnat_and_snat entries (FIPs) to set the gateway_port in the maintenance task (once time).


[1] https://github.com/ovn-org/ovn/commit/15348b7b806f7a9680606c3e9348708980129949
[2] https://github.com/ovn-org/ovn/commit/2d942be7db1799f2778492331513ae2b5a556b92
[3] https://mail.openvswitch.org/pipermail/ovs-discuss/2023-September/052655.html

** Affects: neutron
     Importance: Undecided
     Assignee: Roberto Bartzen Acosta (rbartzen)
         Status: New

** Changed in: neutron
     Assignee: (unassigned) => Roberto Bartzen Acosta (rbartzen)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2035281

Title:
  [ML2/OVN] DGP/Floating IP issue - no flows for chassis gateway port

Status in neutron:
  New

Bug description:
  Hello everyone.

  I noticed a problem with DGP feature when configured by OpenStack
  Neutron using multiple external (provider) subnets.

  For example, the OpenStack external provider network has multiple
  subnets, such as:

  subnet1: 172.16.10.0/24
  subnet2: 172.16.20.0/24

  When the Logical Router attaches the external gateway port to this
  network, only one subnet is configured (static or dinamically), e.g.
  IP address = 172.16.10.1/24.

  If the Floating IP assigned for some VM uses the same subnet range as
  the router's IP network, the dnat_and_snat rule will be created
  correctly and inbound/outbound traffic will work. However, when the
  Floating IP uses the other one subnet (not on the same network of the
  external router port), the dnat_and_snat is not created and we can see
  the warning message in the log as below:

  2023-09-08T13:29:40.721Z|00202|northd|WARN|Unable to determine
  gateway_port for NAT with external_ip: 172.16.20.157 configured on
  logical router: neutron-477cf920-21e3-46e5-8c8f-7b8caef7f549 with
  multiple distributed gateway ports

  This problem occurs because Neutron has not configured the "gateway-
  port" param in the OVN NAT rule. In this case, the northd [1]
  automatically obtains the gateway port using the external IP from the
  NAT rule and the external network configured on the OVN logical
  router. This issue was introduced with these commits [1][2] and affect
  ML2/OVN backend since OVN version 21.09.

  This problem was discussed on the ovs-discuss mailing list [3], but
  technically it seems to me that it is a required change in the CMS to
  guarantee the creation of FIP flows without having to rely on OVN to
  automatically discover the gateway port.

  
  If OVN is using Distributed Gateway Port on the router, the FIP created by Neutron will not work due to lack of openflow flows to the gateway port:

  Before set the gateway_port:

  ovn-nbctl lr-nat-list 078fd69b-f4c7-4469-a900-918d0a229bd1 
  TYPE             GATEWAY_PORT          EXTERNAL_IP        EXTERNAL_PORT    LOGICAL_IP          EXTERNAL_MAC         LOGICAL_PORT
  dnat_and_snat                          172.16.20.10                        10.0.0.232
  snat                                   172.16.10.41                        10.0.0.0/24

  ovn-sbctl lflow-list | grep 172.16.20.10
    table=25(ls_in_l2_lkup      ), priority=80   , match=(flags[1] == 0 && arp.op == 1 && arp.tpa == 172.16.20.10), action=(clone {outport = "admin-rt1-tenant1"; output; }; outport = "_MC_flood_l2"; output;)
    table=25(ls_in_l2_lkup      ), priority=80   , match=(flags[1] == 0 && arp.op == 1 && arp.tpa == 172.16.20.10), action=(clone {outport = "1cda494c-4e86-4941-9680-b949341b12a5"; output; }; outport = "_MC_flood_l2"; output;)
    table=25(ls_in_l2_lkup      ), priority=80   , match=(flags[1] == 0 && arp.op == 1 && arp.tpa == 172.16.20.10), action=(clone {outport = "bdf0ad70-8677-4340-b5ec-f26af6575e5e"; output; }; outport = "_MC_flood_l2"; output;)
    table=25(ls_in_l2_lkup      ), priority=80   , match=(flags[1] == 0 && arp.op == 1 && arp.tpa == 172.16.20.10), action=(clone {outport = "e77e522c-5170-4566-a7b5-1b6ef9f88000"; output; }; outport = "_MC_flood_l2"; output;)
    table=3 (lr_in_ip_input     ), priority=90   , match=(arp.op == 1 && arp.tpa == 172.16.20.10), action=(eth.dst = eth.src; eth.src = xreg0[0..47]; arp.op = 2; /* ARP reply */ arp.tha = arp.sha; arp.sha = xreg0[0..47]; arp.tpa <-> arp.spa; outport = inport; flags.loopback = 1; output;)
    table=25(ls_in_l2_lkup      ), priority=80   , match=(flags[1] == 0 && arp.op == 1 && arp.tpa == 172.16.20.10), action=(clone {outport = "bb89ed8d-a60d-4a9d-8210-205770490180"; output; }; outport = "_MC_flood_l2"; output;)

  
  After set the gateway_port:
  ovn-nbctl lr-nat-list 078fd69b-f4c7-4469-a900-918d0a229bd1 
  TYPE             GATEWAY_PORT          EXTERNAL_IP        EXTERNAL_PORT    LOGICAL_IP          EXTERNAL_MAC         LOGICAL_PORT
  dnat_and_snat    lrp-bb89ed8d-a60d-    172.16.20.10                        10.0.0.232
  snat                                   172.16.10.41                        10.0.0.0/24

  ovn-sbctl lflow-list | grep 172.16.20.10
    table=25(ls_in_l2_lkup      ), priority=80   , match=(flags[1] == 0 && arp.op == 1 && arp.tpa == 172.16.20.10), action=(clone {outport = "admin-rt1-tenant1"; output; }; outport = "_MC_flood_l2"; output;)
    table=25(ls_in_l2_lkup      ), priority=80   , match=(flags[1] == 0 && arp.op == 1 && arp.tpa == 172.16.20.10), action=(clone {outport = "1cda494c-4e86-4941-9680-b949341b12a5"; output; }; outport = "_MC_flood_l2"; output;)
    table=25(ls_in_l2_lkup      ), priority=80   , match=(flags[1] == 0 && arp.op == 1 && arp.tpa == 172.16.20.10), action=(clone {outport = "bdf0ad70-8677-4340-b5ec-f26af6575e5e"; output; }; outport = "_MC_flood_l2"; output;)
    table=25(ls_in_l2_lkup      ), priority=80   , match=(flags[1] == 0 && arp.op == 1 && arp.tpa == 172.16.20.10), action=(clone {outport = "e77e522c-5170-4566-a7b5-1b6ef9f88000"; output; }; outport = "_MC_flood_l2"; output;)
    table=3 (lr_in_ip_input     ), priority=92   , match=(inport == "lrp-bb89ed8d-a60d-4a9d-8210-205770490180" && arp.op == 1 && arp.tpa == 172.16.20.10 && is_chassis_resident("cr-lrp-bb89ed8d-a60d-4a9d-8210-205770490180")), action=(eth.dst = eth.src; eth.src = xreg0[0..47]; arp.op = 2; /* ARP reply */ arp.tha = arp.sha; arp.sha = xreg0[0..47]; arp.tpa <-> arp.spa; outport = inport; flags.loopback = 1; output;)
    table=3 (lr_in_ip_input     ), priority=91   , match=(inport == "lrp-bb89ed8d-a60d-4a9d-8210-205770490180" && arp.op == 1 && arp.tpa == 172.16.20.10), action=(drop;)
    table=3 (lr_in_ip_input     ), priority=90   , match=(arp.op == 1 && arp.tpa == 172.16.20.10), action=(eth.dst = eth.src; eth.src = xreg0[0..47]; arp.op = 2; /* ARP reply */ arp.tha = arp.sha; arp.sha = xreg0[0..47]; arp.tpa <-> arp.spa; outport = inport; flags.loopback = 1; output;)
    table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.16.20.10 && inport == "lrp-bb89ed8d-a60d-4a9d-8210-205770490180" && flags.loopback == 0 && is_chassis_resident("cr-lrp-bb89ed8d-a60d-4a9d-8210-205770490180")), action=(ct_snat_in_czone;)
    table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.16.20.10 && inport == "lrp-bb89ed8d-a60d-4a9d-8210-205770490180" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lrp-bb89ed8d-a60d-4a9d-8210-205770490180")), action=(ct_snat;)
    table=7 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.16.20.10 && inport == "lrp-bb89ed8d-a60d-4a9d-8210-205770490180" && is_chassis_resident("cr-lrp-bb89ed8d-a60d-4a9d-8210-205770490180")), action=(ct_dnat_in_czone(10.0.0.232);)
    table=17(lr_in_arp_resolve  ), priority=150  , match=(inport == "lrp-bb89ed8d-a60d-4a9d-8210-205770490180" && outport == "lrp-bb89ed8d-a60d-4a9d-8210-205770490180" && ip4.dst == 172.16.20.10), action=(drop;)
    table=17(lr_in_arp_resolve  ), priority=100  , match=(outport == "lrp-bb89ed8d-a60d-4a9d-8210-205770490180" && reg0 == 172.16.20.10), action=(eth.dst = fa:16:3e:dc:fb:47; next;)
    table=0 (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.16.20.10 && is_chassis_resident("cr-lrp-bb89ed8d-a60d-4a9d-8210-205770490180")), action=(reg9[4] = 1; next;)
    table=3 (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.232 && outport == "lrp-bb89ed8d-a60d-4a9d-8210-205770490180" && is_chassis_resident("cr-lrp-bb89ed8d-a60d-4a9d-8210-205770490180") && reg9[4] == 1), action=(reg9[4] = 0; ct_snat(172.16.20.10);)
    table=3 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.232 && outport == "lrp-bb89ed8d-a60d-4a9d-8210-205770490180" && is_chassis_resident("cr-lrp-bb89ed8d-a60d-4a9d-8210-205770490180")), action=(ct_snat_in_czone(172.16.20.10);)
    table=5 (lr_out_egr_loop    ), priority=100  , match=(ip4.dst == 172.16.20.10 && outport == "lrp-bb89ed8d-a60d-4a9d-8210-205770490180" && is_chassis_resident("cr-lrp-bb89ed8d-a60d-4a9d-8210-205770490180")), action=(clone { ct_clear; inport = outport; outport = ""; eth.dst <-> eth.src; flags = 0; flags.loopback = 1; flags.use_snat_zone = reg9[4]; reg0 = 0; reg1 = 0; reg2 = 0; reg3 = 0; reg4 = 0; reg5 = 0; reg6 = 0; reg7 = 0; reg8 = 0; reg9 = 0; reg9[0] = 1; next(pipeline=ingress, table=0); };)
    table=25(ls_in_l2_lkup      ), priority=80   , match=(flags[1] == 0 && arp.op == 1 && arp.tpa == 172.16.20.10), action=(clone {outport = "bb89ed8d-a60d-4a9d-8210-205770490180"; output; }; outport = "_MC_flood_l2"; output;)

  
  Basically, after the change [2], OVN allows operation without setting the NAT rule gateway_port for a very restricted use cases and Neutron ends up being vulnerable to different OVN architectures (using DGP, for example). If Neutron configured the gateway_port parameter on NAT rule command it would work, but the issue is that Neutron never configured this before and automatically gateway port discovery may break the working of FIPs.

  
  My suggestion is that Neutron adds this column when creating/updating FIP rules (if the OVN backend supports this column in the Northbound DB Schema), and updating the previously existing dnat_and_snat entries (FIPs) to set the gateway_port in the maintenance task (once time).

  
  [1] https://github.com/ovn-org/ovn/commit/15348b7b806f7a9680606c3e9348708980129949
  [2] https://github.com/ovn-org/ovn/commit/2d942be7db1799f2778492331513ae2b5a556b92
  [3] https://mail.openvswitch.org/pipermail/ovs-discuss/2023-September/052655.html

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2035281/+subscriptions



Follow ups