yahoo-eng-team team mailing list archive

[Alexandre Perreault <1913646@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Hi,
I am running openstack ussuri with ovs and DVR routers.

When there are multiple subnets in one network, Neutron does not
consider the possibility that the subnets could be connected to
different routers. This is a problem when a DVR router is expecting to
receive an ARP reply. In OVS br-int, table 3 contains only one rule per
network which applies to traffic destined to the DVR MAC. This rule
translates the DVR MAC to the MAC of the newest router on the network
but does not take into consideration that a network could have multiple
subnets connected to different routers.

The use case where I am facing this issue is with manila. Manila defines
one network object in the service project but each time a user creates a
new "Share Network", the Manila service creates a new subnet within the
network. So you can end up with many subnets and routers within a
network.

It is a bit confusing so below are more details taking my use case with manila as an example.
Manila has a network called manila_service_network
In manila.conf a CIDR and mask is configured and the subnets that will be created by manila service are configured within the CIDR defined and using the mask that is defined.

On a user project I create NetworkX/SubnetX (172.20.20.0/24) and connect it to routerX. I also have instanceX on this network.
Then I create a Share network. This creates a subnet within network manila_service_network with IP 10.128.16.0/20. 
Manila_subnet1 (10.128.16.0/20) is connected to routerX which is already connected to SubnetX (172.20.20.0/24). 
A ShareInstance is created on the manila_subnet1 and has IP 10.128.19.189.
It is important that the ShareInstance and InstanceX be located on different computes.

Now communication between InstanceX and the ShareInstance should work
but it does not. Here's why.

InstanceX wants to communicate to the ShareInstance so it sends a packet to it's gateway RouterX. 
RouterX needs to route the packet to the ShareInstance but it does not have the MAC address in its ARP table.
Router X sends an ARP request -> ARP, Request who-has 10.128.19.189 tell 10.128.16.1, length 28
RouterX never receives an ARP reply.
I followed the flows in br-int and br-tun.

Since traffic is coming from a DVR router, OVS br-tun changes the router's source MAC to the computes's DVR MAC. fa:16:3e:80:4c:3a is the MAC of the router with IP 10.128.16.1
 cookie=0x7027c9402a453a34, duration=411942.542s, table=1, n_packets=434, n_bytes=33812, idle_age=589, hard_age=65534, priority=1,dl_vlan=31,dl_src=fa:16:3e:80:4c:3a actions=mod_dl_src:fa:16:3f:67:83:30,resubmit(,2) 

Then the packet reaches the ARP responder table 21. There is an entry in table 21 for the ShareInstance MAC so it modifies the packet and sends it back to br-int.
cookie=0x7027c9402a453a34, duration=11769.612s, table=21, n_packets=23, n_bytes=966, idle_age=2, priority=1,arp,dl_vlan=31,arp_tpa=10.128.19.189 actions=load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163ee3273f->NXM_NX_ARP_SHA[],load:0xa8013bd->NXM_OF_ARP_SPA[],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:e3:27:3f,IN_PORT

But remember that the router source MAC became the DVR MAC and, because
of table 21, it is now the destination MAC.

This br-int table 1 rules sends us to table 3 because of the destination MAC being the DVR MAC.
cookie=0xe728ac45412eb352, duration=4676042.487s, table=0, n_packets=10695122, n_bytes=449195124, idle_age=0, hard_age=65534, priority=5,in_port=2,dl_dst=fa:16:3f:67:83:30 actions=resubmit(,3)

In table 3 there is a rule that changes the destination MAC from the DVR MAC to the router MAC based on the VLAN (network). In our case vlan 31.
cookie=0xe728ac45412eb352, duration=23642.517s, table=3, n_packets=10626537, n_bytes=446314554, idle_age=0, priority=5,dl_vlan=31,dl_dst=fa:16:3f:67:83:30 actions=mod_dl_dst:fa:16:3e:4d:d0:f9,strip_vlan,output:725

You can see that the mod_dl_dst MAC (fa:16:3e:4d:d0:f9) is not the original source MAC of my router (fa:16:3e:80:4c:3a).
Why?
Because there are multiple subnets in the network manila_service_network, each connected to a different router.
fa:16:3e:4d:d0:f9 belongs to a router connected to Manila_subnet2 (10.128.48.0/20) which is within manila_service_network.
This means the ARP reply is sent to the wrong router.
All the subnets in manila_service_network use vlan 31 so, by having one rule in table 3 for vlan 31, causes all traffic to be sent to one router (usually the newest).

In my use case there are 6 subnets in manila_service_network and the ARP
replies for all 6 subnets go to the same router (usually the newest).
This means 5 subnets out of 6 are broken.

You dont need to use manila to recreate the problem.
You need networkA with subnetA and network1 with subnet1, subnet2, subnet3, etc...
Connect subnetA and subnet1 to the same router and create a couple instances on subnetA and subnet1 (they need to be on different computes).
Then connect subnet2 to a new router and subnet3 to another new router.
You should see that the instances on subnetA and subnet1 wont be able to communicate as the traffic will stop on the router.

Table 3 is a fairly new addition to neutron/ovs so I believe that the possibility of having multiple subnets in one network was not considered when writing the code.
I think it is related to this commit
https://review.opendev.org/c/openstack/neutron/+/651905/12/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/br_int.py

Though it is rare to have multiple subnets in one network, the manila
service uses this architecture, making it important that it works
correctly.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1913646

Title:
  DVR router ARP traffic broken for networks containing multiple subnets

Status in neutron:
  New

Bug description:
  Hi,
  I am running openstack ussuri with ovs and DVR routers.

  When there are multiple subnets in one network, Neutron does not
  consider the possibility that the subnets could be connected to
  different routers. This is a problem when a DVR router is expecting to
  receive an ARP reply. In OVS br-int, table 3 contains only one rule
  per network which applies to traffic destined to the DVR MAC. This
  rule translates the DVR MAC to the MAC of the newest router on the
  network but does not take into consideration that a network could have
  multiple subnets connected to different routers.

  The use case where I am facing this issue is with manila. Manila
  defines one network object in the service project but each time a user
  creates a new "Share Network", the Manila service creates a new subnet
  within the network. So you can end up with many subnets and routers
  within a network.

  It is a bit confusing so below are more details taking my use case with manila as an example.
  Manila has a network called manila_service_network
  In manila.conf a CIDR and mask is configured and the subnets that will be created by manila service are configured within the CIDR defined and using the mask that is defined.

  On a user project I create NetworkX/SubnetX (172.20.20.0/24) and connect it to routerX. I also have instanceX on this network.
  Then I create a Share network. This creates a subnet within network manila_service_network with IP 10.128.16.0/20. 
  Manila_subnet1 (10.128.16.0/20) is connected to routerX which is already connected to SubnetX (172.20.20.0/24). 
  A ShareInstance is created on the manila_subnet1 and has IP 10.128.19.189.
  It is important that the ShareInstance and InstanceX be located on different computes.

  Now communication between InstanceX and the ShareInstance should work
  but it does not. Here's why.

  InstanceX wants to communicate to the ShareInstance so it sends a packet to it's gateway RouterX. 
  RouterX needs to route the packet to the ShareInstance but it does not have the MAC address in its ARP table.
  Router X sends an ARP request -> ARP, Request who-has 10.128.19.189 tell 10.128.16.1, length 28
  RouterX never receives an ARP reply.
  I followed the flows in br-int and br-tun.

  Since traffic is coming from a DVR router, OVS br-tun changes the router's source MAC to the computes's DVR MAC. fa:16:3e:80:4c:3a is the MAC of the router with IP 10.128.16.1
   cookie=0x7027c9402a453a34, duration=411942.542s, table=1, n_packets=434, n_bytes=33812, idle_age=589, hard_age=65534, priority=1,dl_vlan=31,dl_src=fa:16:3e:80:4c:3a actions=mod_dl_src:fa:16:3f:67:83:30,resubmit(,2) 

  Then the packet reaches the ARP responder table 21. There is an entry in table 21 for the ShareInstance MAC so it modifies the packet and sends it back to br-int.
  cookie=0x7027c9402a453a34, duration=11769.612s, table=21, n_packets=23, n_bytes=966, idle_age=2, priority=1,arp,dl_vlan=31,arp_tpa=10.128.19.189 actions=load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163ee3273f->NXM_NX_ARP_SHA[],load:0xa8013bd->NXM_OF_ARP_SPA[],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:e3:27:3f,IN_PORT

  But remember that the router source MAC became the DVR MAC and,
  because of table 21, it is now the destination MAC.

  This br-int table 1 rules sends us to table 3 because of the destination MAC being the DVR MAC.
  cookie=0xe728ac45412eb352, duration=4676042.487s, table=0, n_packets=10695122, n_bytes=449195124, idle_age=0, hard_age=65534, priority=5,in_port=2,dl_dst=fa:16:3f:67:83:30 actions=resubmit(,3)

  In table 3 there is a rule that changes the destination MAC from the DVR MAC to the router MAC based on the VLAN (network). In our case vlan 31.
  cookie=0xe728ac45412eb352, duration=23642.517s, table=3, n_packets=10626537, n_bytes=446314554, idle_age=0, priority=5,dl_vlan=31,dl_dst=fa:16:3f:67:83:30 actions=mod_dl_dst:fa:16:3e:4d:d0:f9,strip_vlan,output:725

  You can see that the mod_dl_dst MAC (fa:16:3e:4d:d0:f9) is not the original source MAC of my router (fa:16:3e:80:4c:3a).
  Why?
  Because there are multiple subnets in the network manila_service_network, each connected to a different router.
  fa:16:3e:4d:d0:f9 belongs to a router connected to Manila_subnet2 (10.128.48.0/20) which is within manila_service_network.
  This means the ARP reply is sent to the wrong router.
  All the subnets in manila_service_network use vlan 31 so, by having one rule in table 3 for vlan 31, causes all traffic to be sent to one router (usually the newest).

  In my use case there are 6 subnets in manila_service_network and the
  ARP replies for all 6 subnets go to the same router (usually the
  newest). This means 5 subnets out of 6 are broken.

  You dont need to use manila to recreate the problem.
  You need networkA with subnetA and network1 with subnet1, subnet2, subnet3, etc...
  Connect subnetA and subnet1 to the same router and create a couple instances on subnetA and subnet1 (they need to be on different computes).
  Then connect subnet2 to a new router and subnet3 to another new router.
  You should see that the instances on subnetA and subnet1 wont be able to communicate as the traffic will stop on the router.

  Table 3 is a fairly new addition to neutron/ovs so I believe that the possibility of having multiple subnets in one network was not considered when writing the code.
  I think it is related to this commit
  https://review.opendev.org/c/openstack/neutron/+/651905/12/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/br_int.py

  Though it is rare to have multiple subnets in one network, the manila
  service uses this architecture, making it important that it works
  correctly.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1913646/+subscriptions