yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1723472] [NEW] [DVR] Lowering the MTU breaks FIP traffic

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Daniel Alvarez <1723472@xxxxxxxxxxxxxxxxxx>
Date: Fri, 13 Oct 2017 16:16:20 -0000
Reply-to: Bug 1723472 <1723472@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

In a DVR environment, when lowering the MTU of a network, traffic going
to an instance through a floating IP is broken.

Description:

* Ping traffic to a VM through its FIP works.
* Change the MTU of its network through "neutron net-update <network> --mtu 1440".
* Ping to the same FIP doesn't work.

After a long debugging session with Anil Venkata, we've found that
packets reach br-ex and then they hit this OF rule with normal action:

 cookie=0x1f847e4bf0de0aea, duration=70306.532s, table=3,
n_packets=1579251, n_bytes=796614220, idle_age=0, hard_age=65534,
priority=1 actions=NORMAL


We would expect this rule to switch the packet to br-int so that it can be forwarded to the fip namespace (ie. with dst MAC address set to the floating ip gw (owner=network:floatingip_agent_gateway):

$ sudo ovs-vsctl list interface

_uuid               : 1f2b6e86-d303-42f4-9467-5dab78fc7199
admin_state         : down
bfd                 : {}
bfd_status          : {}
cfm_fault           : []
cfm_fault_status    : []
cfm_flap_count      : []
cfm_health          : []
cfm_mpid            : []
cfm_remote_mpids    : []
cfm_remote_opstate  : []
duplex              : []
error               : []
external_ids        : {attached-mac="fa:16:3e:9d:0c:4f", iface-id="8ec34826-b1a6-48ce-9c39-2fd3e8167eb4", iface-status=active}
name                : "fg-8ec34826-b1"


[heat-admin@overcloud-novacompute-0 ~]$ sudo ovs-appctl fdb/show br-ex                                                                                                                        
 port  VLAN  MAC                Age
 [...]
    7    10  fa:16:3e:9d:0c:4f    0


$ sudo ovs-ofctl show br-ex | grep "7("
 7(phy-br-ex): addr:36:63:93:fc:af:e2


And from there, to the fip namespace which would route the packet to the qrouter namespace, etc.

However, when we change the MTU through the following command:

"neutron net-update <network> --mtu 1440"

We see that, after a few seconds, the MAC address of the FIP changes so
when traffic arrives br-ex and NORMAL action is performed, it will not
be output to br-int through the patch-port but instead, through eth1 and
traffic won't work anymore.

[heat-admin@overcloud-novacompute-0 ~]$ arp -n | grep ".113"
10.0.0.113               ether   fa:16:3e:9d:0c:4f   C                     vlan10

neutron port-set xxxxx --mtu 1440

$ arp -n | grep ".113"
10.0.0.113               ether   fa:16:3e:20:f9:85   C                     vlan10


When setting the MAC address manually, ping starts working again:

$ arp -s 10.0.0.113 fa:16:3e:9d:0c:4f
$ ping 10.0.0.113
PING 10.0.0.113 (10.0.0.113) 56(84) bytes of data.
64 bytes from 10.0.0.113: icmp_seq=1 ttl=62 time=1.17 ms
64 bytes from 10.0.0.113: icmp_seq=2 ttl=62 time=0.561 ms


Additional notes:

When we set the MAC address manually and traffic gets working back
again, lowering the MTU doesn't change the MAC address (we can't see any
gARP's coming through).

When we delete the ARP entry for the FIP and try to ping the FIP, the
wrong MAC address is set.

[heat-admin@overcloud-novacompute-0 ~]$ sudo arp -d 10.0.0.113

[heat-admin@overcloud-novacompute-0 ~]$ ping 10.0.0.113 -c 2
PING 10.0.0.113 (10.0.0.113) 56(84) bytes of data.

--- 10.0.0.113 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 999ms

[heat-admin@overcloud-novacompute-0 ~]$ arp -n | grep ".113"
10.0.0.113               ether   fa:16:3e:20:f9:85   C                     vlan10

** Affects: neutron
     Importance: Undecided
     Assignee: Daniel Alvarez (dalvarezs)
         Status: New


** Tags: l3-dvr-backlog

** Changed in: neutron
     Assignee: (unassigned) => Daniel Alvarez (dalvarezs)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1723472

Title:
  [DVR] Lowering the MTU breaks FIP traffic

Status in neutron:
  New

Bug description:
  In a DVR environment, when lowering the MTU of a network, traffic
  going to an instance through a floating IP is broken.

  Description:

  * Ping traffic to a VM through its FIP works.
  * Change the MTU of its network through "neutron net-update <network> --mtu 1440".
  * Ping to the same FIP doesn't work.

  After a long debugging session with Anil Venkata, we've found that
  packets reach br-ex and then they hit this OF rule with normal action:

   cookie=0x1f847e4bf0de0aea, duration=70306.532s, table=3,
  n_packets=1579251, n_bytes=796614220, idle_age=0, hard_age=65534,
  priority=1 actions=NORMAL

  
  We would expect this rule to switch the packet to br-int so that it can be forwarded to the fip namespace (ie. with dst MAC address set to the floating ip gw (owner=network:floatingip_agent_gateway):

  $ sudo ovs-vsctl list interface

  _uuid               : 1f2b6e86-d303-42f4-9467-5dab78fc7199
  admin_state         : down
  bfd                 : {}
  bfd_status          : {}
  cfm_fault           : []
  cfm_fault_status    : []
  cfm_flap_count      : []
  cfm_health          : []
  cfm_mpid            : []
  cfm_remote_mpids    : []
  cfm_remote_opstate  : []
  duplex              : []
  error               : []
  external_ids        : {attached-mac="fa:16:3e:9d:0c:4f", iface-id="8ec34826-b1a6-48ce-9c39-2fd3e8167eb4", iface-status=active}
  name                : "fg-8ec34826-b1"


  [heat-admin@overcloud-novacompute-0 ~]$ sudo ovs-appctl fdb/show br-ex                                                                                                                        
   port  VLAN  MAC                Age
   [...]
      7    10  fa:16:3e:9d:0c:4f    0

  
  $ sudo ovs-ofctl show br-ex | grep "7("
   7(phy-br-ex): addr:36:63:93:fc:af:e2

  
  And from there, to the fip namespace which would route the packet to the qrouter namespace, etc.

  However, when we change the MTU through the following command:

  "neutron net-update <network> --mtu 1440"

  We see that, after a few seconds, the MAC address of the FIP changes
  so when traffic arrives br-ex and NORMAL action is performed, it will
  not be output to br-int through the patch-port but instead, through
  eth1 and traffic won't work anymore.

  [heat-admin@overcloud-novacompute-0 ~]$ arp -n | grep ".113"
  10.0.0.113               ether   fa:16:3e:9d:0c:4f   C                     vlan10

  neutron port-set xxxxx --mtu 1440

  $ arp -n | grep ".113"
  10.0.0.113               ether   fa:16:3e:20:f9:85   C                     vlan10

  
  When setting the MAC address manually, ping starts working again:

  $ arp -s 10.0.0.113 fa:16:3e:9d:0c:4f
  $ ping 10.0.0.113
  PING 10.0.0.113 (10.0.0.113) 56(84) bytes of data.
  64 bytes from 10.0.0.113: icmp_seq=1 ttl=62 time=1.17 ms
  64 bytes from 10.0.0.113: icmp_seq=2 ttl=62 time=0.561 ms

  
  Additional notes:

  When we set the MAC address manually and traffic gets working back
  again, lowering the MTU doesn't change the MAC address (we can't see
  any gARP's coming through).

  When we delete the ARP entry for the FIP and try to ping the FIP, the
  wrong MAC address is set.

  [heat-admin@overcloud-novacompute-0 ~]$ sudo arp -d 10.0.0.113

  [heat-admin@overcloud-novacompute-0 ~]$ ping 10.0.0.113 -c 2
  PING 10.0.0.113 (10.0.0.113) 56(84) bytes of data.

  --- 10.0.0.113 ping statistics ---
  2 packets transmitted, 0 received, 100% packet loss, time 999ms

  [heat-admin@overcloud-novacompute-0 ~]$ arp -n | grep ".113"
  10.0.0.113               ether   fa:16:3e:20:f9:85   C                     vlan10

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1723472/+subscriptions
Follow ups

[Bug 1723472] Re: [DVR] Lowering the MTU breaks FIP traffic
From: Daniel Alvarez, 2017-10-18