yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1931844] [NEW] Can't ping router with packet size greater than 1476 when ovs datapath_type set to netdev

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Oleksiy Butenko <1931844@xxxxxxxxxxxxxxxxxx>
Date: Mon, 14 Jun 2021 08:58:59 -0000
Reply-to: Bug 1931844 <1931844@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Setup:
  Openstack origin/stable/wallaby deployed by devstack on single node
  [stack@localhost devstack]$ cat /etc/centos-release
  CentOS Linux release 8.4.2105
  [stack@localhost devstack]$ uname -a
  Linux localhost.localdomain 4.18.0-305.3.1.el8.x86_64 #1 SMP Tue Jun 1 16:14:33 UTC 2021 x86_64 
  x86_64 x86_64 GNU/Linux
  [stack@localhost devstack]$

  [stack@localhost devstack]$ git status
  HEAD detached at origin/stable/wallaby

  ovs version (builded with DPDK)
  [stack@localhost devstack]$ ovs-vswitchd --version
  ovs-vswitchd (Open vSwitch) 2.13.4
  DPDK 19.11.7

  [stack@localhost devstack]$ cat /etc/neutron/plugins/ml2/ml2_conf.ini |grep datapath_type
  datapath_type = netdev

  [stack@localhost devstack]$ cat /etc/neutron/plugins/ml2/ml2_conf.ini |grep mtu
  path_mtu = 9000
  physical_network_mtus = 9000
  [stack@localhost devstack]$ cat /etc/neutron/neutron.conf |grep mtu
  global_physnet_mtu = 9000

Steps:
  1. Setup hugepages, update flavor (hw:mem_page_size='large', hw_rng:allowed='True')
  2. boot vm and try ping router and vice versa

br-int:

    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        datapath_type: netdev
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port br-int
            Interface br-int
                type: internal
        Port vhua5beaaae-c8
            tag: 1
            Interface vhua5beaaae-c8
                type: dpdkvhostuserclient
                options: {vhost-server-path="/var/run/openvswitch/vhua5beaaae-c8"}
        Port qr-c48b27ee-0f
            tag: 1
            Interface qr-c48b27ee-0f
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port tap3b960ab3-6a
            tag: 1
            Interface tap3b960ab3-6a
                type: internal


>From VM:
[root@test-mtu ~]# ip a |grep mtu
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc fq_codel state UP group default qlen 1000
[root@test-mtu ~]#
[root@test-mtu ~]# ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.275 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.167 ms

--- 192.168.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1035ms
rtt min/avg/max/mdev = 0.167/0.221/0.275/0.054 ms
[root@test-mtu ~]# ping 192.168.1.1 -s 2000
PING 192.168.1.1 (192.168.1.1) 2000(2028) bytes of data.

--- 192.168.1.1 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3086ms

[root@test-mtu ~]# ping 192.168.1.1 -c 5
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.139 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.154 ms
64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=0.144 ms
64 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=0.153 ms
64 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=0.146 ms

--- 192.168.1.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 0.139/0.147/0.154/0.005 ms
[root@test-mtu ~]# ping 192.168.1.1 -c 5 -s 2000
PING 192.168.1.1 (192.168.1.1) 2000(2028) bytes of data.

--- 192.168.1.1 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4084ms

[root@test-mtu ~]#

In ns:
[root@localhost ~]# ip a |grep mtu
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
29: qr-c48b27ee-0f: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 8950 qdisc fq_codel state UNKNOWN group default qlen 1000
[root@localhost ~]#
# default packet size:
10:44:41.276140 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 7, seq 1, length 64
10:44:41.276176 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 7, seq 1, length 64
10:44:42.323564 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 7, seq 2, length 64
10:44:42.323594 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 7, seq 2, length 64
10:44:43.325048 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 7, seq 3, length 64
10:44:43.325075 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 7, seq 3, length 64
10:44:44.326474 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 7, seq 4, length 64
10:44:44.326501 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 7, seq 4, length 64
10:44:45.327838 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 7, seq 5, length 64
10:44:45.327865 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 7, seq 5, length 64

# packet size 2000

[root@localhost ~]# tcpdump -i qr-c48b27ee-0f icmp
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on qr-c48b27ee-0f, link-type EN10MB (Ethernet), capture size 262144 bytes
10:46:00.934011 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 9, seq 1, length 2008
10:46:00.934046 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 9, seq 1, length 2008
10:46:01.941013 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 9, seq 2, length 2008
10:46:01.941051 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 9, seq 2, length 2008
10:46:02.965045 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 9, seq 3, length 2008
10:46:02.965072 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 9, seq 3, length 2008
10:46:03.989043 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 9, seq 4, length 2008
10:46:03.989083 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 9, seq 4, length 2008
10:46:05.013079 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 9, seq 5, length 2008
10:46:05.013108 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 9, seq 5, length 2008

>From ns:

[root@localhost ~]# ping 192.168.1.176 -c 2
PING 192.168.1.176 (192.168.1.176) 56(84) bytes of data.
64 bytes from 192.168.1.176: icmp_seq=1 ttl=64 time=0.193 ms
64 bytes from 192.168.1.176: icmp_seq=2 ttl=64 time=0.163 ms
--- 192.168.1.176 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1038ms
rtt min/avg/max/mdev = 0.163/0.178/0.193/0.015 ms
[root@localhost ~]# ping 192.168.1.176 -c 2 -s 2000
PING 192.168.1.176 (192.168.1.176) 2000(2028) bytes of data.

--- 192.168.1.176 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1060ms
[root@localhost ~]#

in vm:
08:51:37.228218 IP truncated-ip - 524 bytes missing! host-192-168-1-1.openstacklocal > host-192-168-1-176.openstacklocal: ICMP echo request, id 27328, seq 1, length 2008
08:51:38.245796 IP truncated-ip - 524 bytes missing! host-192-168-1-1.openstacklocal > host-192-168-1-176.openstacklocal: ICMP echo request, id 27328, seq 2, length 2008


The same issue if I try to ping dhcp server

If I set ovs datapath_type=system and don't use dpdkvhostuserclient -
all works fine

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: dpdk mtu

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1931844

Title:
  Can't ping router with packet size greater than 1476  when ovs
  datapath_type set to netdev

Status in neutron:
  New

Bug description:
  Setup:
    Openstack origin/stable/wallaby deployed by devstack on single node
    [stack@localhost devstack]$ cat /etc/centos-release
    CentOS Linux release 8.4.2105
    [stack@localhost devstack]$ uname -a
    Linux localhost.localdomain 4.18.0-305.3.1.el8.x86_64 #1 SMP Tue Jun 1 16:14:33 UTC 2021 x86_64 
    x86_64 x86_64 GNU/Linux
    [stack@localhost devstack]$

    [stack@localhost devstack]$ git status
    HEAD detached at origin/stable/wallaby

    ovs version (builded with DPDK)
    [stack@localhost devstack]$ ovs-vswitchd --version
    ovs-vswitchd (Open vSwitch) 2.13.4
    DPDK 19.11.7

    [stack@localhost devstack]$ cat /etc/neutron/plugins/ml2/ml2_conf.ini |grep datapath_type
    datapath_type = netdev

    [stack@localhost devstack]$ cat /etc/neutron/plugins/ml2/ml2_conf.ini |grep mtu
    path_mtu = 9000
    physical_network_mtus = 9000
    [stack@localhost devstack]$ cat /etc/neutron/neutron.conf |grep mtu
    global_physnet_mtu = 9000

  Steps:
    1. Setup hugepages, update flavor (hw:mem_page_size='large', hw_rng:allowed='True')
    2. boot vm and try ping router and vice versa

  br-int:

      Bridge br-int
          Controller "tcp:127.0.0.1:6633"
              is_connected: true
          fail_mode: secure
          datapath_type: netdev
          Port patch-tun
              Interface patch-tun
                  type: patch
                  options: {peer=patch-int}
          Port br-int
              Interface br-int
                  type: internal
          Port vhua5beaaae-c8
              tag: 1
              Interface vhua5beaaae-c8
                  type: dpdkvhostuserclient
                  options: {vhost-server-path="/var/run/openvswitch/vhua5beaaae-c8"}
          Port qr-c48b27ee-0f
              tag: 1
              Interface qr-c48b27ee-0f
                  type: internal
          Port int-br-ex
              Interface int-br-ex
                  type: patch
                  options: {peer=phy-br-ex}
          Port tap3b960ab3-6a
              tag: 1
              Interface tap3b960ab3-6a
                  type: internal

  
  From VM:
  [root@test-mtu ~]# ip a |grep mtu
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
  2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc fq_codel state UP group default qlen 1000
  [root@test-mtu ~]#
  [root@test-mtu ~]# ping 192.168.1.1
  PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
  64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.275 ms
  64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.167 ms

  --- 192.168.1.1 ping statistics ---
  2 packets transmitted, 2 received, 0% packet loss, time 1035ms
  rtt min/avg/max/mdev = 0.167/0.221/0.275/0.054 ms
  [root@test-mtu ~]# ping 192.168.1.1 -s 2000
  PING 192.168.1.1 (192.168.1.1) 2000(2028) bytes of data.

  --- 192.168.1.1 ping statistics ---
  4 packets transmitted, 0 received, 100% packet loss, time 3086ms

  [root@test-mtu ~]# ping 192.168.1.1 -c 5
  PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
  64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.139 ms
  64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.154 ms
  64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=0.144 ms
  64 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=0.153 ms
  64 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=0.146 ms

  --- 192.168.1.1 ping statistics ---
  5 packets transmitted, 5 received, 0% packet loss, time 4005ms
  rtt min/avg/max/mdev = 0.139/0.147/0.154/0.005 ms
  [root@test-mtu ~]# ping 192.168.1.1 -c 5 -s 2000
  PING 192.168.1.1 (192.168.1.1) 2000(2028) bytes of data.

  --- 192.168.1.1 ping statistics ---
  5 packets transmitted, 0 received, 100% packet loss, time 4084ms

  [root@test-mtu ~]#

  In ns:
  [root@localhost ~]# ip a |grep mtu
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
  29: qr-c48b27ee-0f: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 8950 qdisc fq_codel state UNKNOWN group default qlen 1000
  [root@localhost ~]#
  # default packet size:
  10:44:41.276140 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 7, seq 1, length 64
  10:44:41.276176 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 7, seq 1, length 64
  10:44:42.323564 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 7, seq 2, length 64
  10:44:42.323594 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 7, seq 2, length 64
  10:44:43.325048 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 7, seq 3, length 64
  10:44:43.325075 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 7, seq 3, length 64
  10:44:44.326474 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 7, seq 4, length 64
  10:44:44.326501 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 7, seq 4, length 64
  10:44:45.327838 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 7, seq 5, length 64
  10:44:45.327865 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 7, seq 5, length 64

  # packet size 2000

  [root@localhost ~]# tcpdump -i qr-c48b27ee-0f icmp
  dropped privs to tcpdump
  tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
  listening on qr-c48b27ee-0f, link-type EN10MB (Ethernet), capture size 262144 bytes
  10:46:00.934011 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 9, seq 1, length 2008
  10:46:00.934046 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 9, seq 1, length 2008
  10:46:01.941013 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 9, seq 2, length 2008
  10:46:01.941051 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 9, seq 2, length 2008
  10:46:02.965045 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 9, seq 3, length 2008
  10:46:02.965072 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 9, seq 3, length 2008
  10:46:03.989043 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 9, seq 4, length 2008
  10:46:03.989083 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 9, seq 4, length 2008
  10:46:05.013079 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 9, seq 5, length 2008
  10:46:05.013108 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 9, seq 5, length 2008

  From ns:

  [root@localhost ~]# ping 192.168.1.176 -c 2
  PING 192.168.1.176 (192.168.1.176) 56(84) bytes of data.
  64 bytes from 192.168.1.176: icmp_seq=1 ttl=64 time=0.193 ms
  64 bytes from 192.168.1.176: icmp_seq=2 ttl=64 time=0.163 ms
  --- 192.168.1.176 ping statistics ---
  2 packets transmitted, 2 received, 0% packet loss, time 1038ms
  rtt min/avg/max/mdev = 0.163/0.178/0.193/0.015 ms
  [root@localhost ~]# ping 192.168.1.176 -c 2 -s 2000
  PING 192.168.1.176 (192.168.1.176) 2000(2028) bytes of data.

  --- 192.168.1.176 ping statistics ---
  2 packets transmitted, 0 received, 100% packet loss, time 1060ms
  [root@localhost ~]#

  in vm:
  08:51:37.228218 IP truncated-ip - 524 bytes missing! host-192-168-1-1.openstacklocal > host-192-168-1-176.openstacklocal: ICMP echo request, id 27328, seq 1, length 2008
  08:51:38.245796 IP truncated-ip - 524 bytes missing! host-192-168-1-1.openstacklocal > host-192-168-1-176.openstacklocal: ICMP echo request, id 27328, seq 2, length 2008

  
  The same issue if I try to ping dhcp server

  If I set ovs datapath_type=system and don't use dpdkvhostuserclient -
  all works fine

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1931844/+subscriptions
Follow ups

[Bug 1931844] Re: Can't ping router with packet size greater than 1476 when ovs datapath_type set to netdev
From: LIU Yulong, 2021-06-15