yahoo-eng-team team mailing list archive

[OpenStack Infra <1542475@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/283790
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4df8d9a7016ab20fce235833d792b89309ec98a7
Submitter: Jenkins
Branch:    master

commit 4df8d9a7016ab20fce235833d792b89309ec98a7
Author: Kevin Benton <kevin@xxxxxxxxxx>
Date:   Mon Feb 22 16:41:45 2016 -0800

    Make agent interface plugging utilize network MTU
    
    This changes the 'plug' and 'plug_new' interfaces of the
    LinuxInterfaceDriver to accept an MTU argument. It then
    updates the dhcp agent and l3 agent to pass the MTU that
    is set on the network that the port belongs to. This allows
    it to take into account the overhead calculations that are
    done for encapsulation types.
    
    It's necessary for the L3 agent to have the MTU because it
    must recognize when fragmentation is needed so it can fragment
    or generate an ICMP error.
    
    It's necessary for the DHCP agent to have the MTU so it doesn't
    interfere when it plugs into a bridge with a larger than 1500
    MTU (the bridge would reduce its MTU to match the agent).
    
    If an operator sets 'network_device_mtu', the value of that
    will be used instead to preserve previous behavior.
    
    Closes-Bug: #1549470
    Closes-Bug: #1542108
    Closes-Bug: #1542475
    DocImpact: Neutron agents now support arbitrary MTU
               configurations on each network (including
               jumbo frames). This is accomplished by checking
               the MTU value defined for each network on which
               it is wiring VIFs.
    Co-Authored-By: Matt Kassawara <mkassawara@xxxxxxxxx>
    Change-Id: Ic091fa78dfd133179c71cbc847bf955a06cb248a


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1542475

Title:
  MTU concerns for the Open vSwitch agent

Status in neutron:
  Fix Released

Bug description:
  I ran some experiments with the Open vSwitch (OVS) agent [1] to
  determine the source of MTU problems and offer a potential solution.
  The environment for these experiments contains the following items:

  1) A physical (underlying) network supporting MTU of 1500 or 9000 bytes.
  2) One controller node running the neutron server, OVS agent, L3 agent, DHCP agent, metadata agent, and OVS provider network bridge br-ex.
  3) One compute node running the Open vSwitch agent.
  4) A neutron provider/public network.
  5) A neutron self-service/private network.
  6) A neutron router between the provider and self-service networks.
  7) The self-service network uses the VXLAN protocol with IPv4 endpoints which adds 50 bytes of overhead.
  8) An instance on the self-service network with a floating IP address from an allocation pool on the provider network.

  Background:

  1. Interfaces (or ports) on OVS bridges such as those for overlay
  network tunnels appear to use an arbitrarily large MTU. Thus, OVS
  bridges and tunnel interfaces somewhat inherit the MTU of physical
  network interfaces. For example, if OVS uses the IP address of eth0
  for a tunnel overlay network endpoint and eth0 has a 1500 MTU, the
  tunnel interface can only send packets with a payload of up to 1500
  bytes including overlay protocol overhead.

  2. OVS creates interfaces (ports) in the host namespace and moves them
  to the appropriate namespace(s) rather than creating veth pairs
  between namespaces.

  3. For Linux bridge devices such as those on the compute node that
  implement security groups, Linux assumes a 1500 MTU and changes the
  MTU to the lowest MTU of any port on the bridge. For example, a bridge
  without ports has a 1500 MTU. If eth0 has a 9000 MTU and you add it as
  a port on the bridge, the bridge changes to a 9000 MTU. If eth1 has a
  1500 MTU and you add it as a port on the bridge, the bridge changes to
  a 1500 MTU.

  4. Only devices that operate at layer-3 can participate in path MTU
  discovery (PMTUD). Therefore, a change of MTU in a layer-2 device such
  as a bridge or veth pair causes that device to discard packets larger
  than the smallest MTU.

  Observations:

  1. For any physical network MTU, the port for the self-service network
  router interface (qr) in the router namespace (qrouter) has a 1500
  MTU. Background item (2) prevents a MTU disparity at layer-2 between
  the router namespace and OVS bridge br-int. If a packet from the
  provider network to the instance has a payload larger than 1500 bytes,
  the router can send an ICMP message to the source telling it to use a
  1500 MTU. However, the correct MTU for a private network using the
  VXLAN overlay protocol should account for 50 bytes of overhead. Thus,
  OVS fragments the packet over the tunnel and reassembles it on the
  compute node containing the instance.

  2. For a physical network MTU larger than 1500, the port for the
  provider network router gateway (qg) in the router namespace (qrouter)
  has a 1500 MTU. Background item (2) prevents a MTU disparity between
  the router namespace and OVS provider network bridge br-ex. If a
  packet from the provider network to the instance has a payload larger
  than 1500 bytes, the router can send an ICMP messages to the source
  telling it to use a 1500 MTU regardless of the private network overlay
  protocol. Thus, the agent cannot realize a physical network MTU larger
  than 1500.

  3. If a provider or private network uses DHCP, the port in the DHCP
  namespace has a 1500 MTU for any physical network MTU.

  4. The Linux bridge that implements security groups on the compute
  node lacks any ports on physical network interfaces. Background item
  (3) causes the bridge to assume a 1500 MTU. Nova actually manages this
  bridge and creates a veth pair between it and the Open vSwitch bridge
  br-int. Both ends of the veth pair have a 1500 MTU. Background item
  (1) indicates that the OVS bridge br-int could have a larger MTU.
  Thus, OVS discards packets inbound to instances with a payload larger
  than 1500 bytes.

  5. Instances must use a MTU value the accounts for overlay protocol
  overhead. Neutron currently offers a way to provide a correct value
  via DHCP. However, considering observation item (4), providing a MTU
  value larger than 1500 causes a disparity at layer-2 between the VM
  and tap interface port on the Linux bridge that implements security
  groups on the compute node. Thus, the bridge discards packets outbound
  from instances with a payload larger than 1500 bytes.

  6. The nova 'network_device_mtu' option controls the MTU of all
  devices that it manages in observation items (4) and (5). For example,
  using a value of 9000 causes the bridge, veth pair, and tap to have a
  9000 MTU. Combining this option with providing the correct value to
  instances via DHCP essentially resolves MTU problems on compute nodes.

  Potential solution:

  1. The port for the self-service network router interface (qr) in the
  router namespace (qrouter) must use the MTU of the physical network
  accounting for any overlay protocol overhead. For example, if the
  physical network has a 9000 MTU and the private network uses the VXLAN
  overlay protocol, the port must have a 8950 MTU.

  2. The port for the provider network router gateway (qg) in the router
  namespace (qrouter) must use the MTU of the physical network. For
  example, if the physical network has a 9000 MTU, the port must have a
  9000 MTU. If the provider network uses an overlay protocol, the MTU of
  the port must also account for any overhead.

  3. For networks using DHCP, the port in the DHCP namespace (qdhcp)
  should use the MTU of the network on which it provides services
  accounting for any overlay protocol overhead.

  4. The Linux bridge that implements security groups on the compute
  node and all ports on it must use the MTU of the physical network
  accounting for any overlay protocol overhead.

  [1] http://lists.openstack.org/pipermail/openstack-
  dev/2016-January/084241.html

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1542475/+subscriptions