yahoo-eng-team team mailing list archive

[Frode Nordahl <1899487@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

When using OpenStack cloud provider `cloud-init` will write out
/etc/netplan/50-cloud-init.yaml at initial instance boot and not update
it on subsequent boots of the instance.

The OpenStack metadata service provides information about MTU for the
network [0] and cloud-init takes this value and writes it into the
netplan configuration [1].

A side effect of configuring the MTU through netplan is that the
`systemd-networkd` [Link] section [2] gets the MTUBytes value filled and
this in turn makes `systemd-networkd` ignore the MTU value provided by
DHCP [3][4].

During the lifetime of a cloud events occur that will force a operator
to reduce the MTU available to instances attached to its overlay
networks. This may happen because of software imposed change of tunnel
type (GRE -> VXLAN, VXLAN -> GENEVE) or change of topology or
encapsulation in the physical network equipment.

To maximize performance these clouds have configured their instances to
use the maximum available MTU without leaving any headroom to account
for such changes and the only way to move forward is to reduce the
available MTU on the instances. We are facing a concrete challenge with
this now where we have users wanting to migrate from VXLAN tunnels to
GENEVE tunnels with 38 byte header size.

0: # curl http://169.254.169.254/openstack/2018-08-27/network_data.json
{"links": [{"id": "tapa035fb68-01", "vif_id": "a035fb68-010c-42e3-8da7-ea3c36a0d607", "type": "ovs", "mtu": 8942, "ethernet_mac_address": "fa:16:3e:31:26:f7"}], "networks": [{"id": "network0", "type": "ipv4_dhcp", "link": "tapa035fb68-01", "network_id": "b4ef84c0-1235-48a8-aaf7-03fab7ef5367"}], "services": []}

1: # cat /etc/netplan/50-cloud-init.yaml 
# This file is generated from information provided by the datasource.  Changes
# to it will not persist across an instance reboot.  To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    version: 2
    ethernets:
        ens2:
            dhcp4: true
            match:
                macaddress: fa:16:3e:31:26:f7
            mtu: 8950
            set-name: ens2

2: # cat /run/systemd/network/10-netplan-ens2.link 
[Match]
MACAddress=fa:16:3e:31:26:f7

[Link]
Name=ens2
WakeOnLan=off
MTUBytes=8950

3: # cat /run/systemd/network/10-netplan-ens2.network 
[Match]
MACAddress=fa:16:3e:31:26:f7
Name=ens2

[Link]
MTUBytes=8950

[Network]
DHCP=ipv4
LinkLocalAddressing=ipv6

[DHCP]
RouteMetric=100
UseMTU=true

4: Oct 12 13:30:18 canary-3 systemd-networkd[24084]:
/run/systemd/network/10-netplan-ens2.network: MTUBytes= in [Link]
section and UseMTU= in [DHCP] section are set. Disabling UseMTU=.

** Affects: cloud-init
     Importance: Undecided
         Status: New

** Attachment added: "cloud-init.tar.gz"
   https://bugs.launchpad.net/bugs/1899487/+attachment/5421289/+files/cloud-init.tar.gz

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1899487

Title:
  cloud-init hard codes MTU configuration at initial deploy time

Status in cloud-init:
  New

Bug description:
  When using OpenStack cloud provider `cloud-init` will write out
  /etc/netplan/50-cloud-init.yaml at initial instance boot and not
  update it on subsequent boots of the instance.

  The OpenStack metadata service provides information about MTU for the
  network [0] and cloud-init takes this value and writes it into the
  netplan configuration [1].

  A side effect of configuring the MTU through netplan is that the
  `systemd-networkd` [Link] section [2] gets the MTUBytes value filled
  and this in turn makes `systemd-networkd` ignore the MTU value
  provided by DHCP [3][4].

  During the lifetime of a cloud events occur that will force a operator
  to reduce the MTU available to instances attached to its overlay
  networks. This may happen because of software imposed change of tunnel
  type (GRE -> VXLAN, VXLAN -> GENEVE) or change of topology or
  encapsulation in the physical network equipment.

  To maximize performance these clouds have configured their instances
  to use the maximum available MTU without leaving any headroom to
  account for such changes and the only way to move forward is to reduce
  the available MTU on the instances. We are facing a concrete challenge
  with this now where we have users wanting to migrate from VXLAN
  tunnels to GENEVE tunnels with 38 byte header size.

  0: # curl http://169.254.169.254/openstack/2018-08-27/network_data.json
  {"links": [{"id": "tapa035fb68-01", "vif_id": "a035fb68-010c-42e3-8da7-ea3c36a0d607", "type": "ovs", "mtu": 8942, "ethernet_mac_address": "fa:16:3e:31:26:f7"}], "networks": [{"id": "network0", "type": "ipv4_dhcp", "link": "tapa035fb68-01", "network_id": "b4ef84c0-1235-48a8-aaf7-03fab7ef5367"}], "services": []}

  1: # cat /etc/netplan/50-cloud-init.yaml 
  # This file is generated from information provided by the datasource.  Changes
  # to it will not persist across an instance reboot.  To disable cloud-init's
  # network configuration capabilities, write a file
  # /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
  # network: {config: disabled}
  network:
      version: 2
      ethernets:
          ens2:
              dhcp4: true
              match:
                  macaddress: fa:16:3e:31:26:f7
              mtu: 8950
              set-name: ens2

  2: # cat /run/systemd/network/10-netplan-ens2.link 
  [Match]
  MACAddress=fa:16:3e:31:26:f7

  [Link]
  Name=ens2
  WakeOnLan=off
  MTUBytes=8950

  3: # cat /run/systemd/network/10-netplan-ens2.network 
  [Match]
  MACAddress=fa:16:3e:31:26:f7
  Name=ens2

  [Link]
  MTUBytes=8950

  [Network]
  DHCP=ipv4
  LinkLocalAddressing=ipv6

  [DHCP]
  RouteMetric=100
  UseMTU=true

  4: Oct 12 13:30:18 canary-3 systemd-networkd[24084]:
  /run/systemd/network/10-netplan-ens2.network: MTUBytes= in [Link]
  section and UseMTU= in [DHCP] section are set. Disabling UseMTU=.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1899487/+subscriptions