yahoo-eng-team team mailing list archive

[Alexander Balderson <1899487@xxxxxxxxxxxxxxxxxx>]

I'm going to re-open this bug after working through Openstack networking
migration for OVS to OVN.

I ran into an issue where the MTU set in netplan caused all my instances
to be lost after the migration. While this can, and should, be
documented, I also think that reducing places where instances could be
lost should be taken whenever possible.


** Changed in: cloud-init
       Status: Expired => New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1899487

Title:
  cloud-init hard codes MTU configuration at initial deploy time

Status in cloud-init:
  New
Status in netplan.io package in Ubuntu:
  Expired

Bug description:
  When using OpenStack cloud provider `cloud-init` will write out
  /etc/netplan/50-cloud-init.yaml at initial instance boot and not
  update it on subsequent boots of the instance.

  The OpenStack metadata service provides information about MTU for the
  network [0] and cloud-init takes this value and writes it into the
  netplan configuration [1].

  A side effect of configuring the MTU through netplan is that the
  `systemd-networkd` [Link] section [2] gets the MTUBytes value filled
  and this in turn makes `systemd-networkd` ignore the MTU value
  provided by DHCP [3][4].

  During the lifetime of a cloud events occur that will force a operator
  to reduce the MTU available to instances attached to its overlay
  networks. This may happen because of software imposed change of tunnel
  type (GRE -> VXLAN, VXLAN -> GENEVE) or change of topology or
  encapsulation in the physical network equipment.

  To maximize performance these clouds have configured their instances
  to use the maximum available MTU without leaving any headroom to
  account for such changes and the only way to move forward is to reduce
  the available MTU on the instances. We are facing a concrete challenge
  with this now where we have users wanting to migrate from VXLAN
  tunnels to GENEVE tunnels with 38 byte header size.

  0: # curl http://169.254.169.254/openstack/2018-08-27/network_data.json
  {"links": [{"id": "tapa035fb68-01", "vif_id": "a035fb68-010c-42e3-8da7-ea3c36a0d607", "type": "ovs", "mtu": 8942, "ethernet_mac_address": "fa:16:3e:31:26:f7"}], "networks": [{"id": "network0", "type": "ipv4_dhcp", "link": "tapa035fb68-01", "network_id": "b4ef84c0-1235-48a8-aaf7-03fab7ef5367"}], "services": []}

  1: # cat /etc/netplan/50-cloud-init.yaml 
  # This file is generated from information provided by the datasource.  Changes
  # to it will not persist across an instance reboot.  To disable cloud-init's
  # network configuration capabilities, write a file
  # /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
  # network: {config: disabled}
  network:
      version: 2
      ethernets:
          ens2:
              dhcp4: true
              match:
                  macaddress: fa:16:3e:31:26:f7
              mtu: 8950
              set-name: ens2

  2: # cat /run/systemd/network/10-netplan-ens2.link 
  [Match]
  MACAddress=fa:16:3e:31:26:f7

  [Link]
  Name=ens2
  WakeOnLan=off
  MTUBytes=8950

  3: # cat /run/systemd/network/10-netplan-ens2.network 
  [Match]
  MACAddress=fa:16:3e:31:26:f7
  Name=ens2

  [Link]
  MTUBytes=8950

  [Network]
  DHCP=ipv4
  LinkLocalAddressing=ipv6

  [DHCP]
  RouteMetric=100
  UseMTU=true

  4: Oct 12 13:30:18 canary-3 systemd-networkd[24084]:
  /run/systemd/network/10-netplan-ens2.network: MTUBytes= in [Link]
  section and UseMTU= in [DHCP] section are set. Disabling UseMTU=.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1899487/+subscriptions