yahoo-eng-team team mailing list archive

[OpenStack Infra <1806770@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/623066
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f2111e035424bf714099966ad724e9a4bd604c18
Submitter: Zuul
Branch:    master

commit f2111e035424bf714099966ad724e9a4bd604c18
Author: Arjun Baindur <xagent@xxxxxxxxx>
Date:   Wed Dec 5 12:43:05 2018 -0800

    Do not release DHCP lease when no client ID is set on port
    
    The DHCP agent has a really strict enforcement of client ID, which
    is part of the DHCP extra options. If a VM advertises a client ID,
    DHCP agent will automatically release it's lease whenever *any* other
    port is updated/deleted, even if no client ID is set on the port,
    because it thinks the client ID has changed.
    
    When reload_allocations() is called, the DHCP agent parses the leases
    and hosts files, and gets the list of all the ports in the network from the
    DB, computing 3 different sets. The set from the leases file (v4_leases)
    could have a client ID, but the set from the port DB and hosts file will
    have None.
    
    As a result, the set subtraction does not filter out the entry,
    and all ports that have an active lease with a client ID are released.
    
    The Client ID should only be enforced and leases released
    if it's actually set in the port DB's DHCP extra Opts.
    In that case it means someone knows what they are doing,
    and we want to check for a mismatch. If the client ID on a port is
    empty, it should not be treated like an unused lease.
    
    We can't expect end users that just create VMs with auto created ports
    to know/care about DHCP client IDs, then manually update ports or
    change app templates.
    
    In some cases, like Windows VMs, the client ID is advertised as the MAC by default.
    In fact, there is a Windows bug which prevents you from even turning this off:
    https://support.microsoft.com/en-us/help/3004537/dhcp-client-always-includes-option-61-in-the-dhcp-request-in-windows-8
    
    Linux VMs don't have this on by default, but it may be enabled
    in some templates unknown to users.
    
    Change-Id: I8021f740bd78e654915337bd3287b45b2c422e95
    Closes-Bug: #1806770


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1806770

Title:
  DHCP Agent should not release DHCP lease when client ID is not set on
  port

Status in neutron:
  Fix Released

Bug description:
  DHCP agent has a really strict enforcement of client ID, which is part
  of the DHCP extra options. If a VM advertises a client ID, DHCP agent
  will automatically release it's lease whenever *any* other port is
  updated/deleted. This happens even if no client ID is set on the port.

  When reload_allocations() is called, DHCP agent parses the current
  leases file, the hosts file, and gets the list all the ports in the
  network from DB, computing 3 different sets. The set from leases file
  (v4_leases) will have some client ID. The set from port DB will have
  None. As a result the set subtraction does not filter out the entry,
  and the port's DHCP lease is constantly released, whenever the VM
  renews its lease and any other port in the network is deleted:

  https://github.com/openstack/neutron/blob/stable/pike/neutron/agent/linux/dhcp.py#L850

          v4_leases = set()
          for (k, v) in cur_leases.items():
              # IPv4 leases have a MAC, IPv6 ones do not, so we must ignore
              if netaddr.IPAddress(k).version == constants.IP_VERSION_4:
                  # treat '*' as None, see note in _read_leases_file_leases()
                  client_id = v['client_id']
                  if client_id is '*':
                      client_id = None
                  v4_leases.add((k, v['iaid'], client_id))

          new_leases = set()
          for port in self.network.ports:
              client_id = self._get_client_id(port)
              for alloc in port.fixed_ips:
                  new_leases.add((alloc.ip_address, port.mac_address, client_id))

          # If an entry is in the leases or host file(s), but doesn't have
          # a fixed IP on a corresponding neutron port, consider it stale.
          entries_to_release = (v4_leases | old_leases) - new_leases
          if not entries_to_release:
              return

  It was observed in one example of a released lease, its entries looked
  like:

  new_leases (from port DB)
  (u'10.81.96.186', u'fa:16:3e:eb:a1:13', None)
  old_leases (from hosts file)
  ('10.81.96.186', 'fa:16:3e:eb:a1:13', None)
  v4_leases (from leases file - updated by dnsmasq when VM requests)
  ('10.81.96.186', 'fa:16:3e:eb:a1:13', '01:fa:16:3e:eb:a1:13')

  Therefore the entries_to_release did not have that IP, MAC filtered
  out. The client_id in v4_leases entry was coming from a Windows VM,
  which faces a bug that prevents it from disabling client ID.
  entries_to_release in fact had some 50+ entries like that, causing a
  storm of DHCPRELEASE.

  This can cause issues where when the VM later asks to renew its lease
  when the expiry period is coming up (I think about halfway thru),
  dnsmasq sends an DHCP NAK and the lease is re-negotiated and existing
  networking connections can get disrupted. It also just causes DHCP
  agent to do unneccessary work, releasing a ton of leases when it
  technically shouldn't.

  Setting the client ID in the port's DHCP extra opts is not an good
  solution:

  1. In some cases, like Windows VMs, the client ID is advertised as the
  MAC by default. In fact, there is a Windows bug which prevents you
  from even turning this off: https://support.microsoft.com/en-
  us/help/3004537/dhcp-client-always-includes-option-61-in-the-dhcp-
  request-in-windows-8

  Linux VMs dont have this on by default, when I checked, but they may
  be enabled in some templates unknown to users

  2. End users will usually just be deploying a VM, with the port being
  auto created by Nova. They don't know or need to know about advanced
  networking concepts like DHCP client IDs.

  3. We can't expect everyone to modify their existing app templates, or
  end users to make API calls, to update ports everytime they deploy a
  VM

  So, client ID should only be enforced, and leases released, if it's
  actually set on the port DB's DHCP extra Opts. In that case it means
  someone knows what they are doing, and we want to check for a
  mismatch.

  If its None, I suspect in 99.9999% of cases the operator does not know
  or care about client ID field.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1806770/+subscriptions