yahoo-eng-team team mailing list archive

[Aaron Rosen <aaronorosen@xxxxxxxxx>]

** Changed in: neutron
       Status: Incomplete => Invalid

** Changed in: neutron
       Status: Invalid => Fix Released

** Changed in: tripleo
       Status: Triaged => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1271344

Title:
  neutron-dhcp-agent doesn't hand out leases for recently used addresses

Status in OpenStack Neutron (virtual network service):
  Fix Released
Status in tripleo - openstack on openstack:
  Fix Released

Bug description:
  Hi, last two days on ci-overcloud.tripleo.org, the neutron-dhcp-agent
  has stopped answering properly for new VMs - new VMs don't get IP
  addresses until neutron-dhcp-agent is restarted (we see cloud-init
  waiting in nova console-log)

  There is a bunch of rabbit disconnect messages in the log, but they
  are not correlated with the fault.

  Existing VMs can DHCP successfully, and the overlay network seems
  fine.

  I've checked the dhcp agent host file and it is being updated.

  restarting the neutron-dhcp-agent 'fixes' the problem.

  kill -HUPing the dnsmasqs doesn't seem to fix things

  -> The issue is that a prior VM gets (say ) 192.168.1.25, then is shutdown.
  -> we then issue 1.25 to a new VM, but the dnsmasq process still has 1.25 in its in-memory leases table as being for the other adress.

   - DHCPRELEASE *is* being issued, perhaps sporadic failures or some
  other condition/

  # WORKAROUNDS:
   - restart the agent in a cronjob
   - change reload_allocations to return self.restart(), avoiding the cache in dnsmasq but introducing short interrupts in DHCP on each new VM deployed.

  # POSSIBLE SOLUTIONS
   - change dnsmasq to discard leases that were statically allocated and no longer have that MAC address (e.g. to trust the config more)
   - change neutron to only recycle port addresses when the discarded MAC has been deleted more than (lease time).
   - use dnsmasqs multiple-MAC feature to tell dnsmasq that a handover is happening.[1]
   - the lease change script is apparently called for all leases when SIGHUP is received, so maybe we can terminate dead leases there? [2]

  [1] "              As  a  special  case,  in DHCPv4, it is possible to include more than one hardware address. eg:
                --dhcp-host=11:22:33:44:55:66,12:34:56:78:90:12,192.168.0.2 This allows an  IP  address  to  be associated  with  multiple  hardware  addresses, and gives dnsmasq permission to abandon a DHCP lease to one of the hardware addresses when another one asks for a lease. Beware that this is a dangerous  thing  to  do,  it  will only work reliably if only one of the hardware addresses is active at any time and there is no way for dnsmasq to enforce this. It is, for instance, useful to allocate a stable IP address to a laptop which has both wired and wireless interfaces.
  "

  [2] "When it receives a SIGHUP, dnsmasq clears its cache and then re-
  loads /etc/hosts and  /etc/ethers  and any  file given by --dhcp-
  hostsfile, --dhcp-optsfile or --addn-hosts.  The dhcp lease change
  script is called for all existing DHCP leases."

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1271344/+subscriptions