yahoo-eng-team team mailing list archive

[Brian Haley <1783908@xxxxxxxxxxxxxxxxxx>]

There have been a number of fixes in the past four years to the DHCP
code. As I have not seen this issue lately it could have been fixed as
part of another change. For that reason I'll close this but please open
a new bug if the issue persists.

** Changed in: neutron
       Status: Confirmed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1783908

Title:
  dnsmasq does not remove leases for deleted VMs - leases and host files
  point to different MACS

Status in neutron:
  Fix Released

Bug description:
  We see this sporadically, sometimes it works, sometimes it doesn't. We
  delete VMs, then create new ones. The new fixed IP can't get an IP via
  DHCP, because the leases file still points to MAC of some old, deleted
  VMs. The host file is correctly updated.

  For example we see in syslog/messages:

  Jul 26 16:45:04 stan dnsmasq-dhcp[18540]: not using configured address 10.96.0.21 because it is leased to fa:16:3e:ae:c4:b0
  Jul 26 16:45:04 stan dnsmasq-dhcp[18540]: DHCPDISCOVER(tap6d5fc8f6-28) fa:16:3e:2f:29:37 no address available

  And tcpdump on tap interface of qdhcp indicates it is receiving the
  DHCP requests, but nothing is sent out.

  On inspection of the leases and host file, we see the MAC address
  differs for the fixed IP:

  [root@stan ~]# cat /var/opt/pf9/neutron/dhcp/52f8298e-f867-4581-8ae7-fac0c9196ae4/host | grep 10.96.0.21
  fa:16:3e:2f:29:37,test-pf9-vmha-one-region-c7-702498-490-2.platform9.sys.,10.96.0.21
  [root@stan ~]# cat /var/opt/pf9/neutron/dhcp/52f8298e-f867-4581-8ae7-fac0c9196ae4/leases | grep 10.96.0.21
  1532649702 fa:16:3e:ae:c4:b0 10.96.0.21 test-pf9-vmha-one-region-c7-702498-490-2 *

  
  On another dhcp-agent host, we saw the leases file pointed to some OTHER old third MAC.

  We then deleted this VM. We observed that the lease was removed on one
  host, but it remained on the other host. So it appears to be random.
  Enabling DEBUG logs shows the dhcp_release command is being invoked,
  without error:

  2018-07-26 16:52:06.214 10660 DEBUG neutron.agent.dhcp.agent [req-22449d0c-c0e3-4571-b4f9-a3150f8596c5 5412536c76f942d980e1834f48340f77 f175f441ebbb4c2b8fedf6469d6415fc - - -] Calling driver for network: 52f8298e-f867-4581-8ae7-fac0c9196ae4 action: reload_allocations call_driver /opt/pf9/pf9-neutron/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py:133
  2018-07-26 16:52:06.217 10660 DEBUG neutron.agent.linux.utils [req-22449d0c-c0e3-4571-b4f9-a3150f8596c5 5412536c76f942d980e1834f48340f77 f175f441ebbb4c2b8fedf6469d6415fc - - -] Running command: ['sudo', '/opt/pf9/pf9-neutron/bin/neutron-rootwrap', '/opt/pf9/etc/pf9-neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qdhcp-52f8298e-f867-4581-8ae7-fac0c9196ae4', 'dhcp_release', 'tap4d6bb403-59', '10.96.0.21', 'fa:16:3e:2f:29:37'] create_process /opt/pf9/pf9-neutron/lib/python2.7/site-packages/neutron/agent/linux/utils.py:92
  2018-07-26 16:52:06.498 10660 DEBUG neutron.agent.linux.dhcp [req-22449d0c-c0e3-4571-b4f9-a3150f8596c5 5412536c76f942d980e1834f48340f77 f175f441ebbb4c2b8fedf6469d6415fc - - -] Building host file: /var/opt/pf9/neutron/dhcp/52f8298e-f867-4581-8ae7-fac0c9196ae4/host _output_hosts_file /opt/pf9/pf9-neutron/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py:685
  2018-07-26 16:52:06.505 10660 DEBUG neutron.agent.linux.dhcp [req-22449d0c-c0e3-4571-b4f9-a3150f8596c5 5412536c76f942d980e1834f48340f77 f175f441ebbb4c2b8fedf6469d6415fc - - -] Done building host file /var/opt/pf9/neutron/dhcp/52f8298e-f867-4581-8ae7-fac0c9196ae4/host _output_hosts_file /opt/pf9/pf9-neutron/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py:724

  
  The entry was removed from hosts file, but remains in leases despite above dhcp_release remaining. If we spin up a new VM with this fixed IP, it can't get an IP due to old lease:

  root@krusty:~# cat /var/opt/pf9/neutron/dhcp/52f8298e-f867-4581-8ae7-fac0c9196ae4/host | grep 10.96.0.21
  root@krusty:~# cat /var/opt/pf9/neutron/dhcp/52f8298e-f867-4581-8ae7-fac0c9196ae4/leases | grep 10.96.0.21
  1532735433 fa:16:3e:2f:29:37 10.96.0.21 test-pf9-vmha-one-region-c7-702498-490-2 *
  root@krusty:~#

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1783908/+subscriptions