yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1783908] [NEW] dnsmasq does not remove leases for deleted VMs - leases and host files point to different MACS

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Arjun Baindur <xagent@xxxxxxxxx>
Date: Fri, 27 Jul 2018 00:05:36 -0000
Reply-to: Bug 1783908 <1783908@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

We see this sporadically, sometimes it works, sometimes it doesn't. We
delete VMs, then create new ones. The new fixed IP can't get an IP via
DHCP, because the leases file still points to MAC of some old, deleted
VMs. The host file is correctly updated.

For example we see in syslog/messages:

Jul 26 16:45:04 stan dnsmasq-dhcp[18540]: not using configured address 10.96.0.21 because it is leased to fa:16:3e:ae:c4:b0
Jul 26 16:45:04 stan dnsmasq-dhcp[18540]: DHCPDISCOVER(tap6d5fc8f6-28) fa:16:3e:2f:29:37 no address available

And tcpdump on tap interface of qdhcp indicates it is receiving the DHCP
requests, but nothing is sent out.

On inspection of the leases and host file, we see the MAC address
differs for the fixed IP:

[root@stan ~]# cat /var/opt/pf9/neutron/dhcp/52f8298e-f867-4581-8ae7-fac0c9196ae4/host | grep 10.96.0.21
fa:16:3e:2f:29:37,test-pf9-vmha-one-region-c7-702498-490-2.platform9.sys.,10.96.0.21
[root@stan ~]# cat /var/opt/pf9/neutron/dhcp/52f8298e-f867-4581-8ae7-fac0c9196ae4/leases | grep 10.96.0.21
1532649702 fa:16:3e:ae:c4:b0 10.96.0.21 test-pf9-vmha-one-region-c7-702498-490-2 *


On another dhcp-agent host, we saw the leases file pointed to some OTHER old third MAC.

We then deleted this VM. We observed that the lease was removed on one
host, but it remained on the other host. So it appears to be random.
Enabling DEBUG logs shows the dhcp_release command is being invoked,
without error:

2018-07-26 16:52:06.214 10660 DEBUG neutron.agent.dhcp.agent [req-22449d0c-c0e3-4571-b4f9-a3150f8596c5 5412536c76f942d980e1834f48340f77 f175f441ebbb4c2b8fedf6469d6415fc - - -] Calling driver for network: 52f8298e-f867-4581-8ae7-fac0c9196ae4 action: reload_allocations call_driver /opt/pf9/pf9-neutron/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py:133
2018-07-26 16:52:06.217 10660 DEBUG neutron.agent.linux.utils [req-22449d0c-c0e3-4571-b4f9-a3150f8596c5 5412536c76f942d980e1834f48340f77 f175f441ebbb4c2b8fedf6469d6415fc - - -] Running command: ['sudo', '/opt/pf9/pf9-neutron/bin/neutron-rootwrap', '/opt/pf9/etc/pf9-neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qdhcp-52f8298e-f867-4581-8ae7-fac0c9196ae4', 'dhcp_release', 'tap4d6bb403-59', '10.96.0.21', 'fa:16:3e:2f:29:37'] create_process /opt/pf9/pf9-neutron/lib/python2.7/site-packages/neutron/agent/linux/utils.py:92
2018-07-26 16:52:06.498 10660 DEBUG neutron.agent.linux.dhcp [req-22449d0c-c0e3-4571-b4f9-a3150f8596c5 5412536c76f942d980e1834f48340f77 f175f441ebbb4c2b8fedf6469d6415fc - - -] Building host file: /var/opt/pf9/neutron/dhcp/52f8298e-f867-4581-8ae7-fac0c9196ae4/host _output_hosts_file /opt/pf9/pf9-neutron/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py:685
2018-07-26 16:52:06.505 10660 DEBUG neutron.agent.linux.dhcp [req-22449d0c-c0e3-4571-b4f9-a3150f8596c5 5412536c76f942d980e1834f48340f77 f175f441ebbb4c2b8fedf6469d6415fc - - -] Done building host file /var/opt/pf9/neutron/dhcp/52f8298e-f867-4581-8ae7-fac0c9196ae4/host _output_hosts_file /opt/pf9/pf9-neutron/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py:724


The entry was removed from hosts file, but remains in leases despite above dhcp_release remaining. If we spin up a new VM with this fixed IP, it can't get an IP due to old lease:

root@krusty:~# cat /var/opt/pf9/neutron/dhcp/52f8298e-f867-4581-8ae7-fac0c9196ae4/host | grep 10.96.0.21
root@krusty:~# cat /var/opt/pf9/neutron/dhcp/52f8298e-f867-4581-8ae7-fac0c9196ae4/leases | grep 10.96.0.21
1532735433 fa:16:3e:2f:29:37 10.96.0.21 test-pf9-vmha-one-region-c7-702498-490-2 *
root@krusty:~#

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: l3-ipam-dhcp

** Summary changed:

- dnsmasq not removes leases for deleted VMs
+ dnsmasq does not remove leases for deleted VMs - leases and host files point to different MACS

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1783908

Title:
  dnsmasq does not remove leases for deleted VMs - leases and host files
  point to different MACS

Status in neutron:
  New

Bug description:
  We see this sporadically, sometimes it works, sometimes it doesn't. We
  delete VMs, then create new ones. The new fixed IP can't get an IP via
  DHCP, because the leases file still points to MAC of some old, deleted
  VMs. The host file is correctly updated.

  For example we see in syslog/messages:

  Jul 26 16:45:04 stan dnsmasq-dhcp[18540]: not using configured address 10.96.0.21 because it is leased to fa:16:3e:ae:c4:b0
  Jul 26 16:45:04 stan dnsmasq-dhcp[18540]: DHCPDISCOVER(tap6d5fc8f6-28) fa:16:3e:2f:29:37 no address available

  And tcpdump on tap interface of qdhcp indicates it is receiving the
  DHCP requests, but nothing is sent out.

  On inspection of the leases and host file, we see the MAC address
  differs for the fixed IP:

  [root@stan ~]# cat /var/opt/pf9/neutron/dhcp/52f8298e-f867-4581-8ae7-fac0c9196ae4/host | grep 10.96.0.21
  fa:16:3e:2f:29:37,test-pf9-vmha-one-region-c7-702498-490-2.platform9.sys.,10.96.0.21
  [root@stan ~]# cat /var/opt/pf9/neutron/dhcp/52f8298e-f867-4581-8ae7-fac0c9196ae4/leases | grep 10.96.0.21
  1532649702 fa:16:3e:ae:c4:b0 10.96.0.21 test-pf9-vmha-one-region-c7-702498-490-2 *

  
  On another dhcp-agent host, we saw the leases file pointed to some OTHER old third MAC.

  We then deleted this VM. We observed that the lease was removed on one
  host, but it remained on the other host. So it appears to be random.
  Enabling DEBUG logs shows the dhcp_release command is being invoked,
  without error:

  2018-07-26 16:52:06.214 10660 DEBUG neutron.agent.dhcp.agent [req-22449d0c-c0e3-4571-b4f9-a3150f8596c5 5412536c76f942d980e1834f48340f77 f175f441ebbb4c2b8fedf6469d6415fc - - -] Calling driver for network: 52f8298e-f867-4581-8ae7-fac0c9196ae4 action: reload_allocations call_driver /opt/pf9/pf9-neutron/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py:133
  2018-07-26 16:52:06.217 10660 DEBUG neutron.agent.linux.utils [req-22449d0c-c0e3-4571-b4f9-a3150f8596c5 5412536c76f942d980e1834f48340f77 f175f441ebbb4c2b8fedf6469d6415fc - - -] Running command: ['sudo', '/opt/pf9/pf9-neutron/bin/neutron-rootwrap', '/opt/pf9/etc/pf9-neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qdhcp-52f8298e-f867-4581-8ae7-fac0c9196ae4', 'dhcp_release', 'tap4d6bb403-59', '10.96.0.21', 'fa:16:3e:2f:29:37'] create_process /opt/pf9/pf9-neutron/lib/python2.7/site-packages/neutron/agent/linux/utils.py:92
  2018-07-26 16:52:06.498 10660 DEBUG neutron.agent.linux.dhcp [req-22449d0c-c0e3-4571-b4f9-a3150f8596c5 5412536c76f942d980e1834f48340f77 f175f441ebbb4c2b8fedf6469d6415fc - - -] Building host file: /var/opt/pf9/neutron/dhcp/52f8298e-f867-4581-8ae7-fac0c9196ae4/host _output_hosts_file /opt/pf9/pf9-neutron/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py:685
  2018-07-26 16:52:06.505 10660 DEBUG neutron.agent.linux.dhcp [req-22449d0c-c0e3-4571-b4f9-a3150f8596c5 5412536c76f942d980e1834f48340f77 f175f441ebbb4c2b8fedf6469d6415fc - - -] Done building host file /var/opt/pf9/neutron/dhcp/52f8298e-f867-4581-8ae7-fac0c9196ae4/host _output_hosts_file /opt/pf9/pf9-neutron/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py:724

  
  The entry was removed from hosts file, but remains in leases despite above dhcp_release remaining. If we spin up a new VM with this fixed IP, it can't get an IP due to old lease:

  root@krusty:~# cat /var/opt/pf9/neutron/dhcp/52f8298e-f867-4581-8ae7-fac0c9196ae4/host | grep 10.96.0.21
  root@krusty:~# cat /var/opt/pf9/neutron/dhcp/52f8298e-f867-4581-8ae7-fac0c9196ae4/leases | grep 10.96.0.21
  1532735433 fa:16:3e:2f:29:37 10.96.0.21 test-pf9-vmha-one-region-c7-702498-490-2 *
  root@krusty:~#

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1783908/+subscriptions
Follow ups

[Bug 1783908] Re: dnsmasq does not remove leases for deleted VMs - leases and host files point to different MACS
From: Brian Haley, 2022-12-06