← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #76132

[Bug 1807382] [NEW] DNSMASQ wrong addresses allocated after changing DHCP Clients between Neutron vRouters NET

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

I'm using OpenStack, basically creating Virtual Routers with
net/subnets, and DNSMASQ is the DHCP Server for them (via neutron-dhcp
driver).

My test scenario contains 2 virtual routers (2 namespaces as router) with 1 DHCP each (2 namespaces as dhcp servers) on the Neutron Network Server. The Subnet are 100.97.97.0/24 for NET with VLAN 997, and Subnet 100.98.98.0/24 for Network with VLAN 998. Both are on the same physical interface (BR-NET2). 
The 2 DHCP servers have available .11 to .64 IP address range (54 addresses). (dnsmasq config file is bellow)
I'm generation 60 MAC addresses randomly as source MAC address to test DHCP Client requests. Theses 60 addresses are always the same. 


The DHCP Client script tester is connected to Network with VLAN 997 and subnet 100.97.97.0/24. I generate DHCP client MAC addresses and only 54 are allocated, as expected.  So, addresses from range 100.97.97.11 to 100.97.97.64 are allocated to client requests. This is OK. Then I stop the DHCP client script tester and wait some time. (script bellow)

When the DHCP lease timer start expire on this DHCP Server, about 600s
after the first address allocated (it means, start cleaning entries from
/var/lib/neutron/dhcp/xxx/leases file), and when some addresses were
cleaned, then I change the NET VLAN where the DHCP Client tester is
connected (it means, change the DHCP client from VLAN 997 to VLAN 998
network). So, the "randomly" DHCP client requests (60 addresses), are
requesting IP address in the new network. Then, addresses from range
100.98.98.11 to 100.98.98.64 should be allocated to client requests.
Normally it  works.

The problems is, sometimes after this "LAN changing" (change the DHCP client tester from VLAN 997 to VLAN 998), the addresses OFFERED are from a wrong NET. When it happens, the allocated addresses always are from 100.101.1.0/24 network (.11 to .64 too). There is no vROUTER and DHCP Server running with this subnet, like shown bellow:
[root@devel91 (keystone_admin)] openstack subnet list
+--------------------------------------+---------------------------------------------------+--------------------------------------+-------------------+
| ID                                   | Name                                              | Network                              | Subnet            |
+--------------------------------------+---------------------------------------------------+--------------------------------------+-------------------+
| 2ede81d1-9d2d-4df9-9e1e-f0cdf0275fbe | ext-subnet-pub-1                                  | bf2c1841-e09b-4a00-8911-25893ccbf948 | 100.100.0.0/24    |
| 3c3082ce-236e-4ebf-944a-1f0e2b57d309 | vROUTER-997-subnet                                | 3a07cc53-9944-4420-9b76-fa32e41922f5 | 100.97.97.0/24    |
| 4a0adcce-d58e-4b9e-b824-cde0eff3710e | vROUTER-998-subnet                                | 6a8e6738-fc6e-4f06-a714-3f00eb50e3a5 | 100.98.98.0/24    |
| 5bcabde7-41a2-4ce2-b87a-f18c6118d2a0 | HA subnet tenant 56700656a9054052aada06b11465e9d7 | 6bc8e3a4-ee15-4434-8cb8-03b6428add26 | 169.254.192.0/18  |
| 7f7ab506-ab88-457c-9dd0-8852a44f49e7 | HA subnet tenant 0cd6a586a6514322a7ba4df687c3c4c0 | e1fe8a2f-4718-47a5-8b15-3de3e7953def | 169.254.192.0/18  |
| a7483019-5df4-4f86-adde-589b53511f86 | ext-subnet-pub-2                                  | ff831342-d2e1-4351-8703-a2e43cfb0653 | 10.20.30.232/30 |
| a9fda48a-b443-4b94-89e7-d2a9dc7e092a | HA subnet tenant d5c16d8aa5c84fa28ffa7a2d3cef87bf | 1746fba2-1b82-4c4a-92ea-d09c605d5404 | 169.254.192.0/18  |
+--------------------------------------+---------------------------------------------------+--------------------------------------+-------------------+
[root@devel91 (keystone_admin)] 


The problem seems to happen only if some addresses still on leases file from the first network (subnet 100.97.97.0/24) and the same MAC requests a new address without do RELEASE before RENEW. 

How to debug this to find the trigger? 
This is a test scenario, but it happened in real server is use. So, I can do more tests and debugs. 

There is no relevant info on /var/log/neutron/dhcp-agent.log (debug is
ON) or dnsmasq.log.


>>> DHCP Client Discover script (simulates a limited number of client DHCP discovery)
#/sbin/bash
CUSTOMERQTD=200 #This is only a counter for loop test
CUSTOMERCOUNT=1 #Increse 1 customer
echo "Discovery logs on /tmp/dhcptest"
mkdir /tmp/dhcptest 2>/dev/null
cd /tmp/dhcptest
while [ $CUSTOMERCOUNT -le $CUSTOMERQTD ]
do
XX=$(printf "%02X\n" $(shuf -i 11-70 -n 1)) #Generate 60 MAC addresses where only 54 get an address from dnsmasq
sleep 1
dhtest -i em2 -f -m A0:11:$XX:33:44:$XX #-f is a broadcast request
echo "-----------------------------------------"
CUSTOMERCOUNT=$(echo $(($CUSTOMERCOUNT+1)))
done

#DHTEST used is:
#wget https://github.com/saravana815/dhtest/archive/master.zip
#unzip master.zip
#cd dhtest-master/


>>> DNSMASQ config 
/etc/neutron/dnsmasq-neutron.conf 
log-queries 
log-dhcp 
log-facility=/tmp/dnsmasq.log 

dhcp-option-force=26,1500
dhcp-option-force=6,170.231.46.176,8.8.8.8
dhcp-reply-delay=1   

dhcp-range=set:PRIV-192.168.0.0-,192.168.0.10,192.168.0.250,86400s
dhcp-option=tag:PRIV-192.168.0.0-,3,192.168.0.1
dhcp-range=set:PRIV-192.168.1.0-,192.168.1.10,192.168.0.250,86400s
dhcp-option=tag:PRIV-192.168.1.0-,3,192.168.1.1
dhcp-range=set:PRIV-10.0.0.0-,10.0.0.10,10.0.0.250,86400s
dhcp-option=tag:PRIV-10.0.0.0-,3,10.0.0.1
dhcp-range=set:PRIV-10.0.1.0-,10.0.1.10,10.0.1.250,86400s
dhcp-option=tag:PRIV-10.0.1.0-,3,10.0.1.1

dhcp-range=set:infra-70-subnet,100.101.1.11,100.101.1.64,600s
dhcp-option=tag:infra-70-subnet,3,100.101.1.1
dhcp-range=set:infra-71-subnet,100.101.2.11,100.101.2.64,600s
dhcp-option=tag:infra-71-subnet,3,100.101.2.1
dhcp-range=set:infra-72-subnet,100.98.98.11,100.98.98.64,600s
dhcp-option=tag:infra-72-subnet,3,100.98.98.1
dhcp-range=set:infra-73-subnet,100.101.4.11,100.101.4.64,600s
dhcp-option=tag:infra-73-subnet,3,100.101.4.1
dhcp-range=set:infra-74-subnet,100.101.5.11,100.101.5.64,600s
dhcp-option=tag:infra-74-subnet,3,100.101.5.1
dhcp-range=set:infra-75-subnet,100.99.99.11,100.99.99.64,600s
dhcp-option=tag:infra-75-subnet,3,100.99.99.1
dhcp-range=set:infra-76-subnet,100.101.7.11,100.101.7.64,600s
dhcp-option=tag:infra-76-subnet,3,100.101.7.1
dhcp-range=set:infra-77-subnet,100.101.8.11,100.101.8.64,600s
dhcp-option=tag:infra-77-subnet,3,100.101.8.1
dhcp-range=set:infra-78-subnet,100.102.1.11,100.102.1.64,600s
dhcp-option=tag:infra-78-subnet,3,100.102.1.1
dhcp-range=set:infra-79-subnet,100.102.2.11,100.102.2.64,600s
dhcp-option=tag:infra-79-subnet,3,100.102.2.1
dhcp-range=set:infra-80-subnet,100.102.3.11,100.102.3.64,600s
dhcp-option=tag:infra-80-subnet,3,100.102.3.1
dhcp-range=set:infra-81-subnet,100.97.97.11,100.97.97.64,600s
dhcp-option=tag:infra-81-subnet,3,100.97.97.1
dhcp-range=set:infra-82-subnet,100.102.5.11,100.102.5.64,600s
dhcp-option=tag:infra-82-subnet,3,100.102.5.1
dhcp-range=set:infra-83-subnet,100.102.6.11,100.102.6.64,600s
dhcp-option=tag:infra-83-subnet,3,100.102.6.1
dhcp-range=set:infra-84-subnet,100.102.7.11,100.102.7.64,600s
dhcp-option=tag:infra-84-subnet,3,100.102.7.1
dhcp-range=set:infra-85-subnet,100.102.8.11,100.102.8.64,600s
dhcp-option=tag:infra-85-subnet,3,100.102.8.1
dhcp-range=set:infra-86-subnet,100.103.1.11,100.103.1.64,600s
dhcp-option=tag:infra-86-subnet,3,100.103.1.1
dhcp-range=set:infra-87-subnet,100.103.2.11,100.103.2.64,600s
dhcp-option=tag:infra-87-subnet,3,100.103.2.1
dhcp-range=set:infra-88-subnet,100.103.3.11,100.103.3.64,600s
dhcp-option=tag:infra-88-subnet,3,100.103.3.1
dhcp-range=set:infra-89-subnet,100.103.4.11,100.103.4.64,600s
dhcp-option=tag:infra-89-subnet,3,100.103.4.1
dhcp-range=set:infra-90-subnet,100.103.5.11,100.103.5.64,600s
dhcp-option=tag:infra-90-subnet,3,100.103.5.1
dhcp-range=set:infra-91-subnet,100.103.6.11,100.103.6.64,600s
dhcp-option=tag:infra-91-subnet,3,100.103.6.1
dhcp-range=set:infra-92-subnet,100.103.7.11,100.103.7.64,600s
dhcp-option=tag:infra-92-subnet,3,100.103.7.1
dhcp-range=set:infra-93-subnet,100.103.8.11,100.103.8.64,600s
dhcp-option=tag:infra-93-subnet,3,100.103.8.1
dhcp-range=set:infra-94-subnet,100.104.1.11,100.104.1.64,600s
dhcp-option=tag:infra-94-subnet,3,100.104.1.1
dhcp-range=set:infra-95-subnet,100.96.96.11,100.96.96.64,600s
dhcp-option=tag:infra-95-subnet,3,100.96.96.1


>>>  dnsmasq.log when the first problem happens.
Nov 26 11:12:06 dnsmasq-dhcp[3975]: 1712147283 DHCPDISCOVER(tape56ef56c-66) a0:11:2f:33:44:2f 
Nov 26 11:12:06 dnsmasq-dhcp[3975]: 1712147283 tags: infra-81-subnet, tape56ef56c-66
Nov 26 11:12:06 dnsmasq-dhcp[3975]: 1712147283 reply delay: 1
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 DHCPOFFER(tape56ef56c-66) 100.97.97.13 a0:11:2f:33:44:2f 
Nov 26 11:12:07 dnsmasq-dhcp[3975]: Ignoring duplicate dhcp-option 26
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 requested options: 1:netmask, 28:broadcast, 3:router, 15:domain-name, 
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 requested options: 6:dns-server
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 next server: 100.97.97.3
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 broadcast response
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 sent size:  1 option: 53 message-type  2
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 sent size:  4 option: 54 server-identifier  100.97.97.3
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 sent size:  4 option: 51 lease-time  10m
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 sent size:  4 option: 58 T1  5m
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 sent size:  4 option: 59 T2  8m45s
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 sent size:  4 option:  1 netmask  255.255.255.0
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 sent size:  4 option: 28 broadcast  100.97.97.255
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 sent size: 14 option: 15 domain-name  openstacklocal
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 sent size:  4 option:  3 router  100.97.97.1
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 sent size:  8 option:  6 dns-server  170.231.46.176, 8.8.8.8
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 sent size:  2 option: 26 mtu  1500
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP subnet: 100.97.97.0/255.255.255.0
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.101.1.11 -- 100.101.1.64
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.101.2.11 -- 100.101.2.64
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.98.98.11 -- 100.98.98.64
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.101.4.11 -- 100.101.4.64
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.101.5.11 -- 100.101.5.64
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.99.99.11 -- 100.99.99.64
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.101.7.11 -- 100.101.7.64
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.101.8.11 -- 100.101.8.64
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.102.1.11 -- 100.102.1.64
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.102.2.11 -- 100.102.2.64
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.102.3.11 -- 100.102.3.64
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.97.97.11 -- 100.97.97.64
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.102.5.11 -- 100.102.5.64
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.102.6.11 -- 100.102.6.64
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.102.7.11 -- 100.102.7.64
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.102.8.11 -- 100.102.8.64
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.103.1.11 -- 100.103.1.64
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.103.2.11 -- 100.103.2.64
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.103.3.11 -- 100.103.3.64
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.103.4.11 -- 100.103.4.64
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.103.5.11 -- 100.103.5.64
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.103.6.11 -- 100.103.6.64
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.103.7.11 -- 100.103.7.64
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.103.8.11 -- 100.103.8.64
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.104.1.11 -- 100.104.1.64
Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.96.96.11 -- 100.96.96.64
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 DHCPDISCOVER(tape56ef56c-66) a0:11:13:33:44:13 
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 tags: infra-70-subnet, tape56ef56c-66
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 reply delay: 1
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 DHCPOFFER(tape56ef56c-66) 100.101.1.63 a0:11:13:33:44:13 
Nov 26 11:12:10 dnsmasq-dhcp[3975]: Ignoring duplicate dhcp-option 26
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 requested options: 1:netmask, 28:broadcast, 3:router, 15:domain-name, 
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 requested options: 6:dns-server
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 next server: 100.97.97.3
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 broadcast response
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 sent size:  1 option: 53 message-type  2
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 sent size:  4 option: 54 server-identifier  100.97.97.3
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 sent size:  4 option: 51 lease-time  10m
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 sent size:  4 option: 58 T1  5m
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 sent size:  4 option: 59 T2  8m45s
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 sent size:  4 option:  1 netmask  255.0.0.0
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 sent size:  4 option: 28 broadcast  100.97.97.255
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 sent size: 14 option: 15 domain-name  openstacklocal
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 sent size:  4 option:  3 router  100.101.1.1
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 sent size:  8 option:  6 dns-server  170.231.46.176, 8.8.8.8
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 sent size:  2 option: 26 mtu  1500
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 1712147283 available DHCP subnet: 100.97.97.0/255.255.255.0
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 1712147283 available DHCP range: 100.101.1.11 -- 100.101.1.64
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 1712147283 available DHCP range: 100.101.2.11 -- 100.101.2.64
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 1712147283 available DHCP range: 100.98.98.11 -- 100.98.98.64
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 1712147283 available DHCP range: 100.101.4.11 -- 100.101.4.64
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 1712147283 available DHCP range: 100.101.5.11 -- 100.101.5.64
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 1712147283 available DHCP range: 100.99.99.11 -- 100.99.99.64
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 1712147283 available DHCP range: 100.101.7.11 -- 100.101.7.64
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 1712147283 available DHCP range: 100.101.8.11 -- 100.101.8.64
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 1712147283 available DHCP range: 100.102.1.11 -- 100.102.1.64
Nov 26 11:12:10 dnsmasq-dhcp[3975]: 1712147283 available DHCP range: 100.102.2.11 -- 100.102.2.64


The scenario:
- Install Controller+Network with packstack (Tested on Mitaka, Pike and Queens)
- phys_net2 is mapped to access interface where VLAN 997 and 998 
- Create neutron Networks
#External/Uplink side (WAN for each Virtual Router)
openstack network create ext-net-pub-1 --external --provider-network-type flat --provider-physical-network phys_ext --availability-zone-hint int6
openstack subnet create ext-subnet-pub-1 --network ext-net-pub-1 --subnet-range 100.100.0.0/24 --allocation-pool start=100.100.0.5,end=100.100.0.9
#Virtual Router1
openstack project create vROUTER-997
openstack router create vROUTER-997 --availability-zone-hint int6 --project=vROUTER-997
openstack network create vROUTER-997-net --external --provider-network-type vlan --provider-physical-network phys_net2 --provider-segment 997 --availability-zone-hint int6 --project=vROUTER-997
openstack subnet create vROUTER-997-subnet --network vROUTER-997-net --subnet-range 100.97.97.0/24 --allocation-pool start=100.97.97.2,end=100.97.97.10
openstack router add subnet vROUTER-997 vROUTER-997-subnet
openstack router set --external-gateway ext-net-pub-1 vROUTER-997
#Virtual Router2
openstack project create vROUTER-998
openstack router create vROUTER-998 --availability-zone-hint int6 --project=vROUTER-998
openstack network create vROUTER-998-net --external --provider-network-type vlan --provider-physical-network phys_net2 --provider-segment 998 --availability-zone-hint int6 --project=vROUTER-998
openstack subnet create vROUTER-998-subnet --network vROUTER-998-net --subnet-range 100.98.98.0/24 --allocation-pool start=100.98.98.2,end=100.98.98.10
openstack router add subnet vROUTER-998 vROUTER-998-subnet
openstack router set --external-gateway ext-net-pub-1 vROUTER-998
- Configure DNSMASQ with config posted above
- Start DHCP Client script on VLAN 997, wait for some time to lease timeout start clear addresses on /var/lib/neutron/dhcp/xxx/leases file, then change script to run on VLAN 998 (you need a switch connected to phys_net2 or use VLAN TAG on DHCP client tester). Normally I need to swap the tester some times between these VLANs. The problem is get addresses out of range 100.97.97.0/24 and 100.98.98.0/24. 


Also posted on: http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2018q4/012639.html

Thanks any help
Luis

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: dhcp dnsmasq neutron

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1807382

Title:
  DNSMASQ wrong addresses allocated after changing DHCP Clients between
  Neutron vRouters NET

Status in neutron:
  New

Bug description:
  I'm using OpenStack, basically creating Virtual Routers with
  net/subnets, and DNSMASQ is the DHCP Server for them (via neutron-dhcp
  driver).

  My test scenario contains 2 virtual routers (2 namespaces as router) with 1 DHCP each (2 namespaces as dhcp servers) on the Neutron Network Server. The Subnet are 100.97.97.0/24 for NET with VLAN 997, and Subnet 100.98.98.0/24 for Network with VLAN 998. Both are on the same physical interface (BR-NET2). 
  The 2 DHCP servers have available .11 to .64 IP address range (54 addresses). (dnsmasq config file is bellow)
  I'm generation 60 MAC addresses randomly as source MAC address to test DHCP Client requests. Theses 60 addresses are always the same. 

  
  The DHCP Client script tester is connected to Network with VLAN 997 and subnet 100.97.97.0/24. I generate DHCP client MAC addresses and only 54 are allocated, as expected.  So, addresses from range 100.97.97.11 to 100.97.97.64 are allocated to client requests. This is OK. Then I stop the DHCP client script tester and wait some time. (script bellow)

  When the DHCP lease timer start expire on this DHCP Server, about 600s
  after the first address allocated (it means, start cleaning entries
  from /var/lib/neutron/dhcp/xxx/leases file), and when some addresses
  were cleaned, then I change the NET VLAN where the DHCP Client tester
  is connected (it means, change the DHCP client from VLAN 997 to VLAN
  998 network). So, the "randomly" DHCP client requests (60 addresses),
  are requesting IP address in the new network. Then, addresses from
  range 100.98.98.11 to 100.98.98.64 should be allocated to client
  requests. Normally it  works.

  The problems is, sometimes after this "LAN changing" (change the DHCP client tester from VLAN 997 to VLAN 998), the addresses OFFERED are from a wrong NET. When it happens, the allocated addresses always are from 100.101.1.0/24 network (.11 to .64 too). There is no vROUTER and DHCP Server running with this subnet, like shown bellow:
  [root@devel91 (keystone_admin)] openstack subnet list
  +--------------------------------------+---------------------------------------------------+--------------------------------------+-------------------+
  | ID                                   | Name                                              | Network                              | Subnet            |
  +--------------------------------------+---------------------------------------------------+--------------------------------------+-------------------+
  | 2ede81d1-9d2d-4df9-9e1e-f0cdf0275fbe | ext-subnet-pub-1                                  | bf2c1841-e09b-4a00-8911-25893ccbf948 | 100.100.0.0/24    |
  | 3c3082ce-236e-4ebf-944a-1f0e2b57d309 | vROUTER-997-subnet                                | 3a07cc53-9944-4420-9b76-fa32e41922f5 | 100.97.97.0/24    |
  | 4a0adcce-d58e-4b9e-b824-cde0eff3710e | vROUTER-998-subnet                                | 6a8e6738-fc6e-4f06-a714-3f00eb50e3a5 | 100.98.98.0/24    |
  | 5bcabde7-41a2-4ce2-b87a-f18c6118d2a0 | HA subnet tenant 56700656a9054052aada06b11465e9d7 | 6bc8e3a4-ee15-4434-8cb8-03b6428add26 | 169.254.192.0/18  |
  | 7f7ab506-ab88-457c-9dd0-8852a44f49e7 | HA subnet tenant 0cd6a586a6514322a7ba4df687c3c4c0 | e1fe8a2f-4718-47a5-8b15-3de3e7953def | 169.254.192.0/18  |
  | a7483019-5df4-4f86-adde-589b53511f86 | ext-subnet-pub-2                                  | ff831342-d2e1-4351-8703-a2e43cfb0653 | 10.20.30.232/30 |
  | a9fda48a-b443-4b94-89e7-d2a9dc7e092a | HA subnet tenant d5c16d8aa5c84fa28ffa7a2d3cef87bf | 1746fba2-1b82-4c4a-92ea-d09c605d5404 | 169.254.192.0/18  |
  +--------------------------------------+---------------------------------------------------+--------------------------------------+-------------------+
  [root@devel91 (keystone_admin)] 

  
  The problem seems to happen only if some addresses still on leases file from the first network (subnet 100.97.97.0/24) and the same MAC requests a new address without do RELEASE before RENEW. 

  How to debug this to find the trigger? 
  This is a test scenario, but it happened in real server is use. So, I can do more tests and debugs. 

  There is no relevant info on /var/log/neutron/dhcp-agent.log (debug is
  ON) or dnsmasq.log.

  
  >>> DHCP Client Discover script (simulates a limited number of client DHCP discovery)
  #/sbin/bash
  CUSTOMERQTD=200 #This is only a counter for loop test
  CUSTOMERCOUNT=1 #Increse 1 customer
  echo "Discovery logs on /tmp/dhcptest"
  mkdir /tmp/dhcptest 2>/dev/null
  cd /tmp/dhcptest
  while [ $CUSTOMERCOUNT -le $CUSTOMERQTD ]
  do
  XX=$(printf "%02X\n" $(shuf -i 11-70 -n 1)) #Generate 60 MAC addresses where only 54 get an address from dnsmasq
  sleep 1
  dhtest -i em2 -f -m A0:11:$XX:33:44:$XX #-f is a broadcast request
  echo "-----------------------------------------"
  CUSTOMERCOUNT=$(echo $(($CUSTOMERCOUNT+1)))
  done

  #DHTEST used is:
  #wget https://github.com/saravana815/dhtest/archive/master.zip
  #unzip master.zip
  #cd dhtest-master/

  
  >>> DNSMASQ config 
  /etc/neutron/dnsmasq-neutron.conf 
  log-queries 
  log-dhcp 
  log-facility=/tmp/dnsmasq.log 

  dhcp-option-force=26,1500
  dhcp-option-force=6,170.231.46.176,8.8.8.8
  dhcp-reply-delay=1   

  dhcp-range=set:PRIV-192.168.0.0-,192.168.0.10,192.168.0.250,86400s
  dhcp-option=tag:PRIV-192.168.0.0-,3,192.168.0.1
  dhcp-range=set:PRIV-192.168.1.0-,192.168.1.10,192.168.0.250,86400s
  dhcp-option=tag:PRIV-192.168.1.0-,3,192.168.1.1
  dhcp-range=set:PRIV-10.0.0.0-,10.0.0.10,10.0.0.250,86400s
  dhcp-option=tag:PRIV-10.0.0.0-,3,10.0.0.1
  dhcp-range=set:PRIV-10.0.1.0-,10.0.1.10,10.0.1.250,86400s
  dhcp-option=tag:PRIV-10.0.1.0-,3,10.0.1.1

  dhcp-range=set:infra-70-subnet,100.101.1.11,100.101.1.64,600s
  dhcp-option=tag:infra-70-subnet,3,100.101.1.1
  dhcp-range=set:infra-71-subnet,100.101.2.11,100.101.2.64,600s
  dhcp-option=tag:infra-71-subnet,3,100.101.2.1
  dhcp-range=set:infra-72-subnet,100.98.98.11,100.98.98.64,600s
  dhcp-option=tag:infra-72-subnet,3,100.98.98.1
  dhcp-range=set:infra-73-subnet,100.101.4.11,100.101.4.64,600s
  dhcp-option=tag:infra-73-subnet,3,100.101.4.1
  dhcp-range=set:infra-74-subnet,100.101.5.11,100.101.5.64,600s
  dhcp-option=tag:infra-74-subnet,3,100.101.5.1
  dhcp-range=set:infra-75-subnet,100.99.99.11,100.99.99.64,600s
  dhcp-option=tag:infra-75-subnet,3,100.99.99.1
  dhcp-range=set:infra-76-subnet,100.101.7.11,100.101.7.64,600s
  dhcp-option=tag:infra-76-subnet,3,100.101.7.1
  dhcp-range=set:infra-77-subnet,100.101.8.11,100.101.8.64,600s
  dhcp-option=tag:infra-77-subnet,3,100.101.8.1
  dhcp-range=set:infra-78-subnet,100.102.1.11,100.102.1.64,600s
  dhcp-option=tag:infra-78-subnet,3,100.102.1.1
  dhcp-range=set:infra-79-subnet,100.102.2.11,100.102.2.64,600s
  dhcp-option=tag:infra-79-subnet,3,100.102.2.1
  dhcp-range=set:infra-80-subnet,100.102.3.11,100.102.3.64,600s
  dhcp-option=tag:infra-80-subnet,3,100.102.3.1
  dhcp-range=set:infra-81-subnet,100.97.97.11,100.97.97.64,600s
  dhcp-option=tag:infra-81-subnet,3,100.97.97.1
  dhcp-range=set:infra-82-subnet,100.102.5.11,100.102.5.64,600s
  dhcp-option=tag:infra-82-subnet,3,100.102.5.1
  dhcp-range=set:infra-83-subnet,100.102.6.11,100.102.6.64,600s
  dhcp-option=tag:infra-83-subnet,3,100.102.6.1
  dhcp-range=set:infra-84-subnet,100.102.7.11,100.102.7.64,600s
  dhcp-option=tag:infra-84-subnet,3,100.102.7.1
  dhcp-range=set:infra-85-subnet,100.102.8.11,100.102.8.64,600s
  dhcp-option=tag:infra-85-subnet,3,100.102.8.1
  dhcp-range=set:infra-86-subnet,100.103.1.11,100.103.1.64,600s
  dhcp-option=tag:infra-86-subnet,3,100.103.1.1
  dhcp-range=set:infra-87-subnet,100.103.2.11,100.103.2.64,600s
  dhcp-option=tag:infra-87-subnet,3,100.103.2.1
  dhcp-range=set:infra-88-subnet,100.103.3.11,100.103.3.64,600s
  dhcp-option=tag:infra-88-subnet,3,100.103.3.1
  dhcp-range=set:infra-89-subnet,100.103.4.11,100.103.4.64,600s
  dhcp-option=tag:infra-89-subnet,3,100.103.4.1
  dhcp-range=set:infra-90-subnet,100.103.5.11,100.103.5.64,600s
  dhcp-option=tag:infra-90-subnet,3,100.103.5.1
  dhcp-range=set:infra-91-subnet,100.103.6.11,100.103.6.64,600s
  dhcp-option=tag:infra-91-subnet,3,100.103.6.1
  dhcp-range=set:infra-92-subnet,100.103.7.11,100.103.7.64,600s
  dhcp-option=tag:infra-92-subnet,3,100.103.7.1
  dhcp-range=set:infra-93-subnet,100.103.8.11,100.103.8.64,600s
  dhcp-option=tag:infra-93-subnet,3,100.103.8.1
  dhcp-range=set:infra-94-subnet,100.104.1.11,100.104.1.64,600s
  dhcp-option=tag:infra-94-subnet,3,100.104.1.1
  dhcp-range=set:infra-95-subnet,100.96.96.11,100.96.96.64,600s
  dhcp-option=tag:infra-95-subnet,3,100.96.96.1


  >>>  dnsmasq.log when the first problem happens.
  Nov 26 11:12:06 dnsmasq-dhcp[3975]: 1712147283 DHCPDISCOVER(tape56ef56c-66) a0:11:2f:33:44:2f 
  Nov 26 11:12:06 dnsmasq-dhcp[3975]: 1712147283 tags: infra-81-subnet, tape56ef56c-66
  Nov 26 11:12:06 dnsmasq-dhcp[3975]: 1712147283 reply delay: 1
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 DHCPOFFER(tape56ef56c-66) 100.97.97.13 a0:11:2f:33:44:2f 
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: Ignoring duplicate dhcp-option 26
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 requested options: 1:netmask, 28:broadcast, 3:router, 15:domain-name, 
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 requested options: 6:dns-server
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 next server: 100.97.97.3
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 broadcast response
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 sent size:  1 option: 53 message-type  2
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 sent size:  4 option: 54 server-identifier  100.97.97.3
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 sent size:  4 option: 51 lease-time  10m
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 sent size:  4 option: 58 T1  5m
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 sent size:  4 option: 59 T2  8m45s
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 sent size:  4 option:  1 netmask  255.255.255.0
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 sent size:  4 option: 28 broadcast  100.97.97.255
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 sent size: 14 option: 15 domain-name  openstacklocal
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 sent size:  4 option:  3 router  100.97.97.1
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 sent size:  8 option:  6 dns-server  170.231.46.176, 8.8.8.8
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 1712147283 sent size:  2 option: 26 mtu  1500
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP subnet: 100.97.97.0/255.255.255.0
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.101.1.11 -- 100.101.1.64
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.101.2.11 -- 100.101.2.64
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.98.98.11 -- 100.98.98.64
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.101.4.11 -- 100.101.4.64
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.101.5.11 -- 100.101.5.64
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.99.99.11 -- 100.99.99.64
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.101.7.11 -- 100.101.7.64
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.101.8.11 -- 100.101.8.64
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.102.1.11 -- 100.102.1.64
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.102.2.11 -- 100.102.2.64
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.102.3.11 -- 100.102.3.64
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.97.97.11 -- 100.97.97.64
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.102.5.11 -- 100.102.5.64
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.102.6.11 -- 100.102.6.64
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.102.7.11 -- 100.102.7.64
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.102.8.11 -- 100.102.8.64
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.103.1.11 -- 100.103.1.64
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.103.2.11 -- 100.103.2.64
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.103.3.11 -- 100.103.3.64
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.103.4.11 -- 100.103.4.64
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.103.5.11 -- 100.103.5.64
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.103.6.11 -- 100.103.6.64
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.103.7.11 -- 100.103.7.64
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.103.8.11 -- 100.103.8.64
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.104.1.11 -- 100.104.1.64
  Nov 26 11:12:07 dnsmasq-dhcp[3975]: 435880880 available DHCP range: 100.96.96.11 -- 100.96.96.64
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 DHCPDISCOVER(tape56ef56c-66) a0:11:13:33:44:13 
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 tags: infra-70-subnet, tape56ef56c-66
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 reply delay: 1
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 DHCPOFFER(tape56ef56c-66) 100.101.1.63 a0:11:13:33:44:13 
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: Ignoring duplicate dhcp-option 26
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 requested options: 1:netmask, 28:broadcast, 3:router, 15:domain-name, 
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 requested options: 6:dns-server
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 next server: 100.97.97.3
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 broadcast response
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 sent size:  1 option: 53 message-type  2
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 sent size:  4 option: 54 server-identifier  100.97.97.3
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 sent size:  4 option: 51 lease-time  10m
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 sent size:  4 option: 58 T1  5m
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 sent size:  4 option: 59 T2  8m45s
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 sent size:  4 option:  1 netmask  255.0.0.0
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 sent size:  4 option: 28 broadcast  100.97.97.255
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 sent size: 14 option: 15 domain-name  openstacklocal
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 sent size:  4 option:  3 router  100.101.1.1
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 sent size:  8 option:  6 dns-server  170.231.46.176, 8.8.8.8
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 435880880 sent size:  2 option: 26 mtu  1500
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 1712147283 available DHCP subnet: 100.97.97.0/255.255.255.0
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 1712147283 available DHCP range: 100.101.1.11 -- 100.101.1.64
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 1712147283 available DHCP range: 100.101.2.11 -- 100.101.2.64
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 1712147283 available DHCP range: 100.98.98.11 -- 100.98.98.64
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 1712147283 available DHCP range: 100.101.4.11 -- 100.101.4.64
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 1712147283 available DHCP range: 100.101.5.11 -- 100.101.5.64
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 1712147283 available DHCP range: 100.99.99.11 -- 100.99.99.64
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 1712147283 available DHCP range: 100.101.7.11 -- 100.101.7.64
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 1712147283 available DHCP range: 100.101.8.11 -- 100.101.8.64
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 1712147283 available DHCP range: 100.102.1.11 -- 100.102.1.64
  Nov 26 11:12:10 dnsmasq-dhcp[3975]: 1712147283 available DHCP range: 100.102.2.11 -- 100.102.2.64


  The scenario:
  - Install Controller+Network with packstack (Tested on Mitaka, Pike and Queens)
  - phys_net2 is mapped to access interface where VLAN 997 and 998 
  - Create neutron Networks
  #External/Uplink side (WAN for each Virtual Router)
  openstack network create ext-net-pub-1 --external --provider-network-type flat --provider-physical-network phys_ext --availability-zone-hint int6
  openstack subnet create ext-subnet-pub-1 --network ext-net-pub-1 --subnet-range 100.100.0.0/24 --allocation-pool start=100.100.0.5,end=100.100.0.9
  #Virtual Router1
  openstack project create vROUTER-997
  openstack router create vROUTER-997 --availability-zone-hint int6 --project=vROUTER-997
  openstack network create vROUTER-997-net --external --provider-network-type vlan --provider-physical-network phys_net2 --provider-segment 997 --availability-zone-hint int6 --project=vROUTER-997
  openstack subnet create vROUTER-997-subnet --network vROUTER-997-net --subnet-range 100.97.97.0/24 --allocation-pool start=100.97.97.2,end=100.97.97.10
  openstack router add subnet vROUTER-997 vROUTER-997-subnet
  openstack router set --external-gateway ext-net-pub-1 vROUTER-997
  #Virtual Router2
  openstack project create vROUTER-998
  openstack router create vROUTER-998 --availability-zone-hint int6 --project=vROUTER-998
  openstack network create vROUTER-998-net --external --provider-network-type vlan --provider-physical-network phys_net2 --provider-segment 998 --availability-zone-hint int6 --project=vROUTER-998
  openstack subnet create vROUTER-998-subnet --network vROUTER-998-net --subnet-range 100.98.98.0/24 --allocation-pool start=100.98.98.2,end=100.98.98.10
  openstack router add subnet vROUTER-998 vROUTER-998-subnet
  openstack router set --external-gateway ext-net-pub-1 vROUTER-998
  - Configure DNSMASQ with config posted above
  - Start DHCP Client script on VLAN 997, wait for some time to lease timeout start clear addresses on /var/lib/neutron/dhcp/xxx/leases file, then change script to run on VLAN 998 (you need a switch connected to phys_net2 or use VLAN TAG on DHCP client tester). Normally I need to swap the tester some times between these VLANs. The problem is get addresses out of range 100.97.97.0/24 and 100.98.98.0/24. 

  
  Also posted on: http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2018q4/012639.html

  Thanks any help
  Luis

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1807382/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next