yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1896945] [NEW] dnsmasq >= 2.81 not responding to DHCP requests with current q-dhcp configs

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Lee Yarwood <1896945@xxxxxxxxxxxxxxxxxx>
Date: Thu, 24 Sep 2020 12:14:16 -0000
Reply-to: Bug 1896945 <1896945@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

* High level description:

I've been attempting to enable Fedora 32 support in devstack and
encountered the following issue where dnsmasq as configured by q-dhcp
isn't responding to DHCP requests from clients:

https://review.opendev.org/#/c/750292/

Looking at tcpdump and strace it appears that dnsmasq can see the
requests but doesn't reply suggesting a configuration issue either
caused by q-dhcp *or* a regression in dnsmasq itself:

$ openstack server reboot --hard test && sudo ip netns exec qdhcp-df64061e-0784-4bbe-909b-ae1c5f466981 tcpdump -i tapee679459-e1 -n port 67 or port 68
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tapee679459-e1, link-type EN10MB (Ethernet), capture size 262144 bytes
18:40:24.070796 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:16:07:a0, length 300
18:41:24.118961 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:16:07:a0, length 300
18:42:24.192716 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:16:07:a0, length 300

$ openstack server reboot --hard test && sudo ip netns exec qdhcp-df64061e-0784-4bbe-909b-ae1c5f466981 strace -p 196856
strace: Process 196856 attached
restart_syscall(<... resuming interrupted read ...>) = 1
recvmsg(4, {msg_name={sa_family=AF_INET, sin_port=htons(68), sin_addr=inet_addr("0.0.0.0")}, msg_namelen=16, msg_iov=[{iov_base="\1\1\6\0\0041\326S\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\372\26>\26"..., iov_len=548}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_IP, cmsg_type=IP_PKTINFO, cmsg_data={ipi_ifindex=if_nametoindex("tapee679459-e1"), ipi_spec_dst=inet_addr("10.0.0.2"), ipi_addr=inet_addr("255.255.255.255")}}], msg_controllen=32, msg_flags=0}, MSG_PEEK|MSG_TRUNC) = 300
recvmsg(4, {msg_name={sa_family=AF_INET, sin_port=htons(68), sin_addr=inet_addr("0.0.0.0")}, msg_namelen=16, msg_iov=[{iov_base="\1\1\6\0\0041\326S\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\372\26>\26"..., iov_len=548}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_IP, cmsg_type=IP_PKTINFO, cmsg_data={ipi_ifindex=if_nametoindex("tapee679459-e1"), ipi_spec_dst=inet_addr("10.0.0.2"), ipi_addr=inet_addr("255.255.255.255")}}], msg_controllen=32, msg_flags=0}, 0) = 300
ioctl(4, SIOCGIFNAME, {ifr_index=9, ifr_name="tapee679459-e1"}) = 0
ioctl(4, SIOCGIFFLAGS, {ifr_name="tapee679459-e1", ifr_flags=IFF_UP|IFF_BROADCAST|IFF_RUNNING|IFF_MULTICAST}) = 0
ioctl(4, SIOCGIFADDR, {ifr_name="tapee679459-e1", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("10.0.0.2")}}) = 0
poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6, events=POLLIN}, {fd=7, events=POLLIN}, {fd=9, events=POLLIN}, {fd=10, events=POLLIN}, {fd=11, events=POLLIN}, {fd=14, events=POLLIN}], 8, -1

The current configs are listed below:

http://paste.openstack.org/show/798334/

I was able to downgrade dnsmasq on f32 to 2.80 in order to workaround
this:

$ sudo dnf downgrade dnsmasq -y
[..]
$ rpm -qa | grep dnsmasq
dnsmasq-2.80-14.fc32.x86_64
$ sudo killall dnsmasq && sudo systemctl restart devstack@q-*
$ openstack server reboot --hard test && sudo ip netns exec qdhcp-df64061e-0784-4bbe-909b-ae1c5f466981 tcpdump -i tapee679459-e1 -n port 67 or port 68
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tapee679459-e1, link-type EN10MB (Ethernet), capture size 262144 bytes
12:06:57.028953 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:16:07:a0, length 300
12:06:57.029994 IP 10.0.0.2.bootps > 10.0.0.49.bootpc: BOOTP/DHCP, Reply, length 328
12:06:57.042300 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:16:07:a0, length 300
12:06:57.047014 IP 10.0.0.2.bootps > 10.0.0.49.bootpc: BOOTP/DHCP, Reply, length 344

* Pre-conditions:

F32 with dnsmasq >= 2.81 installed.

* Step-by-step reproduction steps:

Deploy F32 with dnsmasq >= 2.81 installed, attempt to spawn an instance
attached to a subnet with dhcp enabled.

* Expected output:

dnsmasq responds to DHCP request from instance.

* Actual output:

dnsmasq doesn't respond to DHCP request from instance.

* Version:
  ** OpenStack version (Specific stable branch, or git hash if from trunk);

  Neutron @ 0fdcc4b1b63dc90fbc9f46f5947f84626f8e5b41

  ** Linux distro, kernel. For a distro, it’s also worth knowing
specific versions of client and server;

  Fedora 32 with kernel 5.8.10-200.fc32.x86_64

  ** DevStack or other _deployment_ mechanism?

  Devstack @ https://review.opendev.org/#/c/750292/

* Environment: what types of services are you running (core services
like DB and AMQP broker, as well as Nova/hypervisor if it matters), and
which type of deployment (clustered servers)? Multi-node or single node,
etc.

  Single node devstack env.

* Perceived severity: is this a blocker for you?

  High, assuming other distros will be impacted by this once they move
to dnsmasq >= 2.81

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: l3-ipam-dhcp

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1896945

Title:
  dnsmasq >= 2.81 not responding to DHCP requests with current q-dhcp
  configs

Status in neutron:
  New

Bug description:
  * High level description:

  I've been attempting to enable Fedora 32 support in devstack and
  encountered the following issue where dnsmasq as configured by q-dhcp
  isn't responding to DHCP requests from clients:

  https://review.opendev.org/#/c/750292/

  Looking at tcpdump and strace it appears that dnsmasq can see the
  requests but doesn't reply suggesting a configuration issue either
  caused by q-dhcp *or* a regression in dnsmasq itself:

  $ openstack server reboot --hard test && sudo ip netns exec qdhcp-df64061e-0784-4bbe-909b-ae1c5f466981 tcpdump -i tapee679459-e1 -n port 67 or port 68
  dropped privs to tcpdump
  tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
  listening on tapee679459-e1, link-type EN10MB (Ethernet), capture size 262144 bytes
  18:40:24.070796 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:16:07:a0, length 300
  18:41:24.118961 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:16:07:a0, length 300
  18:42:24.192716 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:16:07:a0, length 300

  $ openstack server reboot --hard test && sudo ip netns exec qdhcp-df64061e-0784-4bbe-909b-ae1c5f466981 strace -p 196856
  strace: Process 196856 attached
  restart_syscall(<... resuming interrupted read ...>) = 1
  recvmsg(4, {msg_name={sa_family=AF_INET, sin_port=htons(68), sin_addr=inet_addr("0.0.0.0")}, msg_namelen=16, msg_iov=[{iov_base="\1\1\6\0\0041\326S\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\372\26>\26"..., iov_len=548}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_IP, cmsg_type=IP_PKTINFO, cmsg_data={ipi_ifindex=if_nametoindex("tapee679459-e1"), ipi_spec_dst=inet_addr("10.0.0.2"), ipi_addr=inet_addr("255.255.255.255")}}], msg_controllen=32, msg_flags=0}, MSG_PEEK|MSG_TRUNC) = 300
  recvmsg(4, {msg_name={sa_family=AF_INET, sin_port=htons(68), sin_addr=inet_addr("0.0.0.0")}, msg_namelen=16, msg_iov=[{iov_base="\1\1\6\0\0041\326S\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\372\26>\26"..., iov_len=548}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_IP, cmsg_type=IP_PKTINFO, cmsg_data={ipi_ifindex=if_nametoindex("tapee679459-e1"), ipi_spec_dst=inet_addr("10.0.0.2"), ipi_addr=inet_addr("255.255.255.255")}}], msg_controllen=32, msg_flags=0}, 0) = 300
  ioctl(4, SIOCGIFNAME, {ifr_index=9, ifr_name="tapee679459-e1"}) = 0
  ioctl(4, SIOCGIFFLAGS, {ifr_name="tapee679459-e1", ifr_flags=IFF_UP|IFF_BROADCAST|IFF_RUNNING|IFF_MULTICAST}) = 0
  ioctl(4, SIOCGIFADDR, {ifr_name="tapee679459-e1", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("10.0.0.2")}}) = 0
  poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6, events=POLLIN}, {fd=7, events=POLLIN}, {fd=9, events=POLLIN}, {fd=10, events=POLLIN}, {fd=11, events=POLLIN}, {fd=14, events=POLLIN}], 8, -1

  The current configs are listed below:

  http://paste.openstack.org/show/798334/

  I was able to downgrade dnsmasq on f32 to 2.80 in order to workaround
  this:

  $ sudo dnf downgrade dnsmasq -y
  [..]
  $ rpm -qa | grep dnsmasq
  dnsmasq-2.80-14.fc32.x86_64
  $ sudo killall dnsmasq && sudo systemctl restart devstack@q-*
  $ openstack server reboot --hard test && sudo ip netns exec qdhcp-df64061e-0784-4bbe-909b-ae1c5f466981 tcpdump -i tapee679459-e1 -n port 67 or port 68
  dropped privs to tcpdump
  tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
  listening on tapee679459-e1, link-type EN10MB (Ethernet), capture size 262144 bytes
  12:06:57.028953 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:16:07:a0, length 300
  12:06:57.029994 IP 10.0.0.2.bootps > 10.0.0.49.bootpc: BOOTP/DHCP, Reply, length 328
  12:06:57.042300 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:16:07:a0, length 300
  12:06:57.047014 IP 10.0.0.2.bootps > 10.0.0.49.bootpc: BOOTP/DHCP, Reply, length 344

  * Pre-conditions:

  F32 with dnsmasq >= 2.81 installed.

  * Step-by-step reproduction steps:

  Deploy F32 with dnsmasq >= 2.81 installed, attempt to spawn an
  instance attached to a subnet with dhcp enabled.

  * Expected output:

  dnsmasq responds to DHCP request from instance.

  * Actual output:

  dnsmasq doesn't respond to DHCP request from instance.

  * Version:
    ** OpenStack version (Specific stable branch, or git hash if from trunk);

    Neutron @ 0fdcc4b1b63dc90fbc9f46f5947f84626f8e5b41

    ** Linux distro, kernel. For a distro, it’s also worth knowing
  specific versions of client and server;

    Fedora 32 with kernel 5.8.10-200.fc32.x86_64

    ** DevStack or other _deployment_ mechanism?

    Devstack @ https://review.opendev.org/#/c/750292/

  * Environment: what types of services are you running (core services
  like DB and AMQP broker, as well as Nova/hypervisor if it matters),
  and which type of deployment (clustered servers)? Multi-node or single
  node, etc.

    Single node devstack env.

  * Perceived severity: is this a blocker for you?

    High, assuming other distros will be impacted by this once they move
  to dnsmasq >= 2.81

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1896945/+subscriptions
Follow ups

[Bug 1896945] Re: dnsmasq >= 2.81 not responding to DHCP requests with current q-dhcp configs
From: OpenStack Infra, 2020-10-06