debcrafters-packages team mailing list archive

[Lukas Märdian <2119652@xxxxxxxxxxxxxxxxxx>]

I've had a debugging session with @cpaelzer earlier today, where we ran
into some "Temporary failure in name resolution" issues inside a KVM
guest (for all - even global - domains) and "resolvectl dnssec enp1s0
false" made it work, which was scary. – But we fiddled quite a bit with
the host networking, too.


Trying to reproduce this libvirt/KVM issue on a clean Noble host and Questing guest, everything worked as expected.

In libvirt/Qemu we don't have a private zone defined:
$ cat /etc/resolv.conf 
[...]
nameserver 127.0.0.53
options edns0 trust-ad
search .

In Multipass, we have the "multipass" domain defined, causing issues:
$ cat /etc/resolv.conf 
[...]
nameserver 127.0.0.53
options edns0 trust-ad
search multipass

=> This can be worked around in a similar way as the LXD workaround,
using a negative trust-anchor (see comment #9):

$ cat /usr/lib/dnssec-trust-anchors.d/mp.negative
multipass

=> But I did not find a way how to assign the Multipass project to this
bug report on Launchpad. Will reach out to them individually.

In LXD we have the "lxd" domain defined, causing issues:
$ cat /etc/resolv.conf 
[...]
nameserver 127.0.0.53
options edns0 trust-ad
search lxd

=> Interestingly, after installing systemd 257.7-1ubuntu3 (incl.
systemd-resolved-dnssec) and rebooting the LXD container/VM (I tried
both), name resolution was working initially, even on the .lxd local
name.

In the journal log I found some interesting messages:

"""
Aug 26 10:23:32 q1 systemd-resolved[112]: Using degraded feature set UDP instead of UDP+EDNS0+DO for DNS server 10.238.94.1.
Aug 26 10:23:32 q1 systemd-resolved[112]: [🡕] Server 10.238.94.1 does not support DNSSEC, downgrading to non-DNSSEC mode.
Aug 26 10:50:59 q1 systemd-resolved[112]: Grace period over, resuming full feature set (UDP+EDNS0+DO) for DNS server 10.238.94.1.
Aug 26 10:50:59 q1 systemd-resolved[112]: Using degraded feature set UDP instead of UDP+EDNS0+DO for DNS server 10.238.94.1.
"""


systemd-resolved seems to correctly detect that the upstream dnsmasq server is not supporting DNSSEC and enabling the fallback (non-DNSSEC) mode. After a grace period of some 25 sec, it resets to full DNSSEC support and eventually downgrades again.

This whole dance goes on for a while but eventually stops working,
especially after a "systemctl restart systemd-resolved". So something
seems to be wonky in systemd-resolved's detection of the upstream DNS
server feature set and logic for activation of the fallback mode.

Similar issues of feature detection, especially in "local" DNS servers (e.g. home routes or virtualization environments) have been described by upstream systemd some 5 years ago:
* https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx/message/AFHNUEHKC5KJVGBGSJBH2BMESUAGDF4H/
* https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx/message/P63RI3VBQ7NGL3AKMTR7PCVHVSCPYCLF/

It seems like this might still not be as reliable as we'd want it to be,
and I'm pondering if we should downgrade that "Recommends: systemd-
resolved-dnssec" to a "Suggests" after all...

-- 
You received this bug notification because you are a member of
Debcrafters packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/2119652

Title:
  systemd-resolved-dnssec breaks name resolution on lxd domain

Status in lxd:
  New
Status in bind9 package in Ubuntu:
  Invalid
Status in dnsmasq package in Ubuntu:
  Triaged
Status in libvirt package in Ubuntu:
  New
Status in livecd-rootfs package in Ubuntu:
  New
Status in lxd package in Ubuntu:
  New
Status in strongswan package in Ubuntu:
  Fix Released
Status in systemd package in Ubuntu:
  Confirmed

Bug description:
  By default, LXD containers will be configured with DNS pointing to the
  server listening on lxdbr0 on the host. The DHCP leases additionally
  configure the 'lxd' domain. LXD starts a dnsmasq server which is
  DNSSEC compatible, but by default is not actually configured for
  DNSSEC. This leads to DNSSEC validation errors as seen below:

  root@q1:~# apt policy systemd-resolved-dnssec
  systemd-resolved-dnssec:
    Installed: 257.7-1ubuntu3
    Candidate: 257.7-1ubuntu3
    Version table:
   *** 257.7-1ubuntu3 100
          100 http://archive.ubuntu.com/ubuntu questing-proposed/main amd64 Packages
          100 /var/lib/dpkg/status
  root@q1:~# resolvectl 
  Global
           Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=allow-downgrade/supported
    resolv.conf mode: stub

  Link 47 (eth0)
      Current Scopes: DNS
           Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=allow-downgrade/supported
  Current DNS Server: 10.148.181.1
         DNS Servers: 10.148.181.1 fd42:f983:5882:c87f::1 fe80::216:3eff:fed9:e3c1
          DNS Domain: lxd
       Default Route: yes
  root@q1:~# ping q2.lxd
  ping: q2.lxd: Temporary failure in name resolution
  root@q1:~# nslookup q2
  ;; Got SERVFAIL reply from 127.0.0.53
  Server:		127.0.0.53
  Address:	127.0.0.53#53

  ** server can't find q2.lxd: SERVFAIL

  root@q1:~# resolvectl dnssec eth0 no
  root@q1:~# nslookup q2
  Server:		127.0.0.53
  Address:	127.0.0.53#53

  Non-authoritative answer:
  Name:	q2.lxd
  Address: 10.148.181.44
  Name:	q2.lxd
  Address: fd42:f983:5882:c87f:216:3eff:fec5:c96c

  root@q1:~# ping -c 1 q2.lxd
  PING q2.lxd (fd42:f983:5882:c87f:216:3eff:fec5:c96c) 56 data bytes
  64 bytes from q2.lxd (fd42:f983:5882:c87f:216:3eff:fec5:c96c): icmp_seq=1 ttl=64 time=0.205 ms

  --- q2.lxd ping statistics ---
  1 packets transmitted, 1 received, 0% packet loss, time 0ms
  rtt min/avg/max/mdev = 0.205/0.205/0.205/0.000 ms

  root@q1:~# journalctl -b -u systemd-resolved.service --grep "DNSSEC validation failed"
  Aug 06 14:15:33 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question lxd IN DS: no-signature
  Aug 06 14:15:33 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question q1.lxd IN DS: no-signature
  Aug 06 14:15:33 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question q1.lxd IN A: no-signature
  Aug 06 14:15:33 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question q1.lxd IN AAAA: no-signature
  Aug 06 14:15:33 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question lxd IN DS: no-signature
  Aug 06 14:15:33 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question q1.lxd IN DS: no-signature
  Aug 06 14:15:33 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question q1.lxd IN AAAA: no-signature
  Aug 06 14:15:33 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question q1.lxd IN A: no-signature
  Aug 06 14:16:21 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question lxd IN DS: no-signature
  Aug 06 14:16:21 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question q2.lxd IN DS: no-signature
  Aug 06 14:16:21 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question q2.lxd IN A: no-signature
  Aug 06 14:16:21 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question q2.lxd IN AAAA: no-signature
  Aug 06 14:16:21 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question lxd IN DS: no-signature
  Aug 06 14:16:21 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question q2.lxd IN DS: no-signature
  Aug 06 14:16:21 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question q2.lxd IN A: no-signature
  Aug 06 14:16:21 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question q2.lxd IN AAAA: no-signature
  Aug 06 14:16:21 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question lxd IN DS: no-signature
  Aug 06 14:16:21 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question lxd.lxd IN DS: no-signature
  Aug 06 14:16:21 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question q2.lxd.lxd IN DS: no-signature
  Aug 06 14:16:21 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question q2.lxd.lxd IN A: no-signature
  Aug 06 14:16:21 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question q2.lxd.lxd IN AAAA: no-signature
  Aug 06 14:16:21 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question lxd IN DS: no-signature
  Aug 06 14:16:21 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question lxd.lxd IN DS: no-signature
  Aug 06 14:16:21 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question q2.lxd.lxd IN DS: no-signature
  Aug 06 14:16:21 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question q2.lxd.lxd IN A: no-signature
  Aug 06 14:16:21 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question q2.lxd.lxd IN AAAA: no-signature
  Aug 06 14:16:25 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question lxd IN DS: no-signature
  Aug 06 14:16:25 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question q2.lxd IN DS: no-signature
  Aug 06 14:16:25 q1 systemd-resolved[1526]: [🡕] DNSSEC validation failed for question q2.lxd IN A: no-signature

  Again, since the dnsmasq server listening on lxdbr0 is DNSSEC
  *compatible*, the downgrade logic implied by DNSSEC=allow-downgrade
  does not kick in.

To manage notifications about this bug go to:
https://bugs.launchpad.net/lxd/+bug/2119652/+subscriptions