ubuntu-translations-coordinators team mailing list archive

[Launchpad Bug Tracker <1804478@xxxxxxxxxxxxxxxxxx>]

You have been subscribed to a public bug:

[impact]

systemd-networkd, when configured to use ipv4 dhcp for an interface can hang. This triggers in two known cases:
a) if configured to ignore the dhcp server's route
b) the server provides no route
Then systemd-networkd will hang waiting for the interface's configuration to complete (until it times out).

This delays boot as well as any restart to systemd-networkd.

The fix is backporting upstream commit [1]

[1]: https://github.com/systemd/systemd/commit/223932c7

[test case]

There are two ways to test this.
A) make the system ignore Routes (slightly less realistic but easier to test)
configure an interface using systemd-networkd:

$ cat /etc/systemd/network/20-ens7.network
[Match]
Name=ens7

[Network]
DHCP=ipv4

[DHCP]
UseRoutes=false

then reboot, and check:

$ systemctl status systemd-networkd-wait-online
● systemd-networkd-wait-online.service - Wait for Network to be Configured
   Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2019-04-08 23:59:26 UTC; 2min 59s ago
[...]
Apr 08 23:57:27 lp1804478 systemd[1]: Starting Wait for Network to be Configured...
Apr 08 23:57:30 lp1804478 systemd-networkd-wait-online[593]: managing: ens3
Apr 08 23:57:30 lp1804478 systemd-networkd-wait-online[593]: ignoring: lo
Apr 08 23:59:26 lp1804478 systemd-networkd-wait-online[593]: Event loop failed: Connection timed out
Apr 08 23:59:26 lp1804478 systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Apr 08 23:59:26 lp1804478 systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Apr 08 23:59:26 lp1804478 systemd[1]: Failed to start Wait for Network to be Configured.

B) Make a dhcp server to not provide rules
Prepare a Ubuntu Bionic Guest under libvirt, e.g. using uvtool:
  $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=bionic
  $ uvt-kvm create --password ubuntu bionic arch=amd64 release=bionic label=daily

Create a isolated network:
$ cat > isolate.xml << EOF
<network>
  <name>isolated</name>
  <!-- <bridge name='virbriso1' stp='on' delay='0'/> -->
  <ip address='192.168.251.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.251.2' end='192.168.251.254'/>
    </dhcp>
  </ip>
</network>
EOF
$ virsh net-define isolate.xml
$ virsh net-start isolated

Edit the guest and add that network:
$ virsh shutdown bionic
$ virsh edit bionic
#add this:
  <interface type='network'>
    <source network='isolated'/>
    <mac address='52:54:00:c1:69:08'/>
    <model type='virtio'/>
    <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
  </interface>

In the guest make the device use DHCP:
Add this to /etc/netplan/50-cloud-init.yaml
        ens7:
            dhcp4: true
            match:
                macaddress: 52:54:00:c1:69:08
            set-name: ens7
$ sudo netplan apply

When rebooting the guest again it will fail
● systemd-networkd-wait-online.service - Wait for Network to be Configured
   Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2019-04-09 07:07:48 UTC; 1min 41s ago
     Docs: man:systemd-networkd-wait-online.service(8)
  Process: 563 ExecStart=/lib/systemd/systemd-networkd-wait-online (code=exited, status=1/FAILURE)
 Main PID: 563 (code=exited, status=1/FAILURE)

Apr 09 07:05:48 bionic-dgx2 systemd[1]: Starting Wait for Network to be Configured...
Apr 09 07:05:50 bionic-dgx2 systemd-networkd-wait-online[563]: managing: ens3
Apr 09 07:07:48 bionic-dgx2 systemd-networkd-wait-online[563]: Event loop failed: Connection timed out
Apr 09 07:07:48 bionic-dgx2 systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Apr 09 07:07:48 bionic-dgx2 systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Apr 09 07:07:48 bionic-dgx2 systemd[1]: Failed to start Wait for Network to be Configured.

When working cases A and B will both have the service starting fast and happy.
$ sudo systemctl status systemd-networkd-wait-online
   Active: active (exited) since Tue 2019-04-09 07:17:16 UTC; 12s ago
[...]
Apr 09 07:17:16 bionic-dgx2 systemd-networkd-wait-online[575]: managing: ens3
[...]
Apr 09 07:17:16 bionic-dgx2 systemd-networkd-wait-online[575]: managing: ens7

[regression potential]

this alters how systemd-networkd works when it starts or restarts,
specifically how it handles dhcp4; regressions would be around setting
up interface(s) using dhcpv4, possibly failing to correctly configure
dhcpv4 interface(s) or even failing internal assertions and exiting.

The problem is that without a route there will be no netlink even t that
can be tapped.

But the change is only a 4 line change that does explicitly
link->dhcp4_configured = true and calling link_check_ready() in
dhcp4_address_handler() in case link_set_dhcp_routes() sent no netlink
messages (dhcp4_messages is zero).

Therefore the change seems rather small, reviewable and safe.

[other info]

original description:

--

root cause in systemd bug fixed in Commit
223932c786ada7f758a7b7878a6ad2dae0d1e5fb:
https://github.com/systemd/systemd/pull/8728

Environment:
Ubuntu 18.04 LTS amd64
systemd package version 237-3ubuntu10.9

How to trigger:
1. add netplan interface with dhcpv4 client enabled:
        enp0s8:
            addresses: []
            dhcp4: true
2. configure dhcp server to NOT give a default route, or any route for that matter
3. reboot
4. systemd-networkd-wait-online will block until dhcp renew is triggered
root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3
Event loop failed: Connection timed out
root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3 --ignore=enp0s8
managing: enp0s3
ignoring: lo
ignoring: enp0s8

How to fix:
Backport upstream changes from systemd (see related systemd ticket).

** Affects: ubuntu-translations
     Importance: Undecided
         Status: New

** Affects: systemd (Ubuntu)
     Importance: Undecided
         Status: Fix Released

** Affects: systemd (Ubuntu Bionic)
     Importance: Medium
     Assignee: Dan Streetman (ddstreet)
         Status: Fix Committed


** Tags: systemd verification-done verification-done-bionic
-- 
netplan dhcp interface with no default route causes systemd-networkd-wait-online to hang
https://bugs.launchpad.net/bugs/1804478
You received this bug notification because you are a member of Ubuntu Translations Coordinators, which is subscribed to Ubuntu Translations.