group.of.nepali.translators team mailing list archive

Thread
Date
[Bug 1870514] Re: update containerd:amd64 1.3.3-0 stops docker daemon

To: group.of.nepali.translators@xxxxxxxxxxxxxxxxxxx
From: Bryce Harrington <1870514@xxxxxxxxxxxxxxxxxx>
Date: Sat, 05 Dec 2020 01:31:09 -0000
Reply-to: Bug 1870514 <1870514@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
I'd like to give you all an update and outline our plans for this.

The Canonical server team has made analysis of this issue a top
priority.  We've identified and tested out several possible theories.
Our findings suggest that the breakage involves two distinct issues, one
the BindTo= issue mentioned above, the other caused by a bug in the
docker.io package causing the service to stop on package upgrade; see
specifically the service stop command at the end of
/var/lib/dpkg/info/docker.io.prerm.  We'll use LP: #1870514 to track the
former issue, and #1906364 the latter.  LP: #1658691 gives some past
background for reference.

The tricky part is that unfortunately any change we make to docker.io
requires the running of the prerm script (the version of the script
already present on your system, not the one we'd be installing), and
thus triggers the bug.  In other words, updating your system to prevent
the bug will cause one more docker stop.  Thereafter, the upgrade will
not restart the service when rolling out CVE fixes to either containerd
or docker.io; it may prompt to do so if running interactively (e.g.
https://imgur.com/2Za5dbQ.png), otherwise it should respect the debconf
setting.

We would appreciate feedback, testing and/or review of the proposed fix,
available in this PPA:

   https://launchpad.net/~bryce/+archive/ubuntu/containerd-sru-lp1870514
-docker-dh/


** Package changed: containerd (Ubuntu Xenial) => docker.io (Ubuntu
Xenial)

-- 
You received this bug notification because you are a member of नेपाली
भाषा समायोजकहरुको समूह, which is subscribed to Xenial.
Matching subscriptions: Ubuntu 16.04 Bugs
https://bugs.launchpad.net/bugs/1870514

Title:
  update containerd:amd64 1.3.3-0 stops docker daemon

Status in docker.io package in Ubuntu:
  In Progress
Status in docker.io source package in Xenial:
  In Progress
Status in docker.io source package in Bionic:
  In Progress
Status in docker.io source package in Focal:
  In Progress
Status in docker.io source package in Groovy:
  In Progress
Status in docker.io source package in Hirsute:
  In Progress

Bug description:
  hello we have had in several vms, the problem that after updating to "
  containerd:amd64 1.3.3" the docker daemon is stopped and so are all
  running containers

  the vms run with ubuntu 18.04.1/18.04.2

  journal log during the update:

  ```
  -- Logs begin at Tue 2020-01-14 09:58:27 CET, end at Fri 2020-04-03 11:30:39 CEST. --
  Apr 03 06:09:09 server dockerd[1751]: time="2020-04-03T06:09:09+02:00" level=info msg="level=info ts=2020-04-03T04:09:09.648358396Z caller=loki.go:55 container_id=1eefb971e84cdaa4eadb0ac890fe9adf5275a4e9cbcf9635cf941b3aa469dd1f msg=\"ignoring empty line\" line=" plugin=255e05f2d4535991812f30e4cb5ea39fe0b498ca612c26dd89909de318b7f68d
  Apr 03 06:09:09 server dockerd[1751]: time="2020-04-03T06:09:09+02:00" level=info msg="level=info ts=2020-04-03T04:09:09.660781933Z caller=loki.go:55 container_id=1eefb971e84cdaa4eadb0ac890fe9adf5275a4e9cbcf9635cf941b3aa469dd1f msg=\"ignoring empty line\" line=\"    \"" plugin=255e05f2d4535991812f30e4cb5ea39fe0b498ca612c26dd89909de318b7f68d
  Apr 03 06:09:14 server interface_rules[1485]: {"time":"2020-04-03 06:09:14+02:00","message":"interface enp0s3 works fine"}
  Apr 03 06:09:14 server interface_rules[1485]: {"time":"2020-04-03 06:09:14+02:00","message":"interface enp0s8 works fine"}
  Apr 03 06:09:24 server dockerd[1751]: time="2020-04-03T06:09:24+02:00" level=info msg="level=info ts=2020-04-03T04:09:24.650166885Z caller=loki.go:55 container_id=1eefb971e84cdaa4eadb0ac890fe9adf5275a4e9cbcf9635cf941b3aa469dd1f msg=\"ignoring empty line\" line=" plugin=255e05f2d4535991812f30e4cb5ea39fe0b498ca612c26dd89909de318b7f68d
  Apr 03 06:09:24 server dockerd[1751]: time="2020-04-03T06:09:24+02:00" level=info msg="level=info ts=2020-04-03T04:09:24.663239678Z caller=loki.go:55 container_id=1eefb971e84cdaa4eadb0ac890fe9adf5275a4e9cbcf9635cf941b3aa469dd1f msg=\"ignoring empty line\" line=\"    \"" plugin=255e05f2d4535991812f30e4cb5ea39fe0b498ca612c26dd89909de318b7f68d
  Apr 03 06:09:31 server systemd[1]: Starting Daily apt upgrade and clean activities...
  Apr 03 06:09:39 server dockerd[1751]: time="2020-04-03T06:09:39+02:00" level=info msg="level=info ts=2020-04-03T04:09:39.64981528Z caller=loki.go:55 container_id=1eefb971e84cdaa4eadb0ac890fe9adf5275a4e9cbcf9635cf941b3aa469dd1f msg=\"ignoring empty line\" line=" plugin=255e05f2d4535991812f30e4cb5ea39fe0b498ca612c26dd89909de318b7f68d
  Apr 03 06:09:39 server dockerd[1751]: time="2020-04-03T06:09:39+02:00" level=info msg="level=info ts=2020-04-03T04:09:39.665086076Z caller=loki.go:55 container_id=1eefb971e84cdaa4eadb0ac890fe9adf5275a4e9cbcf9635cf941b3aa469dd1f msg=\"ignoring empty line\" line=\"    \"" plugin=255e05f2d4535991812f30e4cb5ea39fe0b498ca612c26dd89909de318b7f68d
  Apr 03 06:09:43 server systemd[1]: Stopping Docker Application Container Engine...
  Apr 03 06:09:43 server dockerd[1751]: time="2020-04-03T06:09:43.571408106+02:00" level=info msg="Processing signal 'terminated'"
  Apr 03 06:09:43 server dockerd[1751]: time="2020-04-03T06:09:43.614841270+02:00" level=info msg="Daemon shutdown complete"
  Apr 03 06:09:43 server dockerd[1751]: time="2020-04-03T06:09:43.614972021+02:00" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=moby
  Apr 03 06:09:43 server systemd[1]: Stopped Docker Application Container Engine.
  Apr 03 06:09:43 server systemd[1]: Stopping containerd container runtime...
  Apr 03 06:09:43 server systemd[1]: Closed Docker Socket for the API.
  Apr 03 06:09:43 server containerd[1736]: time="2020-04-03T06:09:43.896546147+02:00" level=info msg="Stop CRI service"
  Apr 03 06:09:43 server containerd[1736]: time="2020-04-03T06:09:43.897819071+02:00" level=info msg="Stop CRI service"
  Apr 03 06:09:43 server containerd[1736]: time="2020-04-03T06:09:43.899864471+02:00" level=info msg="Event monitor stopped"
  Apr 03 06:09:43 server containerd[1736]: time="2020-04-03T06:09:43.900291027+02:00" level=info msg="Stream server stopped"
  Apr 03 06:09:43 server systemd[1]: Stopped containerd container runtime.
  Apr 03 06:09:47 server systemd[1]: Reloading.
  Apr 03 06:09:48 server systemd[1]: Reloading.
  Apr 03 06:09:49 server systemd[1]: Reloading.
  Apr 03 06:09:49 server systemd[1]: Starting containerd container runtime...
  Apr 03 06:09:49 server containerd[1904]: time="2020-04-03T06:09:49.951538613+02:00" level=info msg="starting containerd" revision= version="1.3.3-0ubuntu1~18.04.2"
  Apr 03 06:09:49 server systemd[1]: Started containerd container runtime.
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.021260455+02:00" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.021461398+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.022876671+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.022992123+02:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.023036103+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.031683852+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.031948994+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.032592571+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.034455570+02:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.034808794+02:00" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.034914325+02:00" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.034952775+02:00" level=info msg="metadata content store policy set" policy=shared
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035299108+02:00" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035362929+02:00" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035585401+02:00" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035642662+02:00" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035676962+02:00" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035751653+02:00" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035797984+02:00" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035899825+02:00" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035955175+02:00" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035994665+02:00" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.504599571+02:00" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.504847245+02:00" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.507240519+02:00" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.514716375+02:00" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515138699+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515385472+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515439083+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515474183+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515521113+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515569144+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515602784+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515645985+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515812346+02:00" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515960658+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.516016449+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.516062909+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.516218490+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.cri\"..." type=io.containerd.grpc.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.516437722+02:00" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntimeName:runc DefaultRuntime:{Type: Engine: PodAnnotations:[] Root: Options:<nil> PrivilegedWithoutHostDevices:false} UntrustedWorkloadRuntime:{Type: Engine: PodAnnotations:[] Root: Options:<nil> PrivilegedWithoutHostDevices:false} Runtimes:map[runc:{Type:io.containerd.runc.v1 Engine: PodAnnotations:[] Root: Options:<nil> PrivilegedWithoutHostDevices:false}] NoPivot:false} CniConfig:{NetworkPluginBinDir:/opt/cni/bin NetworkPluginConfDir:/etc/cni/net.d NetworkPluginMaxConfNum:1 NetworkPluginConfTemplate:} Registry:{Mirrors:map[docker.io:{Endpoints:[https://registry-1.docker.io]}] Configs:map[] Auths:map[]} DisableTCPService:true StreamServerAddress:127.0.0.1 StreamServerPort:0 StreamIdleTimeout:4h0m0s EnableSelinux:false SandboxImage:k8s.gcr.io/pause:3.1 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false RestrictOOMScoreAdj:false MaxConcurrentDownloads:3 DisableProcMount:false} ContainerdRootDir:/var/lib/containerd ContainerdEndpoint:/run/containerd/containerd.sock RootDir:/var/lib/containerd/io.containerd.grpc.v1.cri StateDir:/run/containerd/io.containerd.grpc.v1.cri}"
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.516732916+02:00" level=info msg="Connect containerd service"
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.517528134+02:00" level=info msg="Get image filesystem path \"/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs\""
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.517888717+02:00" level=error msg="Failed to load cni during init, please check CRI plugin status before setting up network for pods" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.518436663+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.518837507+02:00" level=info msg="Start subscribing containerd event"
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.519057900+02:00" level=info msg="Start recovering state"
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.519208531+02:00" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.519444914+02:00" level=info msg=serving... address=/run/containerd/containerd.sock
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.519481864+02:00" level=info msg="Start event monitor"
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.519573005+02:00" level=info msg="Start snapshots syncer"
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.519657075+02:00" level=info msg="Start streaming server"
  Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.519497564+02:00" level=info msg="containerd successfully booted in 0.569453s"
  Apr 03 06:10:08 server systemd[1]: Reloading.
  Apr 03 06:10:08 server systemd[1]: Stopping LSB: automatic crash report generation...
  Apr 03 06:10:08 server apport[2377]:  * Stopping automatic crash report generation: apport
  Apr 03 06:10:08 server apport[2377]:    ...done.
  Apr 03 06:10:08 server systemd[1]: Stopped LSB: automatic crash report generation.
  Apr 03 06:10:09 server systemd[1]: Reloading.
  Apr 03 06:10:09 server systemd[1]: Reloading.
  Apr 03 06:10:11 server systemd[1]: Reloading.
  Apr 03 06:10:11 server systemd[1]: Reloading.
  Apr 03 06:10:12 server systemd[1]: Reloading.
  Apr 03 06:10:12 server systemd[1]: Starting LSB: automatic crash report generation...
  Apr 03 06:10:12 server apport[2666]:  * Starting automatic crash report generation: apport
  Apr 03 06:10:12 server apport[2666]:    ...done.
  Apr 03 06:10:12 server systemd[1]: Started LSB: automatic crash report generation.
  Apr 03 06:10:12 server systemd[1]: Reloading.
  Apr 03 06:10:18 server systemd[1]: Started Daily apt upgrade and clean activities.

  ```

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1870514/+subscriptions