group.of.nepali.translators team mailing list archive

Thread
Date
[Bug 2084667] [NEW] AWS Pro xenial images do not fully boot due to snapd.seeded.service never finishing

To: group.of.nepali.translators@xxxxxxxxxxxxxxxxxxx
From: Alberto Contreras <2084667@xxxxxxxxxxxxxxxxxx>
Date: Wed, 16 Oct 2024 11:10:23 -0000
Reply-to: Bug 2084667 <2084667@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

The boot transaction doesn't finish on Ubuntu AWS Pro Xenial instances:

```
# cat /etc/cloud/build.info
build_name: pro-server
serial: 20241002

# systemctl status | grep State
    State: starting
```

snapd.seeded.service does never finish and blocks cloud-config.service,
ua-auto-attach.service, cloud-final.service and the rest of the boot
transaction:

```
# systemctl status snapd.seeded.service 
● snapd.seeded.service - Wait until snapd is fully seeded
   Loaded: loaded (/lib/systemd/system/snapd.seeded.service; enabled; vendor preset: enabled)
   Active: activating (start) since Tue 2024-10-08 10:32:22 UTC; 15min ago
 Main PID: 1277 (snap)
    Tasks: 8
   Memory: 12.0M
      CPU: 649ms
   CGroup: /system.slice/snapd.seeded.service
           └─1277 /usr/bin/snap wait system seed.loaded

Oct 08 10:32:22 ip-10-0-3-253 systemd[1]: Starting Wait until snapd is fully seeded...
root@ip-10-0-3-253:~/aciba# systemctl list-dependencies --reverse snapd.seeded.service 
snapd.seeded.service
● ├─cloud-final.service
● └─multi-user.target
●   └─graphical.target

# systemctl status ua-auto-attach.service                                                                                                                                                
● ua-auto-attach.service - Ubuntu Pro auto attach
   Loaded: loaded (/lib/systemd/system/ua-auto-attach.service; enabled; vendor preset: enabled)
   Active: activating (start) since Tue 2024-10-08 10:32:19 UTC; 22min ago
 Main PID: 1094 (python3)
    Tasks: 9
   Memory: 98.3M
      CPU: 3.170s
   CGroup: /system.slice/ua-auto-attach.service
           ├─1094 /usr/bin/python3 /usr/lib/ubuntu-advantage/auto_attach.py
           └─2185 /usr/bin/snap wait system seed.loaded

Oct 08 10:32:19 ip-10-0-3-253 systemd[1]: Starting Ubuntu Pro auto attach...
Oct 08 10:33:04 ip-10-0-3-253 python3[1094]: ["2024-10-08T10:33:04.175", "INFO", "ubuntupro.timer.update_messaging", "update_motd_messages", 70, "Updating Ubuntu Pro messages for MOTD.", {}]
Oct 08 10:33:04 ip-10-0-3-253 python3[1094]: ["2024-10-08T10:33:04.272", "WARNING", "ubuntupro.system", "subp", 740, "Stderr: error: too early for operation, device not yet seeded or device model not acknowledge
Oct 08 10:33:04 ip-10-0-3-253 python3[1094]: ["2024-10-08T10:33:04.788", "WARNING", "ubuntupro.system", "subp", 740, "Stderr: error: too early for operation, device not yet seeded or device model not acknowledge
Oct 08 10:33:05 ip-10-0-3-253 python3[1094]: ["2024-10-08T10:33:05.806", "WARNING", "ubuntupro.system", "subp", 740, "Stderr: error: too early for operation, device not yet seeded or device model not acknowledge
Oct 08 10:33:10 ip-10-0-3-253 python3[1094]: ["2024-10-08T10:33:10.827", "WARNING", "ubuntupro.system", "subp", 740, "Stderr: error: too early for operation, device not yet seeded or device model not acknowledge
Oct 08 10:33:10 ip-10-0-3-253 python3[1094]: ["2024-10-08T10:33:10.827", "WARNING", "ubuntupro.entitlements.livepatch", "_perform_enable", 112, "Failed to install snapd as a snap", {"exc_info": "Traceback (most 
...skipping...
● ua-auto-attach.service - Ubuntu Pro auto attach
   Loaded: loaded (/lib/systemd/system/ua-auto-attach.service; enabled; vendor preset: enabled)
   Active: activating (start) since Tue 2024-10-08 10:32:19 UTC; 22min ago
 Main PID: 1094 (python3)
    Tasks: 9
   Memory: 98.3M
      CPU: 3.170s
   CGroup: /system.slice/ua-auto-attach.service
           ├─1094 /usr/bin/python3 /usr/lib/ubuntu-advantage/auto_attach.py
           └─2185 /usr/bin/snap wait system seed.loaded

Oct 08 10:32:19 ip-10-0-3-253 systemd[1]: Starting Ubuntu Pro auto attach...
Oct 08 10:33:04 ip-10-0-3-253 python3[1094]: ["2024-10-08T10:33:04.175", "INFO", "ubuntupro.timer.update_messaging", "update_motd_messages", 70, "Updating Ubuntu Pro messages for MOTD.", {}]
Oct 08 10:33:04 ip-10-0-3-253 python3[1094]: ["2024-10-08T10:33:04.272", "WARNING", "ubuntupro.system", "subp", 740, "Stderr: error: too early for operation, device not yet seeded or device model not acknowledge
Oct 08 10:33:04 ip-10-0-3-253 python3[1094]: ["2024-10-08T10:33:04.788", "WARNING", "ubuntupro.system", "subp", 740, "Stderr: error: too early for operation, device not yet seeded or device model not acknowledge
Oct 08 10:33:05 ip-10-0-3-253 python3[1094]: ["2024-10-08T10:33:05.806", "WARNING", "ubuntupro.system", "subp", 740, "Stderr: error: too early for operation, device not yet seeded or device model not acknowledge
Oct 08 10:33:10 ip-10-0-3-253 python3[1094]: ["2024-10-08T10:33:10.827", "WARNING", "ubuntupro.system", "subp", 740, "Stderr: error: too early for operation, device not yet seeded or device model not acknowledge
Oct 08 10:33:10 ip-10-0-3-253 python3[1094]: ["2024-10-08T10:33:10.827", "WARNING", "ubuntupro.entitlements.livepatch", "_perform_enable", 112, "Failed to install snapd as a snap", {"exc_info": "Traceback (most 
```

The issue is that amazon-ssm-agent is preseeded in the image and it
requires core22, which requires snapd 2.55, but snapd 2.55 is not
available at boot time provoking snaps.preseed.service to never finish:

```
# cat /var/lib/snapd/seed/seed.yaml 
snaps:
  -
    name: core22
    channel: stable
    file: core22_1621.snap
  -
    name: snapd
    channel: stable
    file: snapd_21759.snap
  -
    name: amazon-ssm-agent
    channel: stable/ubuntu-16.04
    classic: true
    file: amazon-ssm-agent_9565.snap

# apt-cache show snapd | grep Version
Version: 2.54.3+16.04.0ubuntu0.1~esm6
Version: 2.48.3
Version: 2.0.2

# journalctl -u snapd.seeded
-- Logs begin at Mon 2024-10-14 13:03:41 UTC, end at Mon 2024-10-14 13:48:54 UTC. --
Oct 14 13:03:52 ip-10-0-3-166 systemd[1]: Starting Wait until snapd is fully seeded...

# journalctl -u snapd*
-- Logs begin at Mon 2024-10-14 13:03:41 UTC, end at Mon 2024-10-14 13:48:54 UTC. --
Oct 14 13:03:45 ubuntu systemd[1]: Starting Load AppArmor profiles managed internally by snapd...
Oct 14 13:03:45 ubuntu snapd-apparmor[667]: find: '/var/lib/snapd/apparmor/profiles/': No such file or directory
Oct 14 13:03:45 ubuntu systemd[1]: Started Load AppArmor profiles managed internally by snapd.
Oct 14 13:03:50 ip-10-0-3-166 systemd[1]: Starting Socket activation for snappy daemon.
Oct 14 13:03:50 ip-10-0-3-166 systemd[1]: Listening on Socket activation for snappy daemon.
Oct 14 13:03:50 ip-10-0-3-166 systemd[1]: Starting Snap Daemon...
Oct 14 13:03:51 ip-10-0-3-166 snapd[1097]: AppArmor status: apparmor is enabled and all features are available
Oct 14 13:03:51 ip-10-0-3-166 snapd[1097]: daemon.go:246: started snapd/2.54.3+16.04.0ubuntu0.1~esm6 (series 16; classic) ubuntu/16.04 (amd64) linux/4.4.0-1174-aws.
Oct 14 13:03:51 ip-10-0-3-166 snapd[1097]: daemon.go:339: adjusting startup timeout by 30s (pessimistic estimate of 30s plus 5s per snap)
Oct 14 13:03:52 ip-10-0-3-166 snapd[1097]: helpers.go:137: error trying to compare the snap system key: system-key missing on disk
Oct 14 13:03:52 ip-10-0-3-166 systemd[1]: Started Snap Daemon.
Oct 14 13:03:52 ip-10-0-3-166 systemd[1]: Starting Wait until snapd is fully seeded...
Oct 14 13:04:10 ip-10-0-3-166 snapd[1097]: stateengine.go:149: state ensure error: devicemgr: cannot seed: snap "core22" assumes unsupported features: snapd2.55.5 (try to refresh snapd)
Oct 14 13:04:10 ip-10-0-3-166 snapd[1097]: api_snaps.go:307: Installing snap "snapd" revision unset
Oct 14 13:04:11 ip-10-0-3-166 snapd[1097]: api_snaps.go:307: Installing snap "snapd" revision unset
Oct 14 13:04:13 ip-10-0-3-166 snapd[1097]: api_snaps.go:307: Installing snap "snapd" revision unset
Oct 14 13:04:18 ip-10-0-3-166 snapd[1097]: api_snaps.go:307: Installing snap "snapd" revision unset
Oct 14 13:08:54 ip-10-0-3-166 snapd[1097]: stateengine.go:149: state ensure error: devicemgr: cannot seed: snap "core22" assumes unsupported features: snapd2.55.5 (try to refresh snapd)
Oct 14 13:13:54 ip-10-0-3-166 snapd[1097]: stateengine.go:149: state ensure error: devicemgr: cannot seed: snap "core22" assumes unsupported features: snapd2.55.5 (try to refresh snapd)
Oct 14 13:18:54 ip-10-0-3-166 snapd[1097]: stateengine.go:149: state ensure error: devicemgr: cannot seed: snap "core22" assumes unsupported features: snapd2.55.5 (try to refresh snapd)
Oct 14 13:23:54 ip-10-0-3-166 snapd[1097]: stateengine.go:149: state ensure error: devicemgr: cannot seed: snap "core22" assumes unsupported features: snapd2.55.5 (try to refresh snapd)
Oct 14 13:28:54 ip-10-0-3-166 snapd[1097]: stateengine.go:149: state ensure error: devicemgr: cannot seed: snap "core22" assumes unsupported features: snapd2.55.5 (try to refresh snapd)
Oct 14 13:33:54 ip-10-0-3-166 snapd[1097]: stateengine.go:149: state ensure error: devicemgr: cannot seed: snap "core22" assumes unsupported features: snapd2.55.5 (try to refresh snapd)
Oct 14 13:38:54 ip-10-0-3-166 snapd[1097]: stateengine.go:149: state ensure error: devicemgr: cannot seed: snap "core22" assumes unsupported features: snapd2.55.5 (try to refresh snapd)
Oct 14 13:43:54 ip-10-0-3-166 snapd[1097]: stateengine.go:149: state ensure error: devicemgr: cannot seed: snap "core22" assumes unsupported features: snapd2.55.5 (try to refresh snapd)
Oct 14 13:48:54 ip-10-0-3-166 snapd[1097]: stateengine.go:149: state ensure error: devicemgr: cannot seed: snap "core22" assumes unsupported features: snapd2.55.5 (try to refresh snapd)
```

** Affects: cloud-images
     Importance: Undecided
         Status: New

** Affects: snapd (Ubuntu)
     Importance: Undecided
         Status: New

** Affects: snapd (Ubuntu Xenial)
     Importance: Undecided
         Status: New

** Also affects: snapd (Ubuntu)
   Importance: Undecided
       Status: New

** Also affects: snapd (Ubuntu Xenial)
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of नेपाली
भाषा समायोजकहरुको समूह, which is subscribed to Xenial.
Matching subscriptions: Ubuntu 16.04 Bugs
https://bugs.launchpad.net/bugs/2084667

Title:
  AWS Pro xenial images do not fully boot due to snapd.seeded.service
  never finishing

Status in cloud-images:
  New
Status in snapd package in Ubuntu:
  New
Status in snapd source package in Xenial:
  New

Bug description:
  The boot transaction doesn't finish on Ubuntu AWS Pro Xenial
  instances:

  ```
  # cat /etc/cloud/build.info
  build_name: pro-server
  serial: 20241002

  # systemctl status | grep State
      State: starting
  ```

  snapd.seeded.service does never finish and blocks cloud-
  config.service, ua-auto-attach.service, cloud-final.service and the
  rest of the boot transaction:

  ```
  # systemctl status snapd.seeded.service 
  ● snapd.seeded.service - Wait until snapd is fully seeded
     Loaded: loaded (/lib/systemd/system/snapd.seeded.service; enabled; vendor preset: enabled)
     Active: activating (start) since Tue 2024-10-08 10:32:22 UTC; 15min ago
   Main PID: 1277 (snap)
      Tasks: 8
     Memory: 12.0M
        CPU: 649ms
     CGroup: /system.slice/snapd.seeded.service
             └─1277 /usr/bin/snap wait system seed.loaded

  Oct 08 10:32:22 ip-10-0-3-253 systemd[1]: Starting Wait until snapd is fully seeded...
  root@ip-10-0-3-253:~/aciba# systemctl list-dependencies --reverse snapd.seeded.service 
  snapd.seeded.service
  ● ├─cloud-final.service
  ● └─multi-user.target
  ●   └─graphical.target

  # systemctl status ua-auto-attach.service                                                                                                                                                
  ● ua-auto-attach.service - Ubuntu Pro auto attach
     Loaded: loaded (/lib/systemd/system/ua-auto-attach.service; enabled; vendor preset: enabled)
     Active: activating (start) since Tue 2024-10-08 10:32:19 UTC; 22min ago
   Main PID: 1094 (python3)
      Tasks: 9
     Memory: 98.3M
        CPU: 3.170s
     CGroup: /system.slice/ua-auto-attach.service
             ├─1094 /usr/bin/python3 /usr/lib/ubuntu-advantage/auto_attach.py
             └─2185 /usr/bin/snap wait system seed.loaded

  Oct 08 10:32:19 ip-10-0-3-253 systemd[1]: Starting Ubuntu Pro auto attach...
  Oct 08 10:33:04 ip-10-0-3-253 python3[1094]: ["2024-10-08T10:33:04.175", "INFO", "ubuntupro.timer.update_messaging", "update_motd_messages", 70, "Updating Ubuntu Pro messages for MOTD.", {}]
  Oct 08 10:33:04 ip-10-0-3-253 python3[1094]: ["2024-10-08T10:33:04.272", "WARNING", "ubuntupro.system", "subp", 740, "Stderr: error: too early for operation, device not yet seeded or device model not acknowledge
  Oct 08 10:33:04 ip-10-0-3-253 python3[1094]: ["2024-10-08T10:33:04.788", "WARNING", "ubuntupro.system", "subp", 740, "Stderr: error: too early for operation, device not yet seeded or device model not acknowledge
  Oct 08 10:33:05 ip-10-0-3-253 python3[1094]: ["2024-10-08T10:33:05.806", "WARNING", "ubuntupro.system", "subp", 740, "Stderr: error: too early for operation, device not yet seeded or device model not acknowledge
  Oct 08 10:33:10 ip-10-0-3-253 python3[1094]: ["2024-10-08T10:33:10.827", "WARNING", "ubuntupro.system", "subp", 740, "Stderr: error: too early for operation, device not yet seeded or device model not acknowledge
  Oct 08 10:33:10 ip-10-0-3-253 python3[1094]: ["2024-10-08T10:33:10.827", "WARNING", "ubuntupro.entitlements.livepatch", "_perform_enable", 112, "Failed to install snapd as a snap", {"exc_info": "Traceback (most 
  ...skipping...
  ● ua-auto-attach.service - Ubuntu Pro auto attach
     Loaded: loaded (/lib/systemd/system/ua-auto-attach.service; enabled; vendor preset: enabled)
     Active: activating (start) since Tue 2024-10-08 10:32:19 UTC; 22min ago
   Main PID: 1094 (python3)
      Tasks: 9
     Memory: 98.3M
        CPU: 3.170s
     CGroup: /system.slice/ua-auto-attach.service
             ├─1094 /usr/bin/python3 /usr/lib/ubuntu-advantage/auto_attach.py
             └─2185 /usr/bin/snap wait system seed.loaded

  Oct 08 10:32:19 ip-10-0-3-253 systemd[1]: Starting Ubuntu Pro auto attach...
  Oct 08 10:33:04 ip-10-0-3-253 python3[1094]: ["2024-10-08T10:33:04.175", "INFO", "ubuntupro.timer.update_messaging", "update_motd_messages", 70, "Updating Ubuntu Pro messages for MOTD.", {}]
  Oct 08 10:33:04 ip-10-0-3-253 python3[1094]: ["2024-10-08T10:33:04.272", "WARNING", "ubuntupro.system", "subp", 740, "Stderr: error: too early for operation, device not yet seeded or device model not acknowledge
  Oct 08 10:33:04 ip-10-0-3-253 python3[1094]: ["2024-10-08T10:33:04.788", "WARNING", "ubuntupro.system", "subp", 740, "Stderr: error: too early for operation, device not yet seeded or device model not acknowledge
  Oct 08 10:33:05 ip-10-0-3-253 python3[1094]: ["2024-10-08T10:33:05.806", "WARNING", "ubuntupro.system", "subp", 740, "Stderr: error: too early for operation, device not yet seeded or device model not acknowledge
  Oct 08 10:33:10 ip-10-0-3-253 python3[1094]: ["2024-10-08T10:33:10.827", "WARNING", "ubuntupro.system", "subp", 740, "Stderr: error: too early for operation, device not yet seeded or device model not acknowledge
  Oct 08 10:33:10 ip-10-0-3-253 python3[1094]: ["2024-10-08T10:33:10.827", "WARNING", "ubuntupro.entitlements.livepatch", "_perform_enable", 112, "Failed to install snapd as a snap", {"exc_info": "Traceback (most 
  ```

  The issue is that amazon-ssm-agent is preseeded in the image and it
  requires core22, which requires snapd 2.55, but snapd 2.55 is not
  available at boot time provoking snaps.preseed.service to never
  finish:

  ```
  # cat /var/lib/snapd/seed/seed.yaml 
  snaps:
    -
      name: core22
      channel: stable
      file: core22_1621.snap
    -
      name: snapd
      channel: stable
      file: snapd_21759.snap
    -
      name: amazon-ssm-agent
      channel: stable/ubuntu-16.04
      classic: true
      file: amazon-ssm-agent_9565.snap

  # apt-cache show snapd | grep Version
  Version: 2.54.3+16.04.0ubuntu0.1~esm6
  Version: 2.48.3
  Version: 2.0.2

  # journalctl -u snapd.seeded
  -- Logs begin at Mon 2024-10-14 13:03:41 UTC, end at Mon 2024-10-14 13:48:54 UTC. --
  Oct 14 13:03:52 ip-10-0-3-166 systemd[1]: Starting Wait until snapd is fully seeded...

  # journalctl -u snapd*
  -- Logs begin at Mon 2024-10-14 13:03:41 UTC, end at Mon 2024-10-14 13:48:54 UTC. --
  Oct 14 13:03:45 ubuntu systemd[1]: Starting Load AppArmor profiles managed internally by snapd...
  Oct 14 13:03:45 ubuntu snapd-apparmor[667]: find: '/var/lib/snapd/apparmor/profiles/': No such file or directory
  Oct 14 13:03:45 ubuntu systemd[1]: Started Load AppArmor profiles managed internally by snapd.
  Oct 14 13:03:50 ip-10-0-3-166 systemd[1]: Starting Socket activation for snappy daemon.
  Oct 14 13:03:50 ip-10-0-3-166 systemd[1]: Listening on Socket activation for snappy daemon.
  Oct 14 13:03:50 ip-10-0-3-166 systemd[1]: Starting Snap Daemon...
  Oct 14 13:03:51 ip-10-0-3-166 snapd[1097]: AppArmor status: apparmor is enabled and all features are available
  Oct 14 13:03:51 ip-10-0-3-166 snapd[1097]: daemon.go:246: started snapd/2.54.3+16.04.0ubuntu0.1~esm6 (series 16; classic) ubuntu/16.04 (amd64) linux/4.4.0-1174-aws.
  Oct 14 13:03:51 ip-10-0-3-166 snapd[1097]: daemon.go:339: adjusting startup timeout by 30s (pessimistic estimate of 30s plus 5s per snap)
  Oct 14 13:03:52 ip-10-0-3-166 snapd[1097]: helpers.go:137: error trying to compare the snap system key: system-key missing on disk
  Oct 14 13:03:52 ip-10-0-3-166 systemd[1]: Started Snap Daemon.
  Oct 14 13:03:52 ip-10-0-3-166 systemd[1]: Starting Wait until snapd is fully seeded...
  Oct 14 13:04:10 ip-10-0-3-166 snapd[1097]: stateengine.go:149: state ensure error: devicemgr: cannot seed: snap "core22" assumes unsupported features: snapd2.55.5 (try to refresh snapd)
  Oct 14 13:04:10 ip-10-0-3-166 snapd[1097]: api_snaps.go:307: Installing snap "snapd" revision unset
  Oct 14 13:04:11 ip-10-0-3-166 snapd[1097]: api_snaps.go:307: Installing snap "snapd" revision unset
  Oct 14 13:04:13 ip-10-0-3-166 snapd[1097]: api_snaps.go:307: Installing snap "snapd" revision unset
  Oct 14 13:04:18 ip-10-0-3-166 snapd[1097]: api_snaps.go:307: Installing snap "snapd" revision unset
  Oct 14 13:08:54 ip-10-0-3-166 snapd[1097]: stateengine.go:149: state ensure error: devicemgr: cannot seed: snap "core22" assumes unsupported features: snapd2.55.5 (try to refresh snapd)
  Oct 14 13:13:54 ip-10-0-3-166 snapd[1097]: stateengine.go:149: state ensure error: devicemgr: cannot seed: snap "core22" assumes unsupported features: snapd2.55.5 (try to refresh snapd)
  Oct 14 13:18:54 ip-10-0-3-166 snapd[1097]: stateengine.go:149: state ensure error: devicemgr: cannot seed: snap "core22" assumes unsupported features: snapd2.55.5 (try to refresh snapd)
  Oct 14 13:23:54 ip-10-0-3-166 snapd[1097]: stateengine.go:149: state ensure error: devicemgr: cannot seed: snap "core22" assumes unsupported features: snapd2.55.5 (try to refresh snapd)
  Oct 14 13:28:54 ip-10-0-3-166 snapd[1097]: stateengine.go:149: state ensure error: devicemgr: cannot seed: snap "core22" assumes unsupported features: snapd2.55.5 (try to refresh snapd)
  Oct 14 13:33:54 ip-10-0-3-166 snapd[1097]: stateengine.go:149: state ensure error: devicemgr: cannot seed: snap "core22" assumes unsupported features: snapd2.55.5 (try to refresh snapd)
  Oct 14 13:38:54 ip-10-0-3-166 snapd[1097]: stateengine.go:149: state ensure error: devicemgr: cannot seed: snap "core22" assumes unsupported features: snapd2.55.5 (try to refresh snapd)
  Oct 14 13:43:54 ip-10-0-3-166 snapd[1097]: stateengine.go:149: state ensure error: devicemgr: cannot seed: snap "core22" assumes unsupported features: snapd2.55.5 (try to refresh snapd)
  Oct 14 13:48:54 ip-10-0-3-166 snapd[1097]: stateengine.go:149: state ensure error: devicemgr: cannot seed: snap "core22" assumes unsupported features: snapd2.55.5 (try to refresh snapd)
  ```

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-images/+bug/2084667/+subscriptions
Follow ups

[Bug 2084667] Re: AWS Pro xenial images do not fully boot due to snapd.seeded.service never finishing
From: Ernest Lotter, 2025-01-13