yahoo-eng-team team mailing list archive

[Brett Holman <1946088@xxxxxxxxxxxxxxxxxx>]

Thanks for the feedback. Since it sounds like there is already a
workable solution in place, I'm going to close this. If you think
further action is required please reopen and communicate what that would
be.

** Changed in: cloud-init
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1946088

Title:
  package module: all packages fail to install if just one has failed on
  http 503, no retries options are available

Status in cloud-init:
  Invalid

Bug description:
  **Explanation**
  Seems like the teleport repo was unavailable temporarily for some reason, maybe just this request was 503d. But this renders big hassle to kick in - engineers should dig why the deployment failed. While most probably with retries it would've probably just installed a bit later. I've searched for retries options in the docs, but failed to find them. Just found one mention in the PRs on GitHub that retries are bad, since it'll just spam the logs. If it's the established policy - can we please have this declared in the docs? Though, it'd be a strange one. It's better to have more logs, than semideployed instance, IMHO. Thanks for reading through all the rumbling.

  **Observed behaviour:**
  Cloud-init fails to install all packages if any one of the packages is unavailable.

  **Desired/expected behaviour:**
  Install available packages, fail to install just the unavailable packages. Have an option to retry installation, for example: retries: 15, wait-before-retry: 1m

  **Cloud provider: selectel**

  **Related config part:**

  ```
  hostname: ${hostname}.${project_fqdn}
  fqdn: ${hostname}.${project_fqdn}

  system_info:
    default_user:
      name: ...
  ...
      groups: [adm, audio, cdrom, dialout, floppy, video, plugdev, dip, netdev]
  ...
      shell: /bin/bash
      ssh_authorized_keys:
        - ${ssh_key}

  users:
    - default

  apt:
    preserve_sources_list: true
    sources:
      teleport:
        source: deb https://deb.releases.teleport.dev/ stable main
        keyid: C87ED53A6282C411
      docker:
        source: deb [arch=amd64] https://download.docker.com/linux/ubuntu $RELEASE stable
        keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88

  debconf_selections: |
    iptables-persistent iptables-persistent/autosave_v4 boolean true

  package_update: true

  packages:
    - mc
    - wget
    - jq
    - unzip
    - curl
    - xfsprogs
    - lvm2
    - teleport
    - docker-ce
    - iptables-persistent
  ```

  **Related log parts:**

  ```
  [   16.597420] cloud-init[994]: Get:1 http://repo.os.selectel.org bionic InRelease [10.7 kB]
  [   16.610710] cloud-init[994]: Get:2 http://mirror.selectel.ru/ubuntu bionic InRelease [242 kB]
  [   16.617046] cloud-init[994]: Get:3 http://mirror.selectel.ru/ubuntu bionic-updates InRelease [88.7 kB]
  [   16.623838] cloud-init[994]: Get:4 http://mirror.selectel.ru/ubuntu bionic-backports InRelease [74.6 kB]
  [   16.678604] cloud-init[994]: Get:5 https://download.docker.com/linux/ubuntu bionic InRelease [64.4 kB]
  [   16.709311] cloud-init[994]: Get:6 http://repo.os.selectel.org bionic/main all Packages [2105 B]
  [   16.720857] cloud-init[994]: Get:7 http://repo.os.selectel.org bionic/main amd64 Packages [2105 B]
  [   16.809186] cloud-init[994]: Get:8 http://mirror.selectel.ru/ubuntu bionic/main Sources [829 kB]
  [   16.843598] cloud-init[994]: Get:9 http://mirror.selectel.ru/ubuntu bionic/multiverse Sources [181 kB]
  [   16.845851] cloud-init[994]: Get:10 http://mirror.selectel.ru/ubuntu bionic/restricted Sources [5324 B]
  [   16.848508] cloud-init[994]: Get:11 http://mirror.selectel.ru/ubuntu bionic/universe Sources [9051 kB]
  [   16.960021] cloud-init[994]: Get:12 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
  [   16.982934] cloud-init[994]: Get:13 http://mirror.selectel.ru/ubuntu bionic/main amd64 Packages [1019 kB]
  [   16.999303] cloud-init[994]: Get:14 http://mirror.selectel.ru/ubuntu bionic/restricted amd64 Packages [9184 B]
  [   17.002854] cloud-init[994]: Get:15 http://mirror.selectel.ru/ubuntu bionic/universe amd64 Packages [8570 kB]
  [   17.110796] cloud-init[994]: Get:16 http://mirror.selectel.ru/ubuntu bionic/multiverse amd64 Packages [151 kB]
  [   17.116528] cloud-init[994]: Get:17 http://mirror.selectel.ru/ubuntu bionic-updates/restricted Sources [23.7 kB]
  [   17.119188] cloud-init[994]: Get:18 http://mirror.selectel.ru/ubuntu bionic-updates/universe Sources [456 kB]
  [   17.122502] cloud-init[994]: Err:19 https://deb.releases.teleport.dev stable InRelease
  [   17.124650] cloud-init[994]:   503  Service Unavailable [IP: 13.33.246.28 443]
  [   17.126251] cloud-init[994]: Get:20 http://mirror.selectel.ru/ubuntu bionic-updates/multiverse Sources [15.9 kB]
  [   17.128904] cloud-init[994]: Get:21 http://mirror.selectel.ru/ubuntu bionic-updates/main Sources [517 kB]
  [   17.138325] cloud-init[994]: Get:22 http://mirror.selectel.ru/ubuntu bionic-updates/main amd64 Packages [2249 kB]
  [   17.162915] cloud-init[994]: Get:23 https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages [19.9 kB]
  [   17.169036] cloud-init[994]: Get:24 http://mirror.selectel.ru/ubuntu bionic-updates/restricted amd64 Packages [492 kB]
  [   17.221550] cloud-init[994]: Get:25 http://mirror.selectel.ru/ubuntu bionic-updates/universe amd64 Packages [1755 kB]
  [   17.245138] cloud-init[994]: Get:26 http://mirror.selectel.ru/ubuntu bionic-updates/multiverse amd64 Packages [27.3 kB]
  [   17.247623] cloud-init[994]: Get:27 http://mirror.selectel.ru/ubuntu bionic-backports/main Sources [5440 B]
  [   17.250036] cloud-init[994]: Get:28 http://mirror.selectel.ru/ubuntu bionic-backports/universe Sources [5360 B]
  [   17.252199] cloud-init[994]: Get:29 http://mirror.selectel.ru/ubuntu bionic-backports/main amd64 Packages [10.0 kB]
  [   17.254677] cloud-init[994]: Get:30 http://mirror.selectel.ru/ubuntu bionic-backports/universe amd64 Packages [10.3 kB]
  [   17.475859] cloud-init[994]: Get:31 http://security.ubuntu.com/ubuntu bionic-security/main Sources [258 kB]
  [   17.849630] cloud-init[994]: Get:32 http://security.ubuntu.com/ubuntu bionic-security/restricted Sources [20.9 kB]
  [   17.964417] cloud-init[994]: Get:33 http://security.ubuntu.com/ubuntu bionic-security/universe Sources [282 kB]
  [   18.153695] cloud-init[994]: Get:34 http://security.ubuntu.com/ubuntu bionic-security/multiverse Sources [7972 B]
  [   18.262633] cloud-init[994]: Get:35 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1904 kB]
  [   18.682527] cloud-init[994]: Get:36 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [468 kB]
  [   18.815528] cloud-init[994]: Get:37 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1140 kB]
  [   18.995874] cloud-init[994]: Get:38 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [20.9 kB]
  ...
  [   25.899115] cloud-init[994]: Reading package lists...
  [   25.943450] cloud-init[994]: W: Failed to fetch https://deb.releases.teleport.dev/dists/stable/InRelease  503  Service Unavailable [IP: 13.33.246.28 443]
  [   25.945614] cloud-init[994]: W: Some index files failed to download. They have been ignored, or old ones used instead.
  [   26.004245] cloud-init[994]: Cloud-init v. 19.4 running 'modules:config' at Mon, 04 Oct 2021 15:58:08 +0000. Up 14.40 seconds.
  [   26.490598] cloud-init[1892]: Reading package lists...
  [   26.818564] cloud-init[1892]: Building dependency tree...
  [   26.820171] cloud-init[1892]: Reading state information...
  [   26.961994] cloud-init[1892]: E: Unable to locate package teleport
  [   26.965159] cloud-init[1892]: Cloud-init v. 19.4 running 'modules:final' at Mon, 04 Oct 2021 15:58:20 +0000. Up 26.27 seconds.
  [   26.972239] cloud-init[1892]: 2021-10-04 15:58:20,841 - util.py[WARNING]: Failed to install packages: ['mc', 'wget', 'jq', 'unzip', 'curl', 'xfsprogs', 'lvm2', 'teleport', 'docker-ce', 'iptables-persistent']
  [   26.979781] cloud-init[1892]: 2021-10-04 15:58:20,842 - cc_package_update_upgrade_install.py[WARNING]: 1 failed with exceptions, re-raising the last one
  [   26.982565] cloud-init[1892]: 2021-10-04 15:58:20,843 - util.py[WARNING]: Running module package-update-upgrade-install (<module 'cloudinit.config.cc_package_update_upgrade_install' from '/opt/cloud-init/lib/python3.6/site-packages/cloudinit/config/cc_package_update_upgrade_install.py'>) failed
  [   26.989141] cloud-init[1892]: Failed to enable unit: Unit file teleport.service does not exist.
  [   26.991104] cloud-init[1892]: 2021-10-04 15:58:20,868 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/50_init.cfg [1]
  [   27.535825] cloud-init[1892]: 2021-10-04 15:58:21,413 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
  [   27.539081] cloud-init[1892]: 2021-10-04 15:58:21,414 - util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/opt/cloud-init/lib/python3.6/site-packages/cloudinit/config/cc_scripts_user.py'>) failed
  ...
  [   27.566504] cloud-init[1892]: Cloud-init v. 19.4 finished at Mon, 04 Oct 2021 15:58:21 +0000. Datasource DataSourceConfigDrive [net,ver=2][source=/dev/sr0].  Up 27.55 seconds
  ```

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1946088/+subscriptions