dx-packages team mailing list archive

[Thomas Bechtold <2002659@xxxxxxxxxxxxxxxxxx>]

** Package changed: compiz-plugins-main (Ubuntu) => cloud-images

** Changed in: cloud-images
       Status: New => Confirmed

** Changed in: cloud-images
     Assignee: (unassigned) => Thomas Bechtold (toabctl)

** Changed in: cloud-images
   Importance: Undecided => High

** Changed in: cloud-images
       Status: Confirmed => In Progress

-- 
You received this bug notification because you are a member of DX
Packages, which is subscribed to compiz-plugins-main in Ubuntu.
Matching subscriptions: dx-packages
https://bugs.launchpad.net/bugs/2002659

Title:
  Ubuntu AMI (ami-06114b38b9273f7c2) failed to join cluster in UAE
  region due to 403 on pause container

Status in cloud-images:
  In Progress

Bug description:
  Customer is working on a POC to test EKS in the me-central-1 region
  and they shared EC2 instances based of the Ubuntu EKS Optimized AMI
  failed to join cluster when using managed node groups.

  I've been able to repeat and identify the issue with the following
  steps:

  1. Created a new EKS cluster in me-central-1 with the following
  cluster configuration:

  ---
  apiVersion: eksctl.io/v1alpha5
  kind: ClusterConfig

  metadata:
    name: uae-poc-test
    region: me-central-1

  managedNodeGroups:
    - name: custom-ng-2
      minSize: 1
      maxSize: 4
      amiFamily: Ubuntu2004

  2. The CloudFormation stack rolls back due to Ubuntu node unable to
  join cluster. It used this AMI: ami-06114b38b9273f7c2.

  3. Looking the Cloud init logs we can see the following 403 error on
  the pause container:

  Cloud-init v. 22.4.2-0ubuntu0~20.04.2 running 'modules:config' at Wed, 11 Jan 2023 14:43:31 +0000. Up 40.30 seconds.
  eksctl: running /etc/eks/bootstrap
  Aliasing EKS k8s snap commands
  Added:
    - kubelet-eks.kubelet as kubelet
  Added:
    - kubectl-eks.kubectl as kubectl
  Stopping k8s daemons until configured
  Stopped.
  Cluster "kubernetes" set.
  Container runtime is containerd
  Attempt 5 of 5
  ctr: failed to resolve reference "602401143452.dkr.ecr.me-central-1.amazonaws.com/eks/pause:3.5": pulling from host 602401143452.dkr.ecr.me-central-1.amazonaws.com failed with status code [manifests 3.5]:403 Forbidden

  Based on the Amazon container image registries
  (https://docs.aws.amazon.com/eks/latest/userguide/add-ons-
  images.html), it looks like it's using the wrong AWS region ECR
  registry as by specifying the AMI used by the managed node group and
  overriding --pause-container-account in the bootstrap command as per
  the below configuration, the node registers as expected.

  ---
  apiVersion: eksctl.io/v1alpha5
  kind: ClusterConfig

  metadata:
    name: uae-poc-test
    region: me-central-1

  managedNodeGroups:
    - name: custom-ng-3
      ami: ami-06114b38b9273f7c2
      minSize: 1
      maxSize: 4
      overrideBootstrapCommand: |
        #!/bin/bash
        /etc/eks/bootstrap.sh <cluster> --pause-container-account 759879836304

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-images/+bug/2002659/+subscriptions