sts-sponsors team mailing list archive
-
sts-sponsors team
-
Mailing list archive
-
Message #02606
[Bug 1913583] Re: [plugin][k8s] Canonical Distribution of Kubernetes fixes
To test this I deployed a Focal based CDK environment, then launched a
machine running groovy and scp'ed /root from kubernetes-master/0 to that
new machine and executed sosreport, the verification executed correctly.
Here it's the evidence.
Before the patch:
root@juju-321ff4-k8s-11:~# cat sosreport-juju-321ff4-k8s-11-2021-02-21-ogpqrgd/sos_commands/kubernetes/kubectl_--kubeconfig_.root.cdk.kubeproxyconfig_get_namespaces.1
Error from server (Forbidden): namespaces is forbidden: User "system:kube-proxy" cannot list resource "namespaces" in API group "" at the cluster scope
versus
After patch:
root@juju-321ff4-k8s-11:~# cat sosreport-juju-321ff4-k8s-11-2021-02-21-lmlgkbh/sos_commands/kubernetes/kubectl_--kubeconfig_.root.cdk.cdk_addons_kubectl_config_get_namespaces.1
NAME STATUS AGE
default Active 46m
ingress-nginx-kubernetes-worker Active 43m
kube-node-lease Active 46m
kube-public Active 46m
kube-system Active 46m
kubernetes-dashboard Active 46m
$ juju add-machine --series groovy
created machine 11
$ juju ssh kubernetes-master/0 sudo -i
root@juju-321ff4-k8s-4:~# tar czf /tmp/root.tgz /root
tar: Removing leading `/' from member names
tar: /root/cdk/audit/audit.log: file changed as we read it
root@juju-321ff4-k8s-4:~# logout
Connection to 10.7.1.146 closed.
$ juju scp kubernetes-master/0:/tmp/root.tgz ./
$ juju scp root.tgz 11:
$ juju ssh 11
Welcome to Ubuntu 20.10 (GNU/Linux 5.8.0-43-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Sun Feb 21 19:02:28 UTC 2021
System load: 0.04 Processes: 98
Usage of /: 8.6% of 19.21GB Users logged in: 0
Memory usage: 12% IPv4 address for ens3: 10.7.1.51
Swap usage: 0%
0 updates can be installed immediately.
0 of these updates are security updates.
*** System restart required ***
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.
ubuntu@juju-321ff4-k8s-11:~$ ls
root.tgz
ubuntu@juju-321ff4-k8s-11:~$ sudo tar xzf root.tgz -C /
ubuntu@juju-321ff4-k8s-11:~$ sudo snap install kubectl
error: This revision of snap "kubectl" was published using classic confinement and thus may perform
arbitrary system changes outside of the security sandbox that snaps are usually confined to,
which may put your system at risk.
If you understand and want to proceed repeat the command including --classic.
ubuntu@juju-321ff4-k8s-11:~$ sudo snap install kubectl --classic
kubectl 1.20.4 from Canonical✓ installed
ubuntu@juju-321ff4-k8s-11:~$ sudo -i
root@juju-321ff4-k8s-11:~# ls cdk/
audit ca.crt client.key known_tokens.csv kubeproxyconfig rbac-proxy.yaml serviceaccount.key
auth-webhook cdk_addons_kubectl_config etcd kube-scheduler-config.yaml kubeschedulerconfig server.crt system-monitoring-rbac-role.yaml
basic_auth.csv client.crt keystone kubecontrollermanagerconfig pod-security-policy.yaml server.key
root@juju-321ff4-k8s-11:~# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
ingress-nginx-kubernetes-worker default-http-backend-kubernetes-worker-6494cbc7fd-jr7g4 1/1 Running 0 34m
ingress-nginx-kubernetes-worker nginx-ingress-controller-kubernetes-worker-jbvgh 1/1 Running 0 33m
ingress-nginx-kubernetes-worker nginx-ingress-controller-kubernetes-worker-kj8x5 1/1 Running 0 34m
kube-system coredns-7bb4d77796-q6sck 1/1 Running 0 36m
kube-system csi-cinder-controllerplugin-0 5/5 Running 0 36m
kube-system csi-cinder-nodeplugin-8bdl4 2/2 Running 0 33m
kube-system csi-cinder-nodeplugin-n825s 2/2 Running 0 34m
kube-system k8s-keystone-auth-5976c99b8b-2zx25 1/1 Running 0 36m
kube-system k8s-keystone-auth-5976c99b8b-pr9w6 1/1 Running 0 36m
kube-system kube-state-metrics-6f586bb967-f5jt7 1/1 Running 0 36m
kube-system metrics-server-v0.3.6-f6cf867b4-87dxm 2/2 Running 0 31m
kube-system openstack-cloud-controller-manager-rcsx8 1/1 Running 0 34m
kube-system openstack-cloud-controller-manager-v5cjd 1/1 Running 0 34m
kubernetes-dashboard dashboard-metrics-scraper-74757fb5b7-jzqsq 1/1 Running 0 36m
kubernetes-dashboard kubernetes-dashboard-64f87676d4-m458m 1/1 Running 0 36m
root@juju-321ff4-k8s-11:~# apt policy sosreport
sosreport:
Installed: 4.0-1ubuntu2.1
Candidate: 4.0-1ubuntu2.1
Version table:
*** 4.0-1ubuntu2.1 500
500 http://nova.clouds.archive.ubuntu.com/ubuntu groovy-updates/main amd64 Packages
100 /var/lib/dpkg/status
4.0-1ubuntu2 500
500 http://nova.clouds.archive.ubuntu.com/ubuntu groovy/main amd64 Packages
root@juju-321ff4-k8s-11:~# sosreport -o kubernetes
Please note the 'sosreport' command has been deprecated in favor of the new 'sos' command, E.G. 'sos report'.
Redirecting to 'sos report -o kubernetes'
sosreport (version 4.0)
This command will collect system configuration and diagnostic
information from this Ubuntu system.
For more information on Canonical visit:
https://www.ubuntu.com/
The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.
No changes will be made to system configuration.
Press ENTER to continue, or CTRL-C to quit.
Please enter the case id that you are generating this report for []:
Setting up archive ...
Setting up plugins ...
Running plugins. Please wait ...
Starting 1/1 kubernetes [Running: kubernetes]
Finished running plugins
Creating compressed archive...
Your sosreport has been generated and saved in:
/tmp/sosreport-juju-321ff4-k8s-11-2021-02-21-ogpqrgd.tar.xz
Size 7.55KiB
Owner root
md5 15c65efc8b615dc8e585ed5038bea51f
Please send this file to your support representative.
root@juju-321ff4-k8s-11:~# tar xJf /tmp/sosreport-juju-321ff4-k8s-11-2021-02-21-ogpqrgd.tar.xz
root@juju-321ff4-k8s-11:~# cat sosreport-juju-321ff4-k8s-11-2021-02-21-ogpqrgd/sos_commands/kubernetes/kubectl_--kubeconfig_.root.cdk.kubeproxyconfig_get_namespaces.1
Error from server (Forbidden): namespaces is forbidden: User "system:kube-proxy" cannot list resource "namespaces" in API group "" at the cluster scope
root@juju-321ff4-k8s-11:~# vim /etc/apt/sources.list
root@juju-321ff4-k8s-11:~# apt-get update -qq
root@juju-321ff4-k8s-11:~# apt policy sosreport
sosreport:
Installed: 4.0-1ubuntu2.1
Candidate: 4.0-1ubuntu2.2
Version table:
4.0-1ubuntu2.2 500
500 http://nova.clouds.archive.ubuntu.com/ubuntu groovy-proposed/main amd64 Packages
*** 4.0-1ubuntu2.1 500
500 http://nova.clouds.archive.ubuntu.com/ubuntu groovy-updates/main amd64 Packages
100 /var/lib/dpkg/status
4.0-1ubuntu2 500
500 http://nova.clouds.archive.ubuntu.com/ubuntu groovy/main amd64 Packages
root@juju-321ff4-k8s-11:~# apt-get install sosreport
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
sosreport
1 upgraded, 0 newly installed, 0 to remove and 17 not upgraded.
Need to get 238 kB of archives.
After this operation, 4096 B of additional disk space will be used.
Get:1 http://nova.clouds.archive.ubuntu.com/ubuntu groovy-proposed/main amd64 sosreport amd64 4.0-1ubuntu2.2 [238 kB]
Fetched 238 kB in 0s (5475 kB/s)
(Reading database ... 64769 files and directories currently installed.)
Preparing to unpack .../sosreport_4.0-1ubuntu2.2_amd64.deb ...
Unpacking sosreport (4.0-1ubuntu2.2) over (4.0-1ubuntu2.1) ...
Setting up sosreport (4.0-1ubuntu2.2) ...
Processing triggers for man-db (2.9.3-2) ...
root@juju-321ff4-k8s-11:~# apt policy sosreport
sosreport:
Installed: 4.0-1ubuntu2.2
Candidate: 4.0-1ubuntu2.2
Version table:
*** 4.0-1ubuntu2.2 500
500 http://nova.clouds.archive.ubuntu.com/ubuntu groovy-proposed/main amd64 Packages
100 /var/lib/dpkg/status
4.0-1ubuntu2.1 500
500 http://nova.clouds.archive.ubuntu.com/ubuntu groovy-updates/main amd64 Packages
4.0-1ubuntu2 500
500 http://nova.clouds.archive.ubuntu.com/ubuntu groovy/main amd64 Packages
root@juju-321ff4-k8s-11:~# sosreport -o kubernetes
Please note the 'sosreport' command has been deprecated in favor of the new 'sos' command, E.G. 'sos report'.
Redirecting to 'sos report -o kubernetes'
sosreport (version 4.0)
This command will collect system configuration and diagnostic
information from this Ubuntu system.
For more information on Canonical visit:
https://www.ubuntu.com/
The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.
No changes will be made to system configuration.
Press ENTER to continue, or CTRL-C to quit.
Please enter the case id that you are generating this report for []:
Setting up archive ...
Setting up plugins ...
Running plugins. Please wait ...
Starting 1/1 kubernetes [Running: kubernetes]
Finished running plugins
Creating compressed archive...
Your sosreport has been generated and saved in:
/tmp/sosreport-juju-321ff4-k8s-11-2021-02-21-lmlgkbh.tar.xz
Size 73.12KiB
Owner root
md5 9fccd12638633af0f0c7979c08c09d43
Please send this file to your support representative.
root@juju-321ff4-k8s-11:~# tar xJf /tmp/sosreport-juju-321ff4-k8s-11-2021-02-21-lmlgkbh.tar.xz
root@juju-321ff4-k8s-11:~# cat sosreport-juju-321ff4-k8s-11-2021-02-21-lmlgkbh/sos_commands/kubernetes/kubectl_--kubeconfig_.root.cdk.cdk_addons_kubectl_config_get_namespaces.1
NAME STATUS AGE
default Active 46m
ingress-nginx-kubernetes-worker Active 43m
kube-node-lease Active 46m
kube-public Active 46m
kube-system Active 46m
kubernetes-dashboard Active 46m
** Tags removed: verification-needed verification-needed-focal verification-needed-groovy
** Tags added: verification-done verification-done-focal verification-done-groovy
--
You received this bug notification because you are a member of STS
Sponsors, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1913583
Title:
[plugin][k8s] Canonical Distribution of Kubernetes fixes
Status in sosreport package in Ubuntu:
Fix Released
Status in sosreport source package in Bionic:
New
Status in sosreport source package in Focal:
Fix Committed
Status in sosreport source package in Groovy:
Fix Committed
Status in sosreport source package in Hirsute:
Fix Released
Bug description:
[Impact]
Running sosreport in a CDK deployed environment won't collect as much
information as the plugin could, this is because the kubectl calls are
using the wrong paths for the kubeconfig files, this prevents from
having more detailed sosreports on the state of the cluster which
leads to a back and forth running extra commands to collect the rest
of the data.
[Test Case]
* Deploy CDK: juju deploy charmed-kubernetes # https://ubuntu.com/kubernetes/docs/quickstart
* ssh into the kubernetes-master/0
* Run sosreport
Expected result:
The sosreport contains a 'kubernetes' directory where all the commands
executed successfully
Actual result:
The sosreport contains a 'kubernetes' directory where some of the
commands contain "Forbidden" errors.
find sosreport-*/ -type d -name kubernetes -exec grep -H -i forbidden
{} \;
[Where problems could occur]
Any issues with this SRU should show themselves as failures in the
execution of the kubernetes plugin and that can be verified in the
sos.log file.
[Other Info]
Upstream:
https://github.com/sosreport/sos/pull/2387
https://github.com/sosreport/sos/pull/2387/commits
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sosreport/+bug/1913583/+subscriptions
Follow ups
References
