sts-sponsors team mailing list archive
-
sts-sponsors team
-
Mailing list archive
-
Message #01704
[Bug 1860813] Re: LXC container reports spike in swap occasionally
Revised debdiff, test information for 18.04/bionic
Autopkgtest:
autopkgtest [16:25:54]: test exercise: -----------------------]
autopkgtest [16:25:54]: test exercise: - - - - - - - - - - results - - - - - - - - - -
exercise PASS
autopkgtest [16:25:54]: @@@@@@@@@@@@@@@@@@@@ summary
exercise PASS
Manual testing:
ubuntu@test-lxcfs-bionic:~$ dpkg -l | grep lxcfs
ii lxcfs 3.0.3-0ubuntu1~18.04.1 amd64 FUSE based filesystem for LXC
ubuntu@test-lxcfs-bionic:~$ lxc launch ubuntu:b test1
ubuntu@test-lxcfs-bionic:~$ lxc shell test1 #opened two in separate sessions
Session 1
root@test1:~# watch -n 0.1 "grep Swap /proc/meminfo"
Session 2
root@test1:~# while true; do free | grep Swap; done
...
Swap: 2097148 0 2097148
Swap: 2097148 2097148 0
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 2097148 0
Swap: 2097148 0 2097148
...
ubuntu@test-lxcfs-bionic:~$ lxc stop test1
ubuntu@test-lxcfs-bionic:~$ sudo apt install ./lxcfs_3.0.3-0ubuntu1~18.04.2_amd64.deb
Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'lxcfs' instead of './lxcfs_3.0.3-0ubuntu1~18.04.2_amd64.deb'
The following packages will be upgraded:
lxcfs
1 upgraded, 0 newly installed, 0 to remove and 6 not upgraded.
Need to get 0 B/44.2 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 /home/ubuntu/lxcfs_3.0.3-0ubuntu1~18.04.2_amd64.deb lxcfs amd64 3.0.3-0ubuntu1~18.04.2 [44.2 kB]
(Reading database ... 66998 files and directories currently installed.)
Preparing to unpack .../lxcfs_3.0.3-0ubuntu1~18.04.2_amd64.deb ...
Unpacking lxcfs (3.0.3-0ubuntu1~18.04.2) over (3.0.3-0ubuntu1~18.04.1) ...
Setting up lxcfs (3.0.3-0ubuntu1~18.04.2) ...
Processing triggers for ureadahead (0.100.0-21) ...
Processing triggers for systemd (237-3ubuntu10.39) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
ubuntu@test-lxcfs-bionic:~$ lxc start test1
ubuntu@test-lxcfs-bionic:~$ lxc shell test1 #opened two in separate sessions
Session 1
root@test1:~# watch -n 0.1 "grep Swap /proc/meminfo"
Session 2
root@test1:~# while true; do free | grep Swap; done
...
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
Swap: 2097148 0 2097148
...
Issue doesn't occur
** Patch added: "lp1860813_bionic.debdiff"
https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1860813/+attachment/5343837/+files/lp1860813_bionic.debdiff
** Patch removed: "lp1860813_bionic.debdiff"
https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1860813/+attachment/5342354/+files/lp1860813_bionic.debdiff
--
You received this bug notification because you are a member of STS
Sponsors, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1860813
Title:
LXC container reports spike in swap occasionally
Status in lxcfs package in Ubuntu:
Fix Released
Status in lxcfs source package in Xenial:
Invalid
Status in lxcfs source package in Bionic:
In Progress
Status in lxcfs source package in Eoan:
In Progress
Bug description:
[Impact]
* lxcfs provides a container-specific view of /proc/meminfo.
Occasionally, with near zero or zero swap usage *and* swap accounting
turned on (kernel parameter swapaccount=1), the container will report
100% swap utilization.
* This issue has been encountered and could result in unecessary
alerts or potential automated attempts at remediating a non-existent
"full swap" issue.
* This fix changed the logic used for SwapFree when swap accounting
is enabled to better handle situations where memswusage is less than
memusage, caused by the fuzziness of the usage_in_bytes counters used
as the source. Specifically, it added a check for memusage being
larger than memswusage and if so, sets 0 as the value of swapusage.
Otherwise the calculation (memswusage - memusage) remains the same.
[Test Case]
* Requires a Bionic (18.04) or Eoan (19.10) host with swap space.
* Enable swap accounting with the "swapaccount=1" kernel parameter on
the kernel command line. Edit /etc/default/grub, add "swapaccount=1"
to the GRUB_CMDLINE_LINUX_DEFAULT="other parameters" line, then run
"update-grub" and reboot to make the change active.
* Ensure lxd is installed, "sudo apt install lxd"
* Create a lxd/lxc container with "lxc launch ubuntu:X
container_name" with X being either b[ionic] or e[oan].
* Open two shells to the container with "lxc shell container_name"
* In one of the shells, run: watch -n 0.1 "grep Swap /proc/meminfo"
* In the other, run: while true; do free; done
* You should see SwapFree intermittently drop to zero in the first
terminal.
* The fix results in small non-zero swap "usage" intermittently
instead of intermittent SwapFree = 0.
[Regression Potential]
* Low, as swap accounting must be enabled to encounter the bug and
the fix.
* Potential for unanticipated edge cases in the values of memusage
and memswusage to cause incorrect swap reporting within the container,
with swap accounting turned on.
* Any tooling that expected, compensated, or relied on the behavior
may no longer work as expected.
[Other Info]
* Cherrypick of a one line fix to address this specific situation.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1860813/+subscriptions
