kernel-packages team mailing list archive

[Launchpad Bug Tracker <1348688@xxxxxxxxxxxxxxxxxx>]

You have been subscribed to a public bug:

(Sorry I'm not sure exactly what package to report this against - kernel
perhaps? libvirt is what I was using to replicate the problem)

Host platform: ubuntu 14.04 amd64, Mac Mini, 16GB RAM.

Short version: create an LXC domain with memtune > swap_hard_limit set
in the XML:

<domain type='lxc'>
  <name>gold-lxc-20140717</name>
  <uuid>b2a02d49-bb1e-4aec-81d1-58910892780e</uuid>
  <memory unit='KiB'>327680</memory>
  <currentMemory unit='KiB'>327680</currentMemory>
  <memtune>
    <swap_hard_limit unit='KiB'>131072</swap_hard_limit>
  </memtune>
  ...

(full version at end of this report)

Now try to start it:

$ virsh -c lxc: start gold-lxc-20140717error: Failed to start domain gold-lxc-20140717
error: internal error: guest failed to start: Unable to write to '/sys/fs/cgroup/memory/machine/gold-lxc-20140717.libvirt-lxc/memory.memsw.limit_in_bytes': No such file or directory

The reason this matters is because otherwise the LXC memory limit
applies only to real RAM used. If the guest exceeds this it can still
use as much swap space as it likes, and is therefore effectively
unlimited (and can happily DoS the swap disk).

Long version:

I created an ubuntu 14.04 i386 VM image using python-vmbuilder,
loopback-mounted it with qemu-nbd, and rsync'd it to create a root
filesystem for an LXC guest.

Then defined a guest using libvirt XML and started it using "virsh -c
lxc: start <domain>" (as per XML at end but without the <memtune>
section). It starts successfully, networking is fine, I can get a
console etc.

Now, the libvirt XML description says the guest's memory limit is 320MB:

  <memory unit='KiB'>327680</memory>
  <currentMemory unit='KiB'>327680</currentMemory>

and indeed the cgroups setting has been set:

$ cat /sys/fs/cgroup/memory/machine/gold-lxc-20140717.libvirt-lxc/memory.limit_in_bytes
335544320

However inside the guest I can happily allocate as much memory as I
like, up to just under 4GB, which is the limit for a 32-bit guest.

Here's the test program I ran in the guest (usemem.c):

#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>

int main(void)
{
    char *p;
    int i,j;
    int ok=0, fail=0;
    for (i=0; i<4096; i++) {
        p = malloc(1024*1024);
        if (p) {
            ok++;
            for (j=0; j<1024*1024; j++)
                p[j] = rand();
        }
        else
            fail++;
    }
    fprintf(stderr, "Done: %d ok, %d fail\n", ok, fail);
    sleep(600);
    return fail ? 1 : 0;
}

Result from running:

Done: 4076 ok, 20 fail

View from the host:

nsrc@kit1:~/workshop-kit$ ps auxwww | grep usemem | grep -v grep
nsrc     10506 96.1  1.3 4192152 224776 ?      S+   14:41   0:55 ./usemem

$ cat /sys/fs/cgroup/memory/machine/gold-lxc-20140717.libvirt-lxc/memory.limit_in_bytes
335544320
$ cat /sys/fs/cgroup/memory/machine/gold-lxc-20140717.libvirt-lxc/memory.max_usage_in_bytes
335544320
$ cat /sys/fs/cgroup/memory/machine/gold-lxc-20140717.libvirt-lxc/memory.usage_in_bytes
292331520

You can see there's definitely 4GB in use by this process, and yet the
cgroup thinks less than 280MB is in use, which is below the 320MB limit.

However if you look at swap usage in the host while the memory suck
program is running:

$ free
             total       used       free     shared    buffers     cached
Mem:      16338300    3066952   13271348       1684     135512    1489268
-/+ buffers/cache:    1442172   14896128
Swap:     16678908    3971248   12707660

and after it has terminated:

$ free
             total       used       free     shared    buffers     cached
Mem:      16338300    2774440   13563860       1684     135544    1489188
-/+ buffers/cache:    1149708   15188592
Swap:     16678908       5484   16673424

i.e. the LXC guest used nearly 4GB of swap, and then gave it up when it
terminated.

Additional info:

cgroup view from inside the guest:

$ cat /proc/self/cgroup
11:name=systemd:/
10:hugetlb:/
9:perf_event:/machine/gold-lxc-20140717.libvirt-lxc
8:blkio:/machine/gold-lxc-20140717.libvirt-lxc
7:freezer:/machine/gold-lxc-20140717.libvirt-lxc
6:devices:/machine/gold-lxc-20140717.libvirt-lxc
5:memory:/machine/gold-lxc-20140717.libvirt-lxc
4:cpuacct:/machine/gold-lxc-20140717.libvirt-lxc
3:cpu:/machine/gold-lxc-20140717.libvirt-lxc
2:cpuset:/machine/gold-lxc-20140717.libvirt-lxc

cgroup settings visible in the host:

$ ls /sys/fs/cgroup/memory/machine/gold-lxc-20140717.libvirt-lxc/
cgroup.clone_children               memory.limit_in_bytes
cgroup.event_control                memory.max_usage_in_bytes
cgroup.procs                        memory.move_charge_at_immigrate
memory.failcnt                      memory.numa_stat
memory.force_empty                  memory.oom_control
memory.kmem.failcnt                 memory.pressure_level
memory.kmem.limit_in_bytes          memory.soft_limit_in_bytes
memory.kmem.max_usage_in_bytes      memory.stat
memory.kmem.slabinfo                memory.swappiness
memory.kmem.tcp.failcnt             memory.usage_in_bytes
memory.kmem.tcp.limit_in_bytes      memory.use_hierarchy
memory.kmem.tcp.max_usage_in_bytes  notify_on_release
memory.kmem.tcp.usage_in_bytes      tasks
memory.kmem.usage_in_bytes

And here's the full XML as promised:

<domain type='lxc'>
  <name>gold-lxc-20140717</name>
  <uuid>b2a02d49-bb1e-4aec-81d1-58910892780e</uuid>
  <memory unit='KiB'>327680</memory>
  <currentMemory unit='KiB'>327680</currentMemory>
  <memtune>
    <swap_hard_limit unit='KiB'>131072</swap_hard_limit>
  </memtune>
  <vcpu placement='static'>1</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64'>exe</type>
    <init>/sbin/init</init>
  </os>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/lib/libvirt/libvirt_lxc</emulator>
    <filesystem type='mount' accessmode='passthrough'>
      <source dir='/data1/lxc/gold-20140717/rootfs'/>
      <target dir='/'/>
    </filesystem>
    <interface type='bridge'>
      <mac address='52:54:5d:00:0a:88'/>
      <source bridge='br-lan'/>
    </interface>
    <console type='pty'>
      <target type='lxc' port='0'/>
    </console>
  </devices>
  <seclabel type='none'/>
</domain>

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: libvirt-bin 1.2.2-0ubuntu13.1.1
ProcVersionSignature: Ubuntu 3.13.0-32.57-generic 3.13.11.4
Uname: Linux 3.13.0-32-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.2
Architecture: amd64
Date: Fri Jul 25 14:29:54 2014
InstallationDate: Installed on 2014-07-16 (8 days ago)
InstallationMedia: Ubuntu-Server 14.04 LTS "Trusty Tahr" - Release amd64 (20140416.2)
SourcePackage: libvirt
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.libvirt.qemu.conf: [inaccessible: [Errno 13] Permission denied: '/etc/libvirt/qemu.conf']

** Affects: linux (Ubuntu)
     Importance: Medium
         Status: Confirmed


** Tags: amd64 apport-bug trusty
-- 
LXC does not limit swap usage (memory.memsw.limit_in_bytes missing)
https://bugs.launchpad.net/bugs/1348688
You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.