kernel-packages team mailing list archive
-
kernel-packages team
-
Mailing list archive
-
Message #08240
[Bug 1214085] Re: LXC + BTRFS: creating many containers simultaneously can cause system corruption
Additional note: I've found the problem manifests more easily with the
following modified test script (this one forks all LXC clone/start
operations immediately):
test.sh:
#!/usr/bin/env bash
prefix=$1
test -z "${prefix}" && echo 'error: missing required parameter: prefix' 1>&2 && exit 1
path=/mnt/test
sudo lxc-destroy -n c1 2>/dev/null
sudo lxc-create -t ubuntu -B btrfs -n c1
for i in `seq 1 20`; do
echo $(sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i; sudo lxc-start -d -n $prefix$i) &
done
sleep 10
# Create even more.
for i in `seq 21 40`; do
echo $(sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i; sudo lxc-start -d -n $prefix$i) &
done
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1214085
Title:
LXC + BTRFS: creating many containers simultaneously can cause system
corruption
Status in “linux” package in Ubuntu:
Confirmed
Bug description:
Using lxc with a btrfs filesystem can lead to system corruption and
containers which are unable to boot when many containers are created
and started simultaneously.
Packages involved:
- btrfs-tools
- ubuntu-lxc/daily from http://ppa.launchpad.net/ubuntu-lxc/daily/ubuntu
`lsb_release -rd`:
ubuntu@ip-10-91-30-131:~$ lsb_release -rd
Description: Ubuntu 12.04.2 LTS
Release: 12.04
Here is the relevant discussion thread: http://www.mail-archive.com
/lxc-users@xxxxxxxxxxxxxxxxxxxxx/msg05472.html
And here is a key excerpt regarding reproducing the problem:
The general system state is something like:
N containers already running happily
Launch N+ more containers in rapid succession (in parallell, not serially).
Here is a test script which closely reflects what my application is
actually doing. It slowly launches 10 containers and then uses "&" to
rapidly fork an additional 10 clone/start operations. I have it doing
2 cycles of this, and running the script many times (2-6 times)
eventually triggers the problem.
test.sh:
#!/usr/bin/env bash
prefix=$1
test -z "${prefix}" && echo 'error: missing required parameter:
prefix' 1>&2 && exit 1
path=/mnt
sudo lxc-destroy -n c1 2>/dev/null
sudo lxc-create -t ubuntu -B btrfs -n c1
for i in `seq 1 10`; do
sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i
sudo lxc-start -d -n $prefix$i
done
for i in `seq 11 20`; do
echo $(sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i; sudo lxc-start -d -n $prefix$i) &
done
sleep 10
# Create even more.
for i in `seq 21 30`; do
sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i
sudo lxc-start -d -n $prefix$i
done
for i in `seq 31 40`; do
echo $(sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i; sudo lxc-start -d -n $prefix$i) &
done
stop.sh:
#!/usr/bin/env bash
prefix=$1
test -z "${prefix}" && echo 'error: missing required parameter:
prefix' 1>&2 && exit 1
sudo lxc-destroy -n c1;
for i in `seq 1 40`; do
echo $(sudo lxc-stop -k -n $prefix$i; sudo lxc-destroy -n $prefix$i) &
done
bash ./test.sh x
bash ./test.sh y
bash ./test.sh z
If it doesn't manifest at first, try starting/stopping/destroying
varying quantities of containers for several cycles. Eventually I
consistently end up not ever getting ip addresses or being able to
even successfully start a container:
x1 RUNNING - - NO
x10 RUNNING - - NO
x11 RUNNING - - NO
x12 RUNNING - - NO
x13 RUNNING - - NO
x14 RUNNING - - NO
x15 RUNNING - - NO
x16 RUNNING - - NO
x17 RUNNING - - NO
x18 RUNNING - - NO
x19 RUNNING - - NO
x2 RUNNING - - NO
x20 RUNNING - - NO
x21 RUNNING - - NO
x22 RUNNING - - NO
x23 RUNNING - - NO
x24 RUNNING - - NO
x25 RUNNING - - NO
x26 RUNNING - - NO
x27 RUNNING - - NO
x28 RUNNING - - NO
x29 RUNNING - - NO
x3 RUNNING - - NO
x30 RUNNING - - NO
x31 RUNNING - - NO
x32 RUNNING - - NO
x33 RUNNING - - NO
x34 RUNNING - - NO
x35 RUNNING - - NO
x36 RUNNING - - NO
x37 RUNNING - - NO
x38 RUNNING - - NO
x39 RUNNING - - NO
x4 RUNNING - - NO
x40 RUNNING - - NO
x5 RUNNING - - NO
x6 RUNNING - - NO
x7 RUNNING - - NO
x8 RUNNING - - NO
x9 RUNNING - - NO
ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image-3.2.0-40-virtual 3.2.0-40.64
ProcVersionSignature: Ubuntu 3.2.0-40.64-virtual 3.2.40
Uname: Linux 3.2.0-40-virtual x86_64
AcpiTables:
AlsaDevices:
total 0
crw-rw---T 1 root audio 116, 1 Aug 14 20:40 seq
crw-rw---T 1 root audio 116, 33 Aug 14 20:40 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.0.1-0ubuntu17.3
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
Date: Mon Aug 19 18:20:19 2013
Ec2AMI: ami-d0f89fb9
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: us-east-1a
Ec2InstanceType: m1.large
Ec2Kernel: aki-88aa75e1
Ec2Ramdisk: unavailable
Lspci:
Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
MarkForUpload: True
PciMultimedia:
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB:
ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0
RelatedPackageVersions:
linux-restricted-modules-3.2.0-40-virtual N/A
linux-backports-modules-3.2.0-40-virtual N/A
linux-firmware 1.79.4
RfKill: Error: [Errno 2] No such file or directory
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
WifiSyslog:
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1214085/+subscriptions
References
