← Back to team overview

kernel-packages team mailing list archive

[Bug 1214085] Re: LXC + BTRFS: creating many containers simultaneously can cause system corruption

 

I only started using lxc+btrfs in the past few months, so unfortunately
I have no information about the behavior of any previous kernels.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1214085

Title:
  LXC + BTRFS: creating many containers simultaneously can cause system
  corruption

Status in “linux” package in Ubuntu:
  Confirmed

Bug description:
  Using lxc with a btrfs filesystem can lead to system corruption and
  containers which are unable to boot when many containers are created
  and started simultaneously.

  Packages involved:

      - btrfs-tools
      - ubuntu-lxc/daily from http://ppa.launchpad.net/ubuntu-lxc/daily/ubuntu

  `lsb_release -rd`:

      ubuntu@ip-10-91-30-131:~$ lsb_release -rd
      Description:	Ubuntu 12.04.2 LTS
      Release:	12.04

  Here is the relevant discussion thread: http://www.mail-archive.com
  /lxc-users@xxxxxxxxxxxxxxxxxxxxx/msg05472.html

  And here is a key excerpt regarding reproducing the problem:

  The general system state is something like:
  N containers already running happily
  Launch N+ more containers in rapid succession (in parallell, not serially).

  Here is a test script which closely reflects what my application is
  actually doing.  It slowly launches 10 containers and then uses "&" to
  rapidly fork an additional 10 clone/start operations.  I have it doing
  2 cycles of this, and running the script many times (2-6 times)
  eventually triggers the problem.

  test.sh:
  #!/usr/bin/env bash

  prefix=$1

  test -z "${prefix}" && echo 'error: missing required parameter:
  prefix' 1>&2 && exit 1

  path=/mnt

  sudo lxc-destroy -n c1 2>/dev/null
  sudo lxc-create -t ubuntu -B btrfs -n c1

  for i in `seq 1 10`; do
      sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i
      sudo lxc-start -d -n $prefix$i
  done
  for i in `seq 11 20`; do
      echo $(sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i; sudo lxc-start -d -n $prefix$i) &
  done

  sleep 10

  # Create even more.
  for i in `seq 21 30`; do
      sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i
      sudo lxc-start -d -n $prefix$i
  done
  for i in `seq 31 40`; do
      echo $(sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i; sudo lxc-start -d -n $prefix$i) &
  done

  stop.sh:
  #!/usr/bin/env bash

  prefix=$1

  test -z "${prefix}" && echo 'error: missing required parameter:
  prefix' 1>&2 && exit 1

  sudo lxc-destroy -n c1;

  for i in `seq 1 40`; do
      echo $(sudo lxc-stop -k -n $prefix$i; sudo lxc-destroy -n $prefix$i) &
  done

  bash ./test.sh x
  bash ./test.sh y
  bash ./test.sh z

  If it doesn't manifest at first, try starting/stopping/destroying
  varying quantities of containers for several cycles.  Eventually I
  consistently end up not ever getting ip addresses or being able to
  even successfully start a container:

  x1                          RUNNING  -     -     NO
  x10                         RUNNING  -     -     NO
  x11                         RUNNING  -     -     NO
  x12                         RUNNING  -     -     NO
  x13                         RUNNING  -     -     NO
  x14                         RUNNING  -     -     NO
  x15                         RUNNING  -     -     NO
  x16                         RUNNING  -     -     NO
  x17                         RUNNING  -     -     NO
  x18                         RUNNING  -     -     NO
  x19                         RUNNING  -     -     NO
  x2                          RUNNING  -     -     NO
  x20                         RUNNING  -     -     NO
  x21                         RUNNING  -     -     NO
  x22                         RUNNING  -     -     NO
  x23                         RUNNING  -     -     NO
  x24                         RUNNING  -     -     NO
  x25                         RUNNING  -     -     NO
  x26                         RUNNING  -     -     NO
  x27                         RUNNING  -     -     NO
  x28                         RUNNING  -     -     NO
  x29                         RUNNING  -     -     NO
  x3                          RUNNING  -     -     NO
  x30                         RUNNING  -     -     NO
  x31                         RUNNING  -     -     NO
  x32                         RUNNING  -     -     NO
  x33                         RUNNING  -     -     NO
  x34                         RUNNING  -     -     NO
  x35                         RUNNING  -     -     NO
  x36                         RUNNING  -     -     NO
  x37                         RUNNING  -     -     NO
  x38                         RUNNING  -     -     NO
  x39                         RUNNING  -     -     NO
  x4                          RUNNING  -     -     NO
  x40                         RUNNING  -     -     NO
  x5                          RUNNING  -     -     NO
  x6                          RUNNING  -     -     NO
  x7                          RUNNING  -     -     NO
  x8                          RUNNING  -     -     NO
  x9                          RUNNING  -     -     NO

  ProblemType: Bug
  DistroRelease: Ubuntu 12.04
  Package: linux-image-3.2.0-40-virtual 3.2.0-40.64
  ProcVersionSignature: Ubuntu 3.2.0-40.64-virtual 3.2.40
  Uname: Linux 3.2.0-40-virtual x86_64
  AcpiTables:

  AlsaDevices:
   total 0
   crw-rw---T 1 root audio 116,  1 Aug 14 20:40 seq
   crw-rw---T 1 root audio 116, 33 Aug 14 20:40 timer
  AplayDevices: Error: [Errno 2] No such file or directory
  ApportVersion: 2.0.1-0ubuntu17.3
  Architecture: amd64
  ArecordDevices: Error: [Errno 2] No such file or directory
  AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
  CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
  Date: Mon Aug 19 18:20:19 2013
  Ec2AMI: ami-d0f89fb9
  Ec2AMIManifest: (unknown)
  Ec2AvailabilityZone: us-east-1a
  Ec2InstanceType: m1.large
  Ec2Kernel: aki-88aa75e1
  Ec2Ramdisk: unavailable
  Lspci:

  Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
  MarkForUpload: True
  PciMultimedia:

  ProcEnviron:
   TERM=xterm-256color
   PATH=(custom, no user)
   LANG=en_US.UTF-8
   SHELL=/bin/bash
  ProcFB:

  ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0
  RelatedPackageVersions:
   linux-restricted-modules-3.2.0-40-virtual N/A
   linux-backports-modules-3.2.0-40-virtual  N/A
   linux-firmware                            1.79.4
  RfKill: Error: [Errno 2] No such file or directory
  SourcePackage: linux
  UpgradeStatus: No upgrade log present (probably fresh install)
  WifiSyslog:

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1214085/+subscriptions


References