kernel-packages team mailing list archive

[Peter Waller <p@xxxxxxxxxxx>]

Public bug reported:

This has happened twice now.

I'm on an AWS EC2 m3.large instance with the official Ubuntu AMI ami-
776d9700.

# cat /proc/version_signature
Ubuntu 3.13.0-32.57-generic 3.13.11.4

After running for many days, the machine locked up with the below
messages appearing on the console. The machine would respond to ping but
not SSH or HTTP requests. The machine has one BTRFS volume which is 87%
full and lives on an Logical Volume Manager (LVM) block device on top of
one Amazon Elastic Block Store (EBS) device.

Error messages after first reboot:

[   77.609490] BTRFS error (device dm-0): block group 10766778368 has wrong amount of free space
[   77.613678] BTRFS error (device dm-0): failed to load free space cache for block group 10766778368
[   77.643801] BTRFS error (device dm-0): block group 19356712960 has wrong amount of free space
[   77.648952] BTRFS error (device dm-0): failed to load free space cache for block group 19356712960
[   77.926325] BTRFS error (device dm-0): block group 20430454784 has wrong amount of free space
[   77.931078] BTRFS error (device dm-0): failed to load free space cache for block group 20430454784
[   78.111437] BTRFS error (device dm-0): block group 21504196608 has wrong amount of free space
[   78.116165] BTRFS error (device dm-0): failed to load free space cache for block group 21504196608

Error messages after second reboot:

[   45.390221] BTRFS error (device dm-0): free space inode generation (0) did not match free space cache generation (70012)
[   45.413472] BTRFS error (device dm-0): free space inode generation (0) did not match free space cache generation (70012)
[  467.423961] BTRFS error (device dm-0): block group 518646661120 has wrong amount of free space
[  467.429251] BTRFS error (device dm-0): failed to load free space cache for block group 518646661120

Error messages on the console after second lock-up follow:

[246736.752053] INFO: rcu_sched self-detected stall on CPU { 0}  (t=2220246 jiffies g=35399662 c=35399661 q=0)
[246736.756059] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2220247 jiffies, g=35399662, c=35399661, q=0)
[246764.192014] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u30:2:1828]
[246764.212058] BUG: soft lockup - CPU#1 stuck for 23s! [btrfs-transacti:492]
[246792.192022] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u30:2:1828]
[246792.212057] BUG: soft lockup - CPU#1 stuck for 23s! [btrfs-transacti:492]
[246820.192052] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u30:2:1828]
[246820.212018] BUG: soft lockup - CPU#1 stuck for 23s! [btrfs-transacti:492]
[246848.192052] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u30:2:1828]
[246848.212058] BUG: soft lockup - CPU#1 stuck for 23s! [btrfs-transacti:492]
[246876.192053] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u30:2:1828]
[246876.212057] BUG: soft lockup - CPU#1 stuck for 22s! [btrfs-transacti:492]
[246904.192053] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u30:2:1828]
[246904.212058] BUG: soft lockup - CPU#1 stuck for 22s! [btrfs-transacti:492]
[246916.772052] INFO: rcu_sched self-detected stall on CPU[246916.776058] INFO: rcu_sched detected stalls on CPUs/tasks:
[246944.192053] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u30:2:1828]
[246944.212058] BUG: soft lockup - CPU#1 stuck for 22s! [btrfs-transacti:492]
[246972.192053] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u30:2:1828]
[246972.212018] BUG: soft lockup - CPU#1 stuck for 22s! [btrfs-transacti:492]
[247000.192053] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u30:2:1828]
[247000.212058] BUG: soft lockup - CPU#1 stuck for 22s! [btrfs-transacti:492]
[247028.192054] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u30:2:1828]
[247028.212058] BUG: soft lockup - CPU#1 stuck for 22s! [btrfs-transacti:492]
[247056.192053] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u30:2:1828]
[247056.212061] BUG: soft lockup - CPU#1 stuck for 23s! [btrfs-transacti:492]

** Affects: linux (Ubuntu)
     Importance: Undecided
         Status: Incomplete


** Tags: btrfs btrfs-transaction linux soft-lockup

** Tags added: soft-lockup

** Tags added: btrfs-transaction

** Description changed:

  This has happened twice now.
  
  I'm on an AWS EC2 m3.large instance with the official Ubuntu AMI ami-
  776d9700.
  
- # cat /proc/version_signature 
+ # cat /proc/version_signature
  Ubuntu 3.13.0-32.57-generic 3.13.11.4
  
  After running for many days, the machine locked up with the below
- messages appearing in the terminal. The machine would respond to ping
- but not SSH or HTTP requests. The machine has one BTRFS volume which is
- 87% full and lives on an Logical Volume Manager (LVM) block device on
- top of one Amazon Elastic Block Store (EBS) device.
+ messages appearing on the console. The machine would respond to ping but
+ not SSH or HTTP requests. The machine has one BTRFS volume which is 87%
+ full and lives on an Logical Volume Manager (LVM) block device on top of
+ one Amazon Elastic Block Store (EBS) device.
  
  Error messages after first reboot:
  
  [   77.609490] BTRFS error (device dm-0): block group 10766778368 has wrong amount of free space
  [   77.613678] BTRFS error (device dm-0): failed to load free space cache for block group 10766778368
  [   77.643801] BTRFS error (device dm-0): block group 19356712960 has wrong amount of free space
  [   77.648952] BTRFS error (device dm-0): failed to load free space cache for block group 19356712960
  [   77.926325] BTRFS error (device dm-0): block group 20430454784 has wrong amount of free space
  [   77.931078] BTRFS error (device dm-0): failed to load free space cache for block group 20430454784
  [   78.111437] BTRFS error (device dm-0): block group 21504196608 has wrong amount of free space
  [   78.116165] BTRFS error (device dm-0): failed to load free space cache for block group 21504196608
  
  Error messages after second reboot:
  
  [   45.390221] BTRFS error (device dm-0): free space inode generation (0) did not match free space cache generation (70012)
  [   45.413472] BTRFS error (device dm-0): free space inode generation (0) did not match free space cache generation (70012)
  [  467.423961] BTRFS error (device dm-0): block group 518646661120 has wrong amount of free space
  [  467.429251] BTRFS error (device dm-0): failed to load free space cache for block group 518646661120
  
  Error messages on the console after second lock-up follow:
  
  [246736.752053] INFO: rcu_sched self-detected stall on CPU { 0}  (t=2220246 jiffies g=35399662 c=35399661 q=0)
  [246736.756059] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2220247 jiffies, g=35399662, c=35399661, q=0)
  [246764.192014] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u30:2:1828]
  [246764.212058] BUG: soft lockup - CPU#1 stuck for 23s! [btrfs-transacti:492]
  [246792.192022] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u30:2:1828]
  [246792.212057] BUG: soft lockup - CPU#1 stuck for 23s! [btrfs-transacti:492]
  [246820.192052] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u30:2:1828]
  [246820.212018] BUG: soft lockup - CPU#1 stuck for 23s! [btrfs-transacti:492]
  [246848.192052] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u30:2:1828]
  [246848.212058] BUG: soft lockup - CPU#1 stuck for 23s! [btrfs-transacti:492]
  [246876.192053] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u30:2:1828]
  [246876.212057] BUG: soft lockup - CPU#1 stuck for 22s! [btrfs-transacti:492]
  [246904.192053] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u30:2:1828]
  [246904.212058] BUG: soft lockup - CPU#1 stuck for 22s! [btrfs-transacti:492]
  [246916.772052] INFO: rcu_sched self-detected stall on CPU[246916.776058] INFO: rcu_sched detected stalls on CPUs/tasks:
  [246944.192053] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u30:2:1828]
  [246944.212058] BUG: soft lockup - CPU#1 stuck for 22s! [btrfs-transacti:492]
  [246972.192053] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u30:2:1828]
  [246972.212018] BUG: soft lockup - CPU#1 stuck for 22s! [btrfs-transacti:492]
  [247000.192053] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u30:2:1828]
  [247000.212058] BUG: soft lockup - CPU#1 stuck for 22s! [btrfs-transacti:492]
  [247028.192054] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u30:2:1828]
  [247028.212058] BUG: soft lockup - CPU#1 stuck for 22s! [btrfs-transacti:492]
  [247056.192053] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u30:2:1828]
  [247056.212061] BUG: soft lockup - CPU#1 stuck for 23s! [btrfs-transacti:492]

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349711

Title:
  Machine lockup in btrfs-transaction

Status in “linux” package in Ubuntu:
  Incomplete

Bug description:
  This has happened twice now.

  I'm on an AWS EC2 m3.large instance with the official Ubuntu AMI ami-
  776d9700.

  # cat /proc/version_signature
  Ubuntu 3.13.0-32.57-generic 3.13.11.4

  After running for many days, the machine locked up with the below
  messages appearing on the console. The machine would respond to ping
  but not SSH or HTTP requests. The machine has one BTRFS volume which
  is 87% full and lives on an Logical Volume Manager (LVM) block device
  on top of one Amazon Elastic Block Store (EBS) device.

  Error messages after first reboot:

  [   77.609490] BTRFS error (device dm-0): block group 10766778368 has wrong amount of free space
  [   77.613678] BTRFS error (device dm-0): failed to load free space cache for block group 10766778368
  [   77.643801] BTRFS error (device dm-0): block group 19356712960 has wrong amount of free space
  [   77.648952] BTRFS error (device dm-0): failed to load free space cache for block group 19356712960
  [   77.926325] BTRFS error (device dm-0): block group 20430454784 has wrong amount of free space
  [   77.931078] BTRFS error (device dm-0): failed to load free space cache for block group 20430454784
  [   78.111437] BTRFS error (device dm-0): block group 21504196608 has wrong amount of free space
  [   78.116165] BTRFS error (device dm-0): failed to load free space cache for block group 21504196608

  Error messages after second reboot:

  [   45.390221] BTRFS error (device dm-0): free space inode generation (0) did not match free space cache generation (70012)
  [   45.413472] BTRFS error (device dm-0): free space inode generation (0) did not match free space cache generation (70012)
  [  467.423961] BTRFS error (device dm-0): block group 518646661120 has wrong amount of free space
  [  467.429251] BTRFS error (device dm-0): failed to load free space cache for block group 518646661120

  Error messages on the console after second lock-up follow:

  [246736.752053] INFO: rcu_sched self-detected stall on CPU { 0}  (t=2220246 jiffies g=35399662 c=35399661 q=0)
  [246736.756059] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2220247 jiffies, g=35399662, c=35399661, q=0)
  [246764.192014] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u30:2:1828]
  [246764.212058] BUG: soft lockup - CPU#1 stuck for 23s! [btrfs-transacti:492]
  [246792.192022] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u30:2:1828]
  [246792.212057] BUG: soft lockup - CPU#1 stuck for 23s! [btrfs-transacti:492]
  [246820.192052] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u30:2:1828]
  [246820.212018] BUG: soft lockup - CPU#1 stuck for 23s! [btrfs-transacti:492]
  [246848.192052] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u30:2:1828]
  [246848.212058] BUG: soft lockup - CPU#1 stuck for 23s! [btrfs-transacti:492]
  [246876.192053] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u30:2:1828]
  [246876.212057] BUG: soft lockup - CPU#1 stuck for 22s! [btrfs-transacti:492]
  [246904.192053] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u30:2:1828]
  [246904.212058] BUG: soft lockup - CPU#1 stuck for 22s! [btrfs-transacti:492]
  [246916.772052] INFO: rcu_sched self-detected stall on CPU[246916.776058] INFO: rcu_sched detected stalls on CPUs/tasks:
  [246944.192053] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u30:2:1828]
  [246944.212058] BUG: soft lockup - CPU#1 stuck for 22s! [btrfs-transacti:492]
  [246972.192053] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u30:2:1828]
  [246972.212018] BUG: soft lockup - CPU#1 stuck for 22s! [btrfs-transacti:492]
  [247000.192053] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u30:2:1828]
  [247000.212058] BUG: soft lockup - CPU#1 stuck for 22s! [btrfs-transacti:492]
  [247028.192054] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u30:2:1828]
  [247028.212058] BUG: soft lockup - CPU#1 stuck for 22s! [btrfs-transacti:492]
  [247056.192053] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u30:2:1828]
  [247056.212061] BUG: soft lockup - CPU#1 stuck for 23s! [btrfs-transacti:492]

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1349711/+subscriptions