kernel-packages team mailing list archive

[Thomas Mayer <1559723@xxxxxxxxxxxxxxxxxx>]

I rebooted the system in the meantime. It now had Kernel 4.2.0-30.36
from previous updates:

cat /proc/version_signature
Ubuntu 4.2.0-30.36~14.04.1-generic 4.2.8-ckt3

During the reboot, the system performed some disk I/O for several hours.
Before services like sshd became available. I guess this disk operation
happened before the file system was mounted in read-write mode.

What I can say is that the system tried to continue the failed balance
operation during the boot (from syslog) without intervention from my
side:

BTRFS info (device dm-0): continuing balance
BTRFS info (device dm-0): relocating block group 9553211031552 flags 36
found 106150 extents

The btrfs balance operation did not succeed again, but in a manner which
I was used to before writing this bug report. This is MUCH better than
going into an unpredictable state (ro mode with Input/Output errors). I
guess the OS totally got out of sync.

BTRFS info (device dm-0): 2 enospc errors during balance

The btrfs file system was mounted in READ-WRITE-mode this time (not in
read-only mode as before!). And this time, I can read the files and
navigate through directories without any Input/Output errors (before the
reboot, I had lots of them!).

Finally, everything looks fine for a btrfs file system with no space
left on the disk.

Now that the fs is online and accessible, I can try to add another disk
to the fs to have some empty space again. After that, I hopefully can
delete some data and hopefully redo the btrfs balance operation.

/var/log/syslog gives me:

Mar 22 12:42:37 server kernel: [ 3481.725273] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 22 12:42:37 server kernel: [ 3481.726122] mount           D ffff88043e296640     0   587    455 0x00000000
Mar 22 12:42:37 server kernel: [ 3481.726125]  ffff88042798f6a8 0000000000000086 ffff88042b98b700 ffff880427946e00
Mar 22 12:42:37 server kernel: [ 3481.726127]  ffff88042798f788 ffff880427990000 0000000000000000 7fffffffffffffff
Mar 22 12:42:37 server kernel: [ 3481.726129]  ffff88043e5aeee8 ffffffff817b9240 ffff88042798f6c8 ffffffff817b8ac7
Mar 22 12:42:37 server kernel: [ 3481.726130] Call Trace:
Mar 22 12:42:37 server kernel: [ 3481.726136]  [<ffffffff817b9240>] ? bit_wait+0x60/0x60
Mar 22 12:42:37 server kernel: [ 3481.726138]  [<ffffffff817b8ac7>] schedule+0x37/0x80
Mar 22 12:42:37 server kernel: [ 3481.726140]  [<ffffffff817bb4a1>] schedule_timeout+0x201/0x2a0
Mar 22 12:42:37 server kernel: [ 3481.726143]  [<ffffffff8101d749>] ? read_tsc+0x9/0x10
Mar 22 12:42:37 server kernel: [ 3481.726146]  [<ffffffff810e6e5e>] ? ktime_get+0x3e/0xa0
Mar 22 12:42:37 server kernel: [ 3481.726147]  [<ffffffff817b9240>] ? bit_wait+0x60/0x60
Mar 22 12:42:37 server kernel: [ 3481.726149]  [<ffffffff817b8136>] io_schedule_timeout+0xa6/0x110
Mar 22 12:42:37 server kernel: [ 3481.726151]  [<ffffffff817b925f>] bit_wait_io+0x1f/0x70
Mar 22 12:42:37 server kernel: [ 3481.726152]  [<ffffffff817b8e90>] __wait_on_bit+0x60/0x90
Mar 22 12:42:37 server kernel: [ 3481.726155]  [<ffffffff81176708>] ? find_get_pages_tag+0xc8/0x170
Mar 22 12:42:37 server kernel: [ 3481.726157]  [<ffffffff81175980>] wait_on_page_bit+0xc0/0xd0
Mar 22 12:42:37 server kernel: [ 3481.726159]  [<ffffffff810b76b0>] ? autoremove_wake_function+0x40/0x40
Mar 22 12:42:37 server kernel: [ 3481.726160]  [<ffffffff81175a80>] filemap_fdatawait_range+0xf0/0x180
Mar 22 12:42:37 server kernel: [ 3481.726175]  [<ffffffffc04dc283>] btrfs_wait_ordered_range+0x73/0x110 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726184]  [<ffffffffc05075a3>] btrfs_wait_cache_io+0x63/0x1d0 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726190]  [<ffffffffc04acdbf>] ? btrfs_run_delayed_refs.part.72+0x1bf/0x280 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726195]  [<ffffffffc04b069b>] btrfs_start_dirty_block_groups+0x18b/0x400 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726202]  [<ffffffffc04c1501>] btrfs_commit_transaction+0x1c1/0xb10 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726210]  [<ffffffffc04e35af>] ? free_extent_buffer+0x4f/0x90 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726215]  [<ffffffffc04a7f9b>] ? btrfs_block_rsv_check+0x2b/0x70 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726222]  [<ffffffffc04c287b>] __btrfs_end_transaction+0x39b/0x3d0 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726227]  [<ffffffffc04b2c48>] ? btrfs_update_root+0x198/0x320 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726233]  [<ffffffffc04c2c03>] btrfs_end_transaction_throttle+0x13/0x20 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726239]  [<ffffffffc04ab4c2>] btrfs_drop_snapshot+0x492/0x830 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726247]  [<ffffffffc0514c72>] merge_reloc_roots+0xd2/0x230 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726253]  [<ffffffffc05159f3>] btrfs_recover_relocation+0x353/0x3b0 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726258]  [<ffffffffc04947c7>] btrfs_remount+0x407/0x560 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726260]  [<ffffffff811ee66a>] do_remount_sb+0x6a/0x1c0
Mar 22 12:42:37 server kernel: [ 3481.726262]  [<ffffffff8120c413>] do_mount+0x843/0xcf0
Mar 22 12:42:37 server kernel: [ 3481.726263]  [<ffffffff8120ba9a>] ? copy_mount_options+0x3a/0x150
Mar 22 12:42:37 server kernel: [ 3481.726265]  [<ffffffff8120cbab>] SyS_mount+0x8b/0xe0
Mar 22 12:42:37 server kernel: [ 3481.726266]  [<ffffffff817bc332>] entry_SYSCALL_64_fastpath+0x16/0x75
Mar 22 12:42:37 server kernel: [ 4562.225662] INFO: task mount:587 blocked for more than 120 seconds.
Mar 22 12:42:37 server kernel: [ 4562.226559]       Tainted: G     U          4.2.0-30-generic #36~14.04.1-Ubuntu
Mar 22 12:42:37 server kernel: [ 4562.227415] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[...]
Mar 22 12:42:37 server kernel: [ 3481.726122] mount           D ffff88043e296640     0   587    455 0x00000000
Mar 22 12:42:37 server kernel: [ 3481.726125]  ffff88042798f6a8 0000000000000086 ffff88042b98b700 ffff880427946e00
Mar 22 12:42:37 server kernel: [ 3481.726127]  ffff88042798f788 ffff880427990000 0000000000000000 7fffffffffffffff
Mar 22 12:42:37 server kernel: [ 3481.726129]  ffff88043e5aeee8 ffffffff817b9240 ffff88042798f6c8 ffffffff817b8ac7
Mar 22 12:42:37 server kernel: [ 3481.726130] Call Trace:
Mar 22 12:42:37 server kernel: [ 3481.726136]  [<ffffffff817b9240>] ? bit_wait+0x60/0x60
Mar 22 12:42:37 server kernel: [ 3481.726138]  [<ffffffff817b8ac7>] schedule+0x37/0x80
Mar 22 12:42:37 server kernel: [ 3481.726140]  [<ffffffff817bb4a1>] schedule_timeout+0x201/0x2a0
Mar 22 12:42:37 server kernel: [ 3481.726143]  [<ffffffff8101d749>] ? read_tsc+0x9/0x10
Mar 22 12:42:37 server kernel: [ 3481.726146]  [<ffffffff810e6e5e>] ? ktime_get+0x3e/0xa0
Mar 22 12:42:37 server kernel: [ 3481.726147]  [<ffffffff817b9240>] ? bit_wait+0x60/0x60
Mar 22 12:42:37 server kernel: [ 3481.726149]  [<ffffffff817b8136>] io_schedule_timeout+0xa6/0x110
Mar 22 12:42:37 server kernel: [ 3481.726151]  [<ffffffff817b925f>] bit_wait_io+0x1f/0x70
Mar 22 12:42:37 server kernel: [ 3481.726152]  [<ffffffff817b8e90>] __wait_on_bit+0x60/0x90
Mar 22 12:42:37 server kernel: [ 3481.726155]  [<ffffffff81176708>] ? find_get_pages_tag+0xc8/0x170
Mar 22 12:42:37 server kernel: [ 3481.726157]  [<ffffffff81175980>] wait_on_page_bit+0xc0/0xd0
Mar 22 12:42:37 server kernel: [ 3481.726159]  [<ffffffff810b76b0>] ? autoremove_wake_function+0x40/0x40
Mar 22 12:42:37 server kernel: [ 3481.726160]  [<ffffffff81175a80>] filemap_fdatawait_range+0xf0/0x180
Mar 22 12:42:37 server kernel: [ 3481.726175]  [<ffffffffc04dc283>] btrfs_wait_ordered_range+0x73/0x110 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726184]  [<ffffffffc05075a3>] btrfs_wait_cache_io+0x63/0x1d0 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726190]  [<ffffffffc04acdbf>] ? btrfs_run_delayed_refs.part.72+0x1bf/0x280 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726195]  [<ffffffffc04b069b>] btrfs_start_dirty_block_groups+0x18b/0x400 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726202]  [<ffffffffc04c1501>] btrfs_commit_transaction+0x1c1/0xb10 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726210]  [<ffffffffc04e35af>] ? free_extent_buffer+0x4f/0x90 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726215]  [<ffffffffc04a7f9b>] ? btrfs_block_rsv_check+0x2b/0x70 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726222]  [<ffffffffc04c287b>] __btrfs_end_transaction+0x39b/0x3d0 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726227]  [<ffffffffc04b2c48>] ? btrfs_update_root+0x198/0x320 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726233]  [<ffffffffc04c2c03>] btrfs_end_transaction_throttle+0x13/0x20 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726239]  [<ffffffffc04ab4c2>] btrfs_drop_snapshot+0x492/0x830 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726247]  [<ffffffffc0514c72>] merge_reloc_roots+0xd2/0x230 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726253]  [<ffffffffc05159f3>] btrfs_recover_relocation+0x353/0x3b0 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726258]  [<ffffffffc04947c7>] btrfs_remount+0x407/0x560 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726260]  [<ffffffff811ee66a>] do_remount_sb+0x6a/0x1c0
Mar 22 12:42:37 server kernel: [ 3481.726262]  [<ffffffff8120c413>] do_mount+0x843/0xcf0
Mar 22 12:42:37 server kernel: [ 3481.726263]  [<ffffffff8120ba9a>] ? copy_mount_options+0x3a/0x150
Mar 22 12:42:37 server kernel: [ 3481.726265]  [<ffffffff8120cbab>] SyS_mount+0x8b/0xe0
Mar 22 12:42:37 server kernel: [ 3481.726266]  [<ffffffff817bc332>] entry_SYSCALL_64_fastpath+0x16/0x75
Mar 22 12:42:37 server kernel: [ 4562.225662] INFO: task mount:587 blocked for more than 120 seconds.
[...]
Mar 22 12:42:37 server kernel: [ 4562.228280] mount           D ffff88043e296640     0   587    455 0x00000000
Mar 22 12:42:37 server kernel: [ 4562.228283]  ffff88042798f6a8 0000000000000086 ffff88042b98b700 ffff880427946e00
Mar 22 12:42:37 server kernel: [ 4562.228285]  ffff88042798f788 ffff880427990000 0000000000000000 7fffffffffffffff
Mar 22 12:42:37 server kernel: [ 4562.228286]  ffff88043e5c05e8 ffffffff817b9240 ffff88042798f6c8 ffffffff817b8ac7
Mar 22 12:42:37 server kernel: [ 4562.228288] Call Trace:
Mar 22 12:42:37 server kernel: [ 4562.228294]  [<ffffffff817b9240>] ? bit_wait+0x60/0x60
Mar 22 12:42:37 server kernel: [ 4562.228296]  [<ffffffff817b8ac7>] schedule+0x37/0x80
Mar 22 12:42:37 server kernel: [ 4562.228297]  [<ffffffff817bb4a1>] schedule_timeout+0x201/0x2a0
Mar 22 12:42:37 server kernel: [ 4562.228301]  [<ffffffff8101d749>] ? read_tsc+0x9/0x10
Mar 22 12:42:37 server kernel: [ 4562.228304]  [<ffffffff810e6e5e>] ? ktime_get+0x3e/0xa0
Mar 22 12:42:37 server kernel: [ 4562.228305]  [<ffffffff817b9240>] ? bit_wait+0x60/0x60
Mar 22 12:42:37 server kernel: [ 4562.228307]  [<ffffffff817b8136>] io_schedule_timeout+0xa6/0x110
Mar 22 12:42:37 server kernel: [ 4562.228309]  [<ffffffff817b925f>] bit_wait_io+0x1f/0x70
Mar 22 12:42:37 server kernel: [ 4562.228310]  [<ffffffff817b8e90>] __wait_on_bit+0x60/0x90
Mar 22 12:42:37 server kernel: [ 4562.228313]  [<ffffffff81176708>] ? find_get_pages_tag+0xc8/0x170
Mar 22 12:42:37 server kernel: [ 4562.228315]  [<ffffffff81175980>] wait_on_page_bit+0xc0/0xd0
Mar 22 12:42:37 server kernel: [ 4562.228317]  [<ffffffff810b76b0>] ? autoremove_wake_function+0x40/0x40
Mar 22 12:42:37 server kernel: [ 4562.228318]  [<ffffffff81175a80>] filemap_fdatawait_range+0xf0/0x180
Mar 22 12:42:37 server kernel: [ 4562.228332]  [<ffffffffc04dc283>] btrfs_wait_ordered_range+0x73/0x110 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228334]  [<ffffffff81180df1>] ? set_page_dirty+0x41/0x70
Mar 22 12:42:37 server kernel: [ 4562.228343]  [<ffffffffc05075a3>] btrfs_wait_cache_io+0x63/0x1d0 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228348]  [<ffffffffc0498071>] ? btrfs_release_path+0x21/0xa0 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228354]  [<ffffffffc04b069b>] btrfs_start_dirty_block_groups+0x18b/0x400 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228361]  [<ffffffffc04c1501>] btrfs_commit_transaction+0x1c1/0xb10 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228369]  [<ffffffffc04e35af>] ? free_extent_buffer+0x4f/0x90 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228374]  [<ffffffffc04a7f9b>] ? btrfs_block_rsv_check+0x2b/0x70 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228381]  [<ffffffffc04c287b>] __btrfs_end_transaction+0x39b/0x3d0 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228386]  [<ffffffffc04b2c48>] ? btrfs_update_root+0x198/0x320 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228392]  [<ffffffffc04c2c03>] btrfs_end_transaction_throttle+0x13/0x20 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228398]  [<ffffffffc04ab4c2>] btrfs_drop_snapshot+0x492/0x830 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228405]  [<ffffffffc0514c72>] merge_reloc_roots+0xd2/0x230 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228412]  [<ffffffffc05159f3>] btrfs_recover_relocation+0x353/0x3b0 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228417]  [<ffffffffc04947c7>] btrfs_remount+0x407/0x560 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228419]  [<ffffffff811ee66a>] do_remount_sb+0x6a/0x1c0
Mar 22 12:42:37 server kernel: [ 4562.228421]  [<ffffffff8120c413>] do_mount+0x843/0xcf0
Mar 22 12:42:37 server kernel: [ 4562.228422]  [<ffffffff8120ba9a>] ? copy_mount_options+0x3a/0x150
Mar 22 12:42:37 server kernel: [ 4562.228424]  [<ffffffff8120cbab>] SyS_mount+0x8b/0xe0
Mar 22 12:42:37 server kernel: [ 4562.228425]  [<ffffffff817bc332>] entry_SYSCALL_64_fastpath+0x16/0x75
Mar 22 12:42:37 server kernel: [ 7975.560908] BTRFS info (device dm-0): continuing balance
[...]
Mar 22 12:44:03 server kernel: [ 8073.441009] BTRFS info (device dm-0): relocating block group 9553211031552 flags 36
[...]
Mar 22 12:51:33 server kernel: [ 8524.324198] btrfs-transacti D ffff88043e3d6640     0   409      2 0x00000000
Mar 22 12:51:33 server kernel: [ 8524.324204]  ffff880424983d68 0000000000000046 ffff88042b9b8000 ffff880425be5280
Mar 22 12:51:33 server kernel: [ 8524.324208]  0000000000000246 ffff880424984000 ffff8803ff9884e0 ffff8804265801b0
Mar 22 12:51:33 server kernel: [ 8524.324211]  ffff88042bbe5000 0000000000000000 ffff880424983d88 ffffffff817b8ac7
Mar 22 12:51:33 server kernel: [ 8524.324215] Call Trace:
Mar 22 12:51:33 server kernel: [ 8524.324225]  [<ffffffff817b8ac7>] schedule+0x37/0x80
Mar 22 12:51:33 server kernel: [ 8524.324247]  [<ffffffffc04c06d0>] wait_for_commit.isra.17+0x40/0x70 [btrfs]
Mar 22 12:51:33 server kernel: [ 8524.324252]  [<ffffffff810b7670>] ? prepare_to_wait_event+0xf0/0xf0
Mar 22 12:51:33 server kernel: [ 8524.324267]  [<ffffffffc04c149c>] btrfs_commit_transaction+0x15c/0xb10 [btrfs]
Mar 22 12:51:33 server kernel: [ 8524.324281]  [<ffffffffc04c1ee3>] ? start_transaction+0x93/0x580 [btrfs]
Mar 22 12:51:33 server kernel: [ 8524.324295]  [<ffffffffc04bc99a>] transaction_kthread+0x1ba/0x240 [btrfs]
Mar 22 12:51:33 server kernel: [ 8524.324307]  [<ffffffffc04bc7e0>] ? btrfs_cleanup_transaction+0x540/0x540 [btrfs]
Mar 22 12:51:33 server kernel: [ 8524.324311]  [<ffffffff81095499>] kthread+0xc9/0xe0
Mar 22 12:51:33 server kernel: [ 8524.324315]  [<ffffffff810953d0>] ? kthread_create_on_node+0x1c0/0x1c0
Mar 22 12:51:33 server kernel: [ 8524.324318]  [<ffffffff817bc75f>] ret_from_fork+0x3f/0x70
Mar 22 12:51:33 server kernel: [ 8524.324322]  [<ffffffff810953d0>] ? kthread_create_on_node+0x1c0/0x1c0
Mar 22 12:52:23 server kernel: [ 8574.196463] init: ureadahead main process (449) killed by KILL signal
Mar 22 12:59:29 server kernel: [ 9000.470433] BTRFS info (device dm-0): found 106150 extents
Mar 22 12:59:54 server kernel: [ 9025.568882] BTRFS info (device dm-0): 2 enospc errors during balance


dmesg gives me

[ 3481.723605] INFO: task mount:587 blocked for more than 120 seconds.
[ 3481.724438]       Tainted: G     U          4.2.0-30-generic #36~14.04.1-Ubuntu
[ 3481.725273] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3481.726122] mount           D ffff88043e296640     0   587    455 0x00000000
[ 3481.726125]  ffff88042798f6a8 0000000000000086 ffff88042b98b700 ffff880427946e00
[ 3481.726127]  ffff88042798f788 ffff880427990000 0000000000000000 7fffffffffffffff
[ 3481.726129]  ffff88043e5aeee8 ffffffff817b9240 ffff88042798f6c8 ffffffff817b8ac7
[ 3481.726130] Call Trace:
[ 3481.726136]  [<ffffffff817b9240>] ? bit_wait+0x60/0x60
[ 3481.726138]  [<ffffffff817b8ac7>] schedule+0x37/0x80
[ 3481.726140]  [<ffffffff817bb4a1>] schedule_timeout+0x201/0x2a0
[ 3481.726143]  [<ffffffff8101d749>] ? read_tsc+0x9/0x10
[ 3481.726146]  [<ffffffff810e6e5e>] ? ktime_get+0x3e/0xa0
[ 3481.726147]  [<ffffffff817b9240>] ? bit_wait+0x60/0x60
[ 3481.726149]  [<ffffffff817b8136>] io_schedule_timeout+0xa6/0x110
[ 3481.726151]  [<ffffffff817b925f>] bit_wait_io+0x1f/0x70
[ 3481.726152]  [<ffffffff817b8e90>] __wait_on_bit+0x60/0x90
[ 3481.726155]  [<ffffffff81176708>] ? find_get_pages_tag+0xc8/0x170
[ 3481.726157]  [<ffffffff81175980>] wait_on_page_bit+0xc0/0xd0
[ 3481.726159]  [<ffffffff810b76b0>] ? autoremove_wake_function+0x40/0x40
[ 3481.726160]  [<ffffffff81175a80>] filemap_fdatawait_range+0xf0/0x180
[ 3481.726175]  [<ffffffffc04dc283>] btrfs_wait_ordered_range+0x73/0x110 [btrfs]
[ 3481.726184]  [<ffffffffc05075a3>] btrfs_wait_cache_io+0x63/0x1d0 [btrfs]
[ 3481.726190]  [<ffffffffc04acdbf>] ? btrfs_run_delayed_refs.part.72+0x1bf/0x280 [btrfs]
[ 3481.726195]  [<ffffffffc04b069b>] btrfs_start_dirty_block_groups+0x18b/0x400 [btrfs]
[ 3481.726202]  [<ffffffffc04c1501>] btrfs_commit_transaction+0x1c1/0xb10 [btrfs]
[ 3481.726210]  [<ffffffffc04e35af>] ? free_extent_buffer+0x4f/0x90 [btrfs]
[ 3481.726215]  [<ffffffffc04a7f9b>] ? btrfs_block_rsv_check+0x2b/0x70 [btrfs]
[ 3481.726222]  [<ffffffffc04c287b>] __btrfs_end_transaction+0x39b/0x3d0 [btrfs]
[ 3481.726227]  [<ffffffffc04b2c48>] ? btrfs_update_root+0x198/0x320 [btrfs]
[ 3481.726233]  [<ffffffffc04c2c03>] btrfs_end_transaction_throttle+0x13/0x20 [btrfs]
[ 3481.726239]  [<ffffffffc04ab4c2>] btrfs_drop_snapshot+0x492/0x830 [btrfs]
[ 3481.726247]  [<ffffffffc0514c72>] merge_reloc_roots+0xd2/0x230 [btrfs]
[ 3481.726253]  [<ffffffffc05159f3>] btrfs_recover_relocation+0x353/0x3b0 [btrfs]
[ 3481.726258]  [<ffffffffc04947c7>] btrfs_remount+0x407/0x560 [btrfs]
[ 3481.726260]  [<ffffffff811ee66a>] do_remount_sb+0x6a/0x1c0
[ 3481.726262]  [<ffffffff8120c413>] do_mount+0x843/0xcf0
[ 3481.726263]  [<ffffffff8120ba9a>] ? copy_mount_options+0x3a/0x150
[ 3481.726265]  [<ffffffff8120cbab>] SyS_mount+0x8b/0xe0
[ 3481.726266]  [<ffffffff817bc332>] entry_SYSCALL_64_fastpath+0x16/0x75
[ 4562.225662] INFO: task mount:587 blocked for more than 120 seconds.
[ 4562.226559]       Tainted: G     U          4.2.0-30-generic #36~14.04.1-Ubuntu
[ 4562.227415] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4562.228280] mount           D ffff88043e296640     0   587    455 0x00000000
[ 4562.228283]  ffff88042798f6a8 0000000000000086 ffff88042b98b700 ffff880427946e00
[ 4562.228285]  ffff88042798f788 ffff880427990000 0000000000000000 7fffffffffffffff
[ 4562.228286]  ffff88043e5c05e8 ffffffff817b9240 ffff88042798f6c8 ffffffff817b8ac7
[ 4562.228288] Call Trace:
[ 4562.228294]  [<ffffffff817b9240>] ? bit_wait+0x60/0x60
[ 4562.228296]  [<ffffffff817b8ac7>] schedule+0x37/0x80
[ 4562.228297]  [<ffffffff817bb4a1>] schedule_timeout+0x201/0x2a0
[ 4562.228301]  [<ffffffff8101d749>] ? read_tsc+0x9/0x10
[ 4562.228304]  [<ffffffff810e6e5e>] ? ktime_get+0x3e/0xa0
[ 4562.228305]  [<ffffffff817b9240>] ? bit_wait+0x60/0x60
[ 4562.228307]  [<ffffffff817b8136>] io_schedule_timeout+0xa6/0x110
[ 4562.228309]  [<ffffffff817b925f>] bit_wait_io+0x1f/0x70
[ 4562.228310]  [<ffffffff817b8e90>] __wait_on_bit+0x60/0x90
[ 4562.228313]  [<ffffffff81176708>] ? find_get_pages_tag+0xc8/0x170
[ 4562.228315]  [<ffffffff81175980>] wait_on_page_bit+0xc0/0xd0
[ 4562.228317]  [<ffffffff810b76b0>] ? autoremove_wake_function+0x40/0x40
[ 4562.228318]  [<ffffffff81175a80>] filemap_fdatawait_range+0xf0/0x180
[ 4562.228332]  [<ffffffffc04dc283>] btrfs_wait_ordered_range+0x73/0x110 [btrfs]
[ 4562.228334]  [<ffffffff81180df1>] ? set_page_dirty+0x41/0x70
[ 4562.228343]  [<ffffffffc05075a3>] btrfs_wait_cache_io+0x63/0x1d0 [btrfs]
[ 4562.228348]  [<ffffffffc0498071>] ? btrfs_release_path+0x21/0xa0 [btrfs]
[ 4562.228354]  [<ffffffffc04b069b>] btrfs_start_dirty_block_groups+0x18b/0x400 [btrfs]
[ 4562.228361]  [<ffffffffc04c1501>] btrfs_commit_transaction+0x1c1/0xb10 [btrfs]
[ 4562.228369]  [<ffffffffc04e35af>] ? free_extent_buffer+0x4f/0x90 [btrfs]
[ 4562.228374]  [<ffffffffc04a7f9b>] ? btrfs_block_rsv_check+0x2b/0x70 [btrfs]
[ 4562.228381]  [<ffffffffc04c287b>] __btrfs_end_transaction+0x39b/0x3d0 [btrfs]
[ 4562.228386]  [<ffffffffc04b2c48>] ? btrfs_update_root+0x198/0x320 [btrfs]
[ 4562.228392]  [<ffffffffc04c2c03>] btrfs_end_transaction_throttle+0x13/0x20 [btrfs]
[ 4562.228398]  [<ffffffffc04ab4c2>] btrfs_drop_snapshot+0x492/0x830 [btrfs]
[ 4562.228405]  [<ffffffffc0514c72>] merge_reloc_roots+0xd2/0x230 [btrfs]
[ 4562.228412]  [<ffffffffc05159f3>] btrfs_recover_relocation+0x353/0x3b0 [btrfs]
[ 4562.228417]  [<ffffffffc04947c7>] btrfs_remount+0x407/0x560 [btrfs]
[ 4562.228419]  [<ffffffff811ee66a>] do_remount_sb+0x6a/0x1c0
[ 4562.228421]  [<ffffffff8120c413>] do_mount+0x843/0xcf0
[ 4562.228422]  [<ffffffff8120ba9a>] ? copy_mount_options+0x3a/0x150
[ 4562.228424]  [<ffffffff8120cbab>] SyS_mount+0x8b/0xe0
[ 4562.228425]  [<ffffffff817bc332>] entry_SYSCALL_64_fastpath+0x16/0x75
[ 7975.560908] BTRFS info (device dm-0): continuing balance
[ 7977.590090] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null)
[ 7978.881574] EXT4-fs (sdb2): mounted filesystem with ordered data mode. Opts: (null)
[ 7982.053560] init: failsafe main process (1047) killed by TERM signal
[...]
[ 8073.441009] BTRFS info (device dm-0): relocating block group 9553211031552 flags 36
[ 8081.239604] audit: type=1400 audit(1458647050.858:22): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/mysqld" pid=1299 comm="apparmor_parser"
[ 8171.758863] init: mysql post-start process (1322) terminated with status 1
[ 8190.153034] audit: type=1400 audit(1458647159.668:23): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/var/tmp/mysql/" pid=1321 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=106 ouid=0
[ 8216.355398] init: plymouth-upstart-bridge main process ended, respawning
[ 8524.324045] INFO: task btrfs-transacti:409 blocked for more than 120 seconds.
[ 8524.324102]       Tainted: G     U          4.2.0-30-generic #36~14.04.1-Ubuntu
[ 8524.324149] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 8524.324198] btrfs-transacti D ffff88043e3d6640     0   409      2 0x00000000
[ 8524.324204]  ffff880424983d68 0000000000000046 ffff88042b9b8000 ffff880425be5280
[ 8524.324208]  0000000000000246 ffff880424984000 ffff8803ff9884e0 ffff8804265801b0
[ 8524.324211]  ffff88042bbe5000 0000000000000000 ffff880424983d88 ffffffff817b8ac7
[ 8524.324215] Call Trace:
[ 8524.324225]  [<ffffffff817b8ac7>] schedule+0x37/0x80
[ 8524.324247]  [<ffffffffc04c06d0>] wait_for_commit.isra.17+0x40/0x70 [btrfs]
[ 8524.324252]  [<ffffffff810b7670>] ? prepare_to_wait_event+0xf0/0xf0
[ 8524.324267]  [<ffffffffc04c149c>] btrfs_commit_transaction+0x15c/0xb10 [btrfs]
[ 8524.324281]  [<ffffffffc04c1ee3>] ? start_transaction+0x93/0x580 [btrfs]
[ 8524.324295]  [<ffffffffc04bc99a>] transaction_kthread+0x1ba/0x240 [btrfs]
[ 8524.324307]  [<ffffffffc04bc7e0>] ? btrfs_cleanup_transaction+0x540/0x540 [btrfs]
[ 8524.324311]  [<ffffffff81095499>] kthread+0xc9/0xe0
[ 8524.324315]  [<ffffffff810953d0>] ? kthread_create_on_node+0x1c0/0x1c0
[ 8524.324318]  [<ffffffff817bc75f>] ret_from_fork+0x3f/0x70
[ 8524.324322]  [<ffffffff810953d0>] ? kthread_create_on_node+0x1c0/0x1c0
[ 8574.196463] init: ureadahead main process (449) killed by KILL signal
[ 9000.470433] BTRFS info (device dm-0): found 106150 extents
[ 9025.568882] BTRFS info (device dm-0): 2 enospc errors during balance

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1559723

Title:
  soft lockup during btrfs balance

Status in linux package in Ubuntu:
  Incomplete

Bug description:
  I performed a btrfs balance for a disk which ran out of space. When I
  did this some months ago, I got an error message, but this time, I got
  a soft lockup and the file system got mounted read-only:

  # cat /proc/version_signature
  Ubuntu 4.2.0-27.32~14.04.1-generic 4.2.8-ckt1

  I see multiple errors in dmesg output, like

  [3099814.088753] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [btrfs:27516]
  [3099814.089403] Modules linked in: ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c intel_rapl w83627ehf x86_pkg_temp_thermal intel_powerclamp hwmon_vid coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda
  _codec_generic lp joydev input_leds ppdev gpio_ich 8250_fintek parport_pc parport nuvoton_cir rc_core mac_hid snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_timer snd lpc_ich mei_me soundcore mei s
  hpchp serio_raw btrfs drbg ansi_cprng dm_crypt raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid0 multipath linear raid1 crct10dif_pclmul crc32_pclmul hid_generic i915 aes
  ni_intel ahci aes_x86_64 lrw usbhid video gf128mul i2c_algo_bit glue_helper e1000e ablk_helper drm_kms_helper libahci ptp cryptd hid psmouse pps_core drm pata_acpi
  [3099814.089436] CPU: 3 PID: 27516 Comm: btrfs Tainted: G     U  W    L  4.2.0-27-generic #32~14.04.1-Ubuntu
  [3099814.089437] Hardware name:                  /DQ67OW, BIOS SWQ6710H.86A.0067.2014.0313.1347 03/13/2014
  [3099814.089438] task: ffff8800aca0a940 ti: ffff880110c5c000 task.ti: ffff880110c5c000
  [3099814.089438] RIP: 0010:[<ffffffffc031f9d1>]  [<ffffffffc031f9d1>] __del_reloc_root+0xd1/0x100 [btrfs]
  [3099814.089451] RSP: 0018:ffff880110c5fa60  EFLAGS: 00000246
  [3099814.089452] RAX: ffff880142cd36f8 RBX: ffffffffc02a912a RCX: 00000000ffffffe2
  [3099814.089453] RDX: 000007ba9c902000 RSI: ffffffffc034e280 RDI: ffff88002a9e8d70
  [3099814.089454] RBP: ffff880110c5fa78 R08: 0000000000000000 R09: 00000000ffffffe2
  [3099814.089454] R10: ffffffffc02a912a R11: ffffea00029bc9c0 R12: 00000000ffffffe2
  [3099814.089455] R13: ffff880110c5fa78 R14: 000000002b4bf801 R15: ffff88042b4bf800
  [3099814.089456] FS:  00007f93651f0880(0000) GS:ffff88043e2c0000(0000) knlGS:0000000000000000
  [3099814.089457] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  [3099814.089458] CR2: 0000559c1ffff7d0 CR3: 00000002fca04000 CR4: 00000000000406e0
  [3099814.089459] Stack:
  [3099814.089460]  ffff88002a9e8800 ffff88042b52b000 ffff880110c5fab8 ffff880110c5fa98
  [3099814.089461]  ffffffffc031fa25 ffff88042b4bfbc0 ffff88042b4bfbc0 ffff880110c5faf8
  [3099814.089462]  ffffffffc0325f2a ffff88002a9e8d78 000000002a9e8d78 ffff88042b4bdbc0
  [3099814.089464] Call Trace:
  [3099814.089471]  [<ffffffffc031fa25>] free_reloc_roots+0x25/0x40 [btrfs]
  [3099814.089477]  [<ffffffffc0325f2a>] merge_reloc_roots+0x16a/0x230 [btrfs]
  [3099814.089483]  [<ffffffffc0326246>] relocate_block_group+0x256/0x600 [btrfs]
  [3099814.089489]  [<ffffffffc03267b3>] btrfs_relocate_block_group+0x1c3/0x2d0 [btrfs]
  [3099814.089496]  [<ffffffffc02f9dee>] btrfs_relocate_chunk.isra.39+0x3e/0xc0 [btrfs]
  [3099814.089502]  [<ffffffffc02fb26f>] __btrfs_balance+0x48f/0x8c0 [btrfs]
  [3099814.089509]  [<ffffffffc02fba1d>] btrfs_balance+0x37d/0x650 [btrfs]
  [3099814.089516]  [<ffffffffc03085e2>] ? btrfs_ioctl_balance+0x282/0x530 [btrfs]
  [3099814.089522]  [<ffffffffc0308756>] btrfs_ioctl_balance+0x3f6/0x530 [btrfs]
  [3099814.089528]  [<ffffffffc030a17f>] btrfs_ioctl+0x56f/0x2470 [btrfs]
  [3099814.089532]  [<ffffffff8118559b>] ? lru_cache_add_active_or_unevictable+0x2b/0xa0
  [3099814.089534]  [<ffffffff811a5e55>] ? handle_mm_fault+0xba5/0x1840
  [3099814.089536]  [<ffffffff811fdfbd>] do_vfs_ioctl+0x2cd/0x4b0
  [3099814.089539]  [<ffffffff810642a6>] ? __do_page_fault+0x1b6/0x430
  [3099814.089541]  [<ffffffff811fe219>] SyS_ioctl+0x79/0x90
  [3099814.089543]  [<ffffffff817bc3b2>] entry_SYSCALL_64_fastpath+0x16/0x75
  [3099814.089543] Code: f0 01 00 00 49 89 84 24 c0 03 00 00 49 89 84 24 c8 03 00 00 48 81 c7 38 09 00 00 c6 07 00 66 66 66 90 48 89 df e8 b0 d3 ea c0 5b <41> 5c 41 5d 41 5e 5d c3 76 09 48 8b 5b 08 e9 5a ff ff ff 49 8d 

  I also can't access some of the directories now (even in read-only
  mode):

  cd somedir
  -su: cd: somedir: Eingabe-/Ausgabefehler

  # lsb_release -rd
  Description:    Ubuntu 14.04.4 LTS
  Release:        14.04

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1559723/+subscriptions