kernel-packages team mailing list archive

[Rasmus Borup Hansen <1468039@xxxxxxxxxxxxxxxxxx>]

I' having a similar problem with trusty 3.13.0-55:

[601201.817664] ffff88016e03e000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00  ................
[601201.818224] ffff88016e03e010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00  ..... ..........
[601201.818827] ffff88016e03e020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[601201.819429] ffff88016e03e030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[601201.820013] XFS (dm-0): Internal error xfs_attr3_leaf_write_verify at line 216 of file /build/buildd/linux-3.13.0/fs/xfs/xfs_attr_leaf.c.  Caller 0xffffffffa00996f0
[601201.820910] CPU: 1 PID: 421 Comm: xfsaild/dm-0 Not tainted 3.13.0-55-generic #92-Ubuntu
[601201.820913] Hardware name: Dell Inc. PowerEdge R310/05XKKK, BIOS 1.8.2 08/17/2011
[601201.820914]  0000000000000001 ffff880035cc3bd0 ffffffff81723294 ffff8802ac71d800
[601201.820918]  ffff880035cc3be8 ffffffffa009d6fb ffffffffa00996f0 ffff880035cc3c20
[601201.820919]  ffffffffa009d755 000000d800205500 ffff88001aa2a400 ffff880050acc2b8
[601201.820922] Call Trace:
[601201.820930]  [<ffffffff81723294>] dump_stack+0x45/0x56
[601201.820964]  [<ffffffffa009d6fb>] xfs_error_report+0x3b/0x40 [xfs]
[601201.820974]  [<ffffffffa00996f0>] ? _xfs_buf_ioapply+0x70/0x3a0 [xfs]
[601201.820984]  [<ffffffffa009d755>] xfs_corruption_error+0x55/0x80 [xfs]
[601201.820997]  [<ffffffffa00bab50>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
[601201.821007]  [<ffffffffa00996f0>] ? _xfs_buf_ioapply+0x70/0x3a0 [xfs]
[601201.821016]  [<ffffffffa009b3d5>] ? xfs_bdstrat_cb+0x55/0xb0 [xfs]
[601201.821026]  [<ffffffffa00996f0>] _xfs_buf_ioapply+0x70/0x3a0 [xfs]
[601201.821030]  [<ffffffff8109abc0>] ? wake_up_state+0x20/0x20
[601201.821040]  [<ffffffffa009b3d5>] ? xfs_bdstrat_cb+0x55/0xb0 [xfs]
[601201.821050]  [<ffffffffa009b336>] xfs_buf_iorequest+0x46/0x90 [xfs]
[601201.821060]  [<ffffffffa009b3d5>] xfs_bdstrat_cb+0x55/0xb0 [xfs]
[601201.821070]  [<ffffffffa009b56b>] __xfs_buf_delwri_submit+0x13b/0x210 [xfs]
[601201.821081]  [<ffffffffa009c000>] ? xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs]
[601201.821100]  [<ffffffffa00faaa0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[601201.821110]  [<ffffffffa009c000>] xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs]
[601201.821127]  [<ffffffffa00facd7>] xfsaild+0x237/0x5c0 [xfs]
[601201.821145]  [<ffffffffa00faaa0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[601201.821148]  [<ffffffff8108b702>] kthread+0xd2/0xf0
[601201.821150]  [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0
[601201.821153]  [<ffffffff81733ca8>] ret_from_fork+0x58/0x90
[601201.821155]  [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0
[601201.821158] XFS (dm-0): Corruption detected. Unmount and run xfs_repair
[601201.821581] XFS (dm-0): xfs_do_force_shutdown(0x8) called from line 1320 of file /build/buildd/linux-3.13.0/fs/xfs/xfs_buf.c.  Return address = 0xffffffffa009971c
[601201.832780] XFS (dm-0): Corruption of in-memory data detected.  Shutting down filesystem
[601201.833292] XFS (dm-0): Please umount the filesystem and rectify the problem(s)
[601201.843375] ffff880238b00000: 58 44 32 42 01 00 0e c0 00 00 00 00 00 00 00 00  XD2B............
[601201.843924] ffff880238b00010: 00 00 00 20 3d a7 0c 7b 01 2e 8d 9e 01 88 00 10  ... =..{........
[601201.844472] ffff880238b00020: 00 00 00 19 21 a9 2d 2f 02 2e 2e 9e 01 88 00 20  ....!.-/.......
[601201.845031] ffff880238b00030: 00 00 00 07 39 2b 68 20 18 4d 4f 42 31 41 5f 48  ....9+h .MOB1A_H
[601201.845585] XFS (dm-0): Internal error xfs_da3_node_read_verify at line 240 of file /build/buildd/linux-3.13.0/fs/xfs/xfs_da_btree.c.  Caller 0xffffffffa009a885
[601201.858885] CPU: 1 PID: 2806 Comm: kworker/1:1H Not tainted 3.13.0-55-generic #92-Ubuntu
[601201.858889] Hardware name: Dell Inc. PowerEdge R310/05XKKK, BIOS 1.8.2 08/17/2011
[601201.858938] Workqueue: xfslogd xfs_buf_iodone_work [xfs]
[601201.858941]  0000000000000001 ffff8802af9cbd68 ffffffff81723294 ffff8802ac71d800
[601201.858944]  ffff8802af9cbd80 ffffffffa009d6fb ffffffffa009a885 ffff8802af9cbdb8
[601201.858945]  ffffffffa009d755 000000f0810a0465 ffff880031b35b00 ffff880238b00000
[601201.858948] Call Trace:
[601201.858958]  [<ffffffff81723294>] dump_stack+0x45/0x56
[601201.858969]  [<ffffffffa009d6fb>] xfs_error_report+0x3b/0x40 [xfs]
[601201.858979]  [<ffffffffa009a885>] ? xfs_buf_iodone_work+0x85/0xf0 [xfs]
[601201.858989]  [<ffffffffa009d755>] xfs_corruption_error+0x55/0x80 [xfs]
[601201.859005]  [<ffffffffa00d146c>] xfs_da3_node_read_verify+0x8c/0x180 [xfs]
[601201.859014]  [<ffffffffa009a885>] ? xfs_buf_iodone_work+0x85/0xf0 [xfs]
[601201.859024]  [<ffffffffa009a885>] xfs_buf_iodone_work+0x85/0xf0 [xfs]
[601201.859028]  [<ffffffff81083b22>] process_one_work+0x182/0x450
[601201.859030]  [<ffffffff81084911>] worker_thread+0x121/0x410
[601201.859032]  [<ffffffff810847f0>] ? rescuer_thread+0x430/0x430
[601201.859035]  [<ffffffff8108b702>] kthread+0xd2/0xf0
[601201.859037]  [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0
[601201.859040]  [<ffffffff81733ca8>] ret_from_fork+0x58/0x90
[601201.859042]  [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0
[601201.859044] XFS (dm-0): Corruption detected. Unmount and run xfs_repair
[601201.872288] XFS (dm-0): metadata I/O error: block 0x101ed393c0 ("xfs_trans_read_buf_map") error 117 numblks 8
[601201.942054] XFS (dm-0): xfs_imap_to_bp: xfs_trans_read_buf() returned error 5.
[601209.420268] XFS (dm-0): xfs_log_force: error 5 returned.
[601239.529710] XFS (dm-0): xfs_log_force: error 5 returned.
[601269.639150] XFS (dm-0): xfs_log_force: error 5 returned.
[601299.748593] XFS (dm-0): xfs_log_force: error 5 returned.
[601329.858034] XFS (dm-0): xfs_log_force: error 5 returned.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-lts-trusty in Ubuntu.
https://bugs.launchpad.net/bugs/1468039

Title:
  XFS: Corruption of in-memory data detected

Status in linux-lts-trusty package in Ubuntu:
  New

Bug description:
  I've got 2 NFS servers that have been running Ubuntu 12.04 for a
  number of years. For the last 9 months or so they've been happily
  running kernel 3.8.0-44 with no problem. As soon as I upgraded them to
  3.13.0-53 the both experienced the same problem on an XFS filesystem
  when it started to have a heavy write load:

  [267127.795322] XFS (sda1): Internal error xfs_attr3_leaf_write_verify at line 216 of file /build/buildd/linux-lts-trusty-3.13.0/fs/xfs/xfs_attr_leaf.c.  Caller 0xffffffffa0250ada
  [267127.811202] CPU: 10 PID: 28744 Comm: xfsaild/sda1 Tainted: G          I   3.13.0-53-generic #89~precise1-Ubuntu
  [267127.811206] Hardware name: IBM System x3690 X5 -[7148AC1]-/Node 1, System Card, BIOS -[MLE170CUS-1.70]- 09/23/2011
  [267127.811211]  0000000000000001 ffff881016b7bb38 ffffffff8175a3e4 ffff88103f34f030
  [267127.811221]  ffff881023d9c000 ffff881016b7bb58 ffffffffa025317f ffffffffa0250ada
  [267127.811228]  ffffffff00000001 ffff881016b7bb98 ffffffffa02531ee 00000fe000200000
  [267127.811236] Call Trace:
  [267127.811251]  [<ffffffff8175a3e4>] dump_stack+0x46/0x58
  [267127.811311]  [<ffffffffa025317f>] xfs_error_report+0x3f/0x50 [xfs]
  [267127.811337]  [<ffffffffa0250ada>] ? _xfs_buf_ioapply+0x6a/0x140 [xfs]
  [267127.811363]  [<ffffffffa02531ee>] xfs_corruption_error+0x5e/0x90 [xfs]
  [267127.811396]  [<ffffffffa02711b0>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
  [267127.811421]  [<ffffffffa0250ada>] ? _xfs_buf_ioapply+0x6a/0x140 [xfs]
  [267127.811445]  [<ffffffffa0250e05>] ? xfs_bdstrat_cb+0x65/0xf0 [xfs]
  [267127.811470]  [<ffffffffa0250e05>] ? xfs_bdstrat_cb+0x65/0xf0 [xfs]
  [267127.811494]  [<ffffffffa0250ada>] _xfs_buf_ioapply+0x6a/0x140 [xfs]
  [267127.811505]  [<ffffffff8109fba0>] ? try_to_wake_up+0x210/0x210
  [267127.811530]  [<ffffffffa0250d29>] xfs_buf_iorequest+0x59/0xd0 [xfs]
  [267127.811554]  [<ffffffffa0250e05>] xfs_bdstrat_cb+0x65/0xf0 [xfs]
  [267127.811579]  [<ffffffffa0251013>] __xfs_buf_delwri_submit+0x183/0x210 [xfs]
  [267127.811604]  [<ffffffffa0251ac0>] ? xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs]
  [267127.811630]  [<ffffffffa0251ac0>] xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs]
  [267127.811673]  [<ffffffffa02b4f66>] xfsaild_push+0x126/0x5b0 [xfs]
  [267127.811716]  [<ffffffffa02b548f>] xfsaild+0x9f/0x140 [xfs]
  [267127.811757]  [<ffffffffa02b53f0>] ? xfsaild_push+0x5b0/0x5b0 [xfs]
  [267127.811766]  [<ffffffff8108fcd9>] kthread+0xc9/0xe0
  [267127.811772]  [<ffffffff8108fc10>] ? flush_kthread_worker+0xb0/0xb0
  [267127.811781]  [<ffffffff8176f928>] ret_from_fork+0x58/0x90
  [267127.811787]  [<ffffffff8108fc10>] ? flush_kthread_worker+0xb0/0xb0
  [267127.811791] XFS (sda1): Corruption detected. Unmount and run xfs_repair
  [267127.818679] XFS (sda1): xfs_do_force_shutdown(0x8) called from line 1320 of file /build/buildd/linux-lts-trusty-3.13.0/fs/xfs/xfs_buf.c.  Return address = 0xffffffffa0250bad
  [267127.818927] XFS (sda1): xfs_imap_to_bp: xfs_trans_read_buf() returned error 5.
  [267127.820914] XFS (sda1): Corruption of in-memory data detected.  Shutting down filesystem
  [267127.829131] XFS (sda1): Please umount the filesystem and rectify the problem(s)

  I've not been able to reproduce this on demand outside our production
  workload. Downgrading the machines back to 3.8.0-44 has returned us to
  stability.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-lts-trusty/+bug/1468039/+subscriptions