kernel-packages team mailing list archive

Thread
Date
[Bug 1596550] Re: Metadata corruption detected at xfs_attr3_leaf_write_verify+0xd7/0xf0

To: kernel-packages@xxxxxxxxxxxxxxxxxxx
From: Bjoern Teipel <bjoern.teipel@xxxxxxxxxxxxx>
Date: Mon, 27 Jun 2016 14:47:35 -0000
Reply-to: Bug 1596550 <1596550@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
apport information

** Tags added: apport-collected

** Description changed:

  We noticed a XFS metadata corruption once we ran a lot of small write
  IOs on SSDs in our OpenStack swift environment:
  
  [1468860.211158] XFS (sdax): Metadata corruption detected at xfs_attr3_leaf_write_verify+0xd7/0xf0 [xfs], block 0x7c99480
  [1468860.211195] XFS (sdax): Unmount and run xfs_repair
  [1468860.211215] XFS (sdax): First 64 bytes of corrupted metadata buffer:
  [1468860.211247] ffff880630f66000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00  ................
  [1468860.211268] ffff880630f66010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00  ..... ..........
  [1468860.211289] ffff880630f66020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  [1468860.211309] ffff880630f66030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  [1468860.211328] XFS (sdax): xfs_do_force_shutdown(0x8) called from line 1254 of file /build/linux-lts-xenial-7RlTta/linux-lts-xenial-4.4.0/fs/xfs/xfs_buf.c.  Return address = 0x
  ffffffffc068f616
  [1468860.212214] XFS (sdax): Corruption of in-memory data detected.  Shutting down filesystem
  [1468860.212232] XFS (sdax): Please umount the filesystem and rectify the problem(s)
  [1468860.212323] XFS (sdax): xfs_do_force_shutdown(0x1) called from line 315 of file /build/linux-lts-xenial-7RlTta/linux-lts-xenial-4.4.0/fs/xfs/xfs_trans_buf.c.  Return address
   = 0xffffffffc06bdda2
  [1468860.261436] XFS (sdax): xfs_log_force: error -5 returned.
  
  This error is reported with linux-generic-lts-xenial @4.4.0.22.12 on a
  XFS filesystem formatted with 1024 as inode size and mounted with
  
  rw,noatime,nodiratime,attr2,nobarrier,inode64,logbufs=8,sunit=512,swidth=512,noquota
  
  
  For us this issue seems to be reproducible after several hours of stress testing.
  
  cat /proc/version_signature
  Ubuntu 4.4.0-22.40~14.04.1-generic 4.4.8
  
  Description:    Ubuntu 14.04.3 LTS
  Release:        14.04
+ --- 
+ AlsaDevices:
+  total 0
+  crw-rw---- 1 root audio 116,  1 Jun 10 13:19 seq
+  crw-rw---- 1 root audio 116, 33 Jun 10 13:19 timer
+ AplayDevices: Error: [Errno 2] No such file or directory
+ ApportVersion: 2.14.1-0ubuntu3.11
+ Architecture: amd64
+ ArecordDevices: Error: [Errno 2] No such file or directory
+ AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
+ CRDA: Error: [Errno 2] No such file or directory
+ DistroRelease: Ubuntu 14.04
+ IwConfig: Error: [Errno 2] No such file or directory
+ MachineType: HP ProLiant DL380 Gen9
+ Package: linux (not installed)
+ PciMultimedia:
+  
+ ProcEnviron:
+  TERM=screen
+  PATH=(custom, no user)
+  XDG_RUNTIME_DIR=<set>
+  LANG=en_US.UTF-8
+  SHELL=/bin/bash
+ ProcFB:
+  
+ ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-22-generic root=/dev/mapper/lxc-root00 ro biosdevname=1 net.ifnames=0 usbcore.autosuspend=-1 vga=normal nomodeset nomdmonddf nomdmonisw crashkernel=1024M-:128M
+ ProcVersionSignature: Ubuntu 4.4.0-22.40~14.04.1-generic 4.4.8
+ RelatedPackageVersions:
+  linux-restricted-modules-4.4.0-22-generic N/A
+  linux-backports-modules-4.4.0-22-generic  N/A
+  linux-firmware                            1.127.15
+ RfKill: Error: [Errno 2] No such file or directory
+ Tags:  trusty
+ Uname: Linux 4.4.0-22-generic x86_64
+ UpgradeStatus: No upgrade log present (probably fresh install)
+ UserGroups:
+  
+ _MarkForUpload: True
+ dmi.bios.date: 07/20/2015
+ dmi.bios.vendor: HP
+ dmi.bios.version: P89
+ dmi.chassis.type: 23
+ dmi.chassis.vendor: HP
+ dmi.modalias: dmi:bvnHP:bvrP89:bd07/20/2015:svnHP:pnProLiantDL380Gen9:pvr:cvnHP:ct23:cvr:
+ dmi.product.name: ProLiant DL380 Gen9
+ dmi.sys.vendor: HP

** Attachment added: "BootDmesg.txt"
   https://bugs.launchpad.net/bugs/1596550/+attachment/4691132/+files/BootDmesg.txt

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1596550

Title:
  Metadata corruption detected at xfs_attr3_leaf_write_verify+0xd7/0xf0

Status in linux package in Ubuntu:
  Confirmed

Bug description:
  We noticed a XFS metadata corruption once we ran a lot of small write
  IOs on SSDs in our OpenStack swift environment:

  [1468860.211158] XFS (sdax): Metadata corruption detected at xfs_attr3_leaf_write_verify+0xd7/0xf0 [xfs], block 0x7c99480
  [1468860.211195] XFS (sdax): Unmount and run xfs_repair
  [1468860.211215] XFS (sdax): First 64 bytes of corrupted metadata buffer:
  [1468860.211247] ffff880630f66000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00  ................
  [1468860.211268] ffff880630f66010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00  ..... ..........
  [1468860.211289] ffff880630f66020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  [1468860.211309] ffff880630f66030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  [1468860.211328] XFS (sdax): xfs_do_force_shutdown(0x8) called from line 1254 of file /build/linux-lts-xenial-7RlTta/linux-lts-xenial-4.4.0/fs/xfs/xfs_buf.c.  Return address = 0x
  ffffffffc068f616
  [1468860.212214] XFS (sdax): Corruption of in-memory data detected.  Shutting down filesystem
  [1468860.212232] XFS (sdax): Please umount the filesystem and rectify the problem(s)
  [1468860.212323] XFS (sdax): xfs_do_force_shutdown(0x1) called from line 315 of file /build/linux-lts-xenial-7RlTta/linux-lts-xenial-4.4.0/fs/xfs/xfs_trans_buf.c.  Return address
   = 0xffffffffc06bdda2
  [1468860.261436] XFS (sdax): xfs_log_force: error -5 returned.

  This error is reported with linux-generic-lts-xenial @4.4.0.22.12 on a
  XFS filesystem formatted with 1024 as inode size and mounted with

  rw,noatime,nodiratime,attr2,nobarrier,inode64,logbufs=8,sunit=512,swidth=512,noquota

  
  For us this issue seems to be reproducible after several hours of stress testing.

  cat /proc/version_signature
  Ubuntu 4.4.0-22.40~14.04.1-generic 4.4.8

  Description:    Ubuntu 14.04.3 LTS
  Release:        14.04
  --- 
  AlsaDevices:
   total 0
   crw-rw---- 1 root audio 116,  1 Jun 10 13:19 seq
   crw-rw---- 1 root audio 116, 33 Jun 10 13:19 timer
  AplayDevices: Error: [Errno 2] No such file or directory
  ApportVersion: 2.14.1-0ubuntu3.11
  Architecture: amd64
  ArecordDevices: Error: [Errno 2] No such file or directory
  AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
  CRDA: Error: [Errno 2] No such file or directory
  DistroRelease: Ubuntu 14.04
  IwConfig: Error: [Errno 2] No such file or directory
  MachineType: HP ProLiant DL380 Gen9
  Package: linux (not installed)
  PciMultimedia:
   
  ProcEnviron:
   TERM=screen
   PATH=(custom, no user)
   XDG_RUNTIME_DIR=<set>
   LANG=en_US.UTF-8
   SHELL=/bin/bash
  ProcFB:
   
  ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-22-generic root=/dev/mapper/lxc-root00 ro biosdevname=1 net.ifnames=0 usbcore.autosuspend=-1 vga=normal nomodeset nomdmonddf nomdmonisw crashkernel=1024M-:128M
  ProcVersionSignature: Ubuntu 4.4.0-22.40~14.04.1-generic 4.4.8
  RelatedPackageVersions:
   linux-restricted-modules-4.4.0-22-generic N/A
   linux-backports-modules-4.4.0-22-generic  N/A
   linux-firmware                            1.127.15
  RfKill: Error: [Errno 2] No such file or directory
  Tags:  trusty
  Uname: Linux 4.4.0-22-generic x86_64
  UpgradeStatus: No upgrade log present (probably fresh install)
  UserGroups:
   
  _MarkForUpload: True
  dmi.bios.date: 07/20/2015
  dmi.bios.vendor: HP
  dmi.bios.version: P89
  dmi.chassis.type: 23
  dmi.chassis.vendor: HP
  dmi.modalias: dmi:bvnHP:bvrP89:bd07/20/2015:svnHP:pnProLiantDL380Gen9:pvr:cvnHP:ct23:cvr:
  dmi.product.name: ProLiant DL380 Gen9
  dmi.sys.vendor: HP

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1596550/+subscriptions
References

[Bug 1596550] [NEW] Metadata corruption detected at xfs_attr3_leaf_write_verify+0xd7/0xf0
From: Bjoern Teipel, 2016-06-27