kernel-packages team mailing list archive

Thread
Date
[Bug 1400792] Re: Silent ext4 read error

To: kernel-packages@xxxxxxxxxxxxxxxxxxx
From: Paolo Maero <fabrica64@xxxxxxxxx>
Date: Mon, 15 Dec 2014 13:11:58 -0000
Reply-to: Bug 1400792 <1400792@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
I did some tests this weekend. I made a copy of the test directory into a second HD and I did write a small compare utility to check how many bytes are different for each file.
I've see that there's always just one byte different between files, it's always the same bit wrong and always of big files, while files not matching varies between compare executions.

First compare of "pictures" directory files:

files differ: byte 0000025d4862, 8a != 82 (08), file1: /disk1/pictures/photo/movie/2011/2011-03-06_14-57-43.mov, file2: /opt/pictures/photo/movie/2011/2011-03-06_14-57-43.mov
files differ: byte 000011673a62, 68 != 60 (08), file1: /disk1/pictures/photo/movie/2012/2012-02-01_10-11-31.mov, file2: /opt/pictures/photo/movie/2012/2012-02-01_10-11-31.mov
files differ: byte 0000074dd4e2, 7b != 73 (08), file1: /disk1/pictures/photo/movie/2012/2012-02-26_04-30-24.mov, file2: /opt/pictures/photo/movie/2012/2012-02-26_04-30-24.mov
files differ: byte 000025749162, d8 != d0 (08), file1: /disk1/pictures/photo/movie/2012/2012-02-26_04-30-24.mov, file2: /opt/pictures/photo/movie/2012/2012-02-26_04-30-24.mov
files differ: byte 000025749a62, e9 != e1 (08), file1: /disk1/pictures/photo/movie/2012/2012-02-26_04-30-24.mov, file2: /opt/pictures/photo/movie/2012/2012-02-26_04-30-24.mov
files differ: byte 00000a469962, 39 != 31 (08), file1: /disk1/pictures/photo/movie/2012/2012-04-03_07-39-47.mov, file2: /opt/pictures/photo/movie/2012/2012-04-03_07-39-47.mov
files differ: byte 00000b608162, 9e != 96 (08), file1: /disk1/pictures/photo/movie/2012/2012-12-30_19-24-49.mov, file2: /opt/pictures/photo/movie/2012/2012-12-30_19-24-49.mov
files differ: byte 0000048301e2, 6a != 62 (08), file1: /disk1/pictures/photo/movie/2013/2013-01-13_20-57-28.mov, file2: /opt/pictures/photo/movie/2013/2013-01-13_20-57-28.mov
files differ: byte 00000598a4e2, 8b != 83 (08), file1: /disk1/pictures/photo/movie/2013/2013-02-07_06-24-19.mov, file2: /opt/pictures/photo/movie/2013/2013-02-07_06-24-19.mov
files differ: byte 0000087194e2, fb != f3 (08), file1: /disk1/pictures/photo/movie/2013/2013-02-13_06-09-45.mov, file2: /opt/pictures/photo/movie/2013/2013-02-13_06-09-45.mov
files differ: byte 000014bec162, ca != c2 (08), file1: /disk1/pictures/photo/movie/2013/2013-02-14_13-29-22.mov, file2: /opt/pictures/photo/movie/2013/2013-02-14_13-29-22.mov
files differ: byte 00001b82d462, 2a != 22 (08), file1: /disk1/pictures/photo/movie/2013/2013-02-14_13-38-16.mov, file2: /opt/pictures/photo/movie/2013/2013-02-14_13-38-16.mov
files differ: byte 0000063e74e2, 58 != 50 (08), file1: /disk1/pictures/photo/movie/2013/2013-05-23_16-42-36.mov, file2: /opt/pictures/photo/movie/2013/2013-05-23_16-42-36.mov
files differ: byte 00000f5b5162, 4b != 43 (08), file1: /disk1/pictures/photo/movie/2013/2013-08-18_14-43-36.mov, file2: /opt/pictures/photo/movie/2013/2013-08-18_14-43-36.mov
files differ: byte 00000d82fa62, 7f != 77 (08), file1: /disk1/pictures/photo/movie/2013/2013-12-30_06-05-37.mov, file2: /opt/pictures/photo/movie/2013/2013-12-30_06-05-37.mov
files differ: byte 0000046134e2, ff != f7 (08), file1: /disk1/pictures/photo/movie/2013/2013-12-30_06-24-52.mov, file2: /opt/pictures/photo/movie/2013/2013-12-30_06-24-52.mov
files differ: byte 00001bad84e2, 3b != 33 (08), file1: /disk1/pictures/photo/movie/2013/2013-12-30_06-24-52.mov, file2: /opt/pictures/photo/movie/2013/2013-12-30_06-24-52.mov

Second compare:

files differ: byte 0000025d4862, 8a != 82 (08), file1: /disk1/pictures/photo/movie/2011/2011-03-06_14-57-43.mov, file2: /opt/pictures/photo/movie/2011/2011-03-06_14-57-43.mov
files differ: byte 000011673a62, 68 != 60 (08), file1: /disk1/pictures/photo/movie/2012/2012-02-01_10-11-31.mov, file2: /opt/pictures/photo/movie/2012/2012-02-01_10-11-31.mov
files differ: byte 0000074dd4e2, 7b != 73 (08), file1: /disk1/pictures/photo/movie/2012/2012-02-26_04-30-24.mov, file2: /opt/pictures/photo/movie/2012/2012-02-26_04-30-24.mov
files differ: byte 000025749162, d8 != d0 (08), file1: /disk1/pictures/photo/movie/2012/2012-02-26_04-30-24.mov, file2: /opt/pictures/photo/movie/2012/2012-02-26_04-30-24.mov
files differ: byte 000025749a62, e9 != e1 (08), file1: /disk1/pictures/photo/movie/2012/2012-02-26_04-30-24.mov, file2: /opt/pictures/photo/movie/2012/2012-02-26_04-30-24.mov
files differ: byte 00000a469962, 39 != 31 (08), file1: /disk1/pictures/photo/movie/2012/2012-04-03_07-39-47.mov, file2: /opt/pictures/photo/movie/2012/2012-04-03_07-39-47.mov
files differ: byte 00000b608162, 9e != 96 (08), file1: /disk1/pictures/photo/movie/2012/2012-12-30_19-24-49.mov, file2: /opt/pictures/photo/movie/2012/2012-12-30_19-24-49.mov
files differ: byte 0000048301e2, 6a != 62 (08), file1: /disk1/pictures/photo/movie/2013/2013-01-13_20-57-28.mov, file2: /opt/pictures/photo/movie/2013/2013-01-13_20-57-28.mov
files differ: byte 00000598a4e2, 8b != 83 (08), file1: /disk1/pictures/photo/movie/2013/2013-02-07_06-24-19.mov, file2: /opt/pictures/photo/movie/2013/2013-02-07_06-24-19.mov
files differ: byte 0000087194e2, fb != f3 (08), file1: /disk1/pictures/photo/movie/2013/2013-02-13_06-09-45.mov, file2: /opt/pictures/photo/movie/2013/2013-02-13_06-09-45.mov
files differ: byte 000014bec162, ca != c2 (08), file1: /disk1/pictures/photo/movie/2013/2013-02-14_13-29-22.mov, file2: /opt/pictures/photo/movie/2013/2013-02-14_13-29-22.mov
files differ: byte 00001b82d462, 2a != 22 (08), file1: /disk1/pictures/photo/movie/2013/2013-02-14_13-38-16.mov, file2: /opt/pictures/photo/movie/2013/2013-02-14_13-38-16.mov
files differ: byte 0000063e74e2, 58 != 50 (08), file1: /disk1/pictures/photo/movie/2013/2013-05-23_16-42-36.mov, file2: /opt/pictures/photo/movie/2013/2013-05-23_16-42-36.mov
files differ: byte 00000f5b5162, 4b != 43 (08), file1: /disk1/pictures/photo/movie/2013/2013-08-18_14-43-36.mov, file2: /opt/pictures/photo/movie/2013/2013-08-18_14-43-36.mov
files differ: byte 00000d82fa62, 7f != 77 (08), file1: /disk1/pictures/photo/movie/2013/2013-12-30_06-05-37.mov, file2: /opt/pictures/photo/movie/2013/2013-12-30_06-05-37.mov
files differ: byte 0000046134e2, ff != f7 (08), file1: /disk1/pictures/photo/movie/2013/2013-12-30_06-24-52.mov, file2: /opt/pictures/photo/movie/2013/2013-12-30_06-24-52.mov
files differ: byte 00001bad84e2, 3b != 33 (08), file1: /disk1/pictures/photo/movie/2013/2013-12-30_06-24-52.mov, file2: /opt/pictures/photo/movie/2013/2013-12-30_06-24-52.mov

The compare utility list the byte offset, file 1 byte value, file 2 byte
value and the xor of both (all hex values). You may see that end part of
the file offset is always e2 or 62 and the xor is always 8, so it's
always bit 3 different.

I've tested on 10.04 LTS, 14.04.1 LTS (3.13.0-43-generic) and also on
CentOS 7 with similar results. If I do the compare opening the files
with __O_DIRECT flag the problem is minimized, far less bit errors.

I tested my memory extensively with memtest and never got an error. I
also tried with btrfs file-system and I get similar errors without any
btrfs checksum error.

It seems to me that the problem is in the buffering between file-system
and the application and it also seems to me a memory error with a bitrot
under stress, despite my memory seems ok. Having always just one bit
flipped is probably an HW error, not SW one.

I'll do some more testing today and tomorrow on upstrem kernel

BR
Paolo

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1400792

Title:
  Silent ext4 read error

Status in linux package in Ubuntu:
  Incomplete

Bug description:
  Reading multiple times from an ext4 disk directory and getting
  different files...

  I did calculate the sha256sum of all files three times between system
  reboots and not all the files hash values are equal

  I did this on two different PCs / different disks (both on Ubuntu
  14.04 LTS) with the same result... I really don't know what to do
  because no error is logged in the syslog/kernel files

  I have annexed the sha256sum lists of the three times files were read

  Files are not always the same, but always big

  I did the same on btrfs and got the same result...

  ProblemType: Bug
  DistroRelease: Ubuntu 14.04
  Package: linux-image-3.13.0-40-generic 3.13.0-40.69
  ProcVersionSignature: Ubuntu 3.13.0-40.69-generic 3.13.11.10
  Uname: Linux 3.13.0-40-generic x86_64
  ApportVersion: 2.14.1-0ubuntu3.6
  Architecture: amd64
  AudioDevicesInUse:
   USER        PID ACCESS COMMAND
   /dev/snd/controlC1:  paolom     2251 F.... pulseaudio
   /dev/snd/controlC0:  paolom     2251 F.... pulseaudio
  Date: Tue Dec  9 14:43:10 2014
  HibernationDevice: RESUME=UUID=fc099774-a501-42ba-9635-3ce62385a240
  InstallationDate: Installed on 2014-06-16 (176 days ago)
  InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
  MachineType: MICRO-STAR INTERNATIONAL CO.,LTD MS-7360
  ProcFB: 0 nouveaufb
  ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.13.0-40-generic root=/dev/mapper/ubuntu--vg-root ro quiet splash vt.handoff=7
  PulseList:
   Error: command ['pacmd', 'list'] failed with exit code 1: Home directory not accessible: Permission denied
   No PulseAudio daemon running, or not running as session daemon.
  RelatedPackageVersions:
   linux-restricted-modules-3.13.0-40-generic N/A
   linux-backports-modules-3.13.0-40-generic  N/A
   linux-firmware                             1.127.10
  RfKill:
   0: phy0: Wireless LAN
   	Soft blocked: no
   	Hard blocked: no
  SourcePackage: linux
  UpgradeStatus: No upgrade log present (probably fresh install)
  dmi.bios.date: 12/04/2007
  dmi.bios.vendor: American Megatrends Inc.
  dmi.bios.version: V1.6
  dmi.board.asset.tag: To Be Filled By O.E.M.
  dmi.board.name: MS-7360
  dmi.board.vendor: MICRO-STAR INTERNATIONAL CO.,LTD
  dmi.board.version: 1.0
  dmi.chassis.asset.tag: To Be Filled By O.E.M.
  dmi.chassis.type: 3
  dmi.chassis.vendor: MICRO-STAR INTERNATIONAL CO.,LTD
  dmi.chassis.version: 1.0
  dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvrV1.6:bd12/04/2007:svnMICRO-STARINTERNATIONALCO.,LTD:pnMS-7360:pvr1.0:rvnMICRO-STARINTERNATIONALCO.,LTD:rnMS-7360:rvr1.0:cvnMICRO-STARINTERNATIONALCO.,LTD:ct3:cvr1.0:
  dmi.product.name: MS-7360
  dmi.product.version: 1.0
  dmi.sys.vendor: MICRO-STAR INTERNATIONAL CO.,LTD

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1400792/+subscriptions
References

[Bug 1400792] [NEW] Silent ext4 read error
From: Paolo Maero, 2014-12-09