touch-packages team mailing list archive

[DD Park <deedee6905@xxxxxxxxx>]

Hi. I checked into another server that is waiting to upgrade >16TB
pending this issue get resolved.
Can you provide more information as to the statement "very large
number of hard link". This other system has approx 150K hardlinks and
growing daily. I'm not sure if we are talking about millions or
billions or thousands of hardlinks. Is 150K hardlinks considered "very
large" for this problem?

On 11/13/15, craigslist reply <deedee6905@xxxxxxxxx> wrote:
> FYI. There was about 15TB (back when I initially ran into the problem,
> maybe about 11-14TB) of hardlinks.
> The Hardlinks are used so that rsync can handle directory name changes and
> not have to copy the whole directory over again. Thus each file would have
> a hardlink. I estimate maybe 40K+ or 80K+ files (I'm not sure if rsync
> counter includes hardlinks or not, but there would have been more than 80K
> files according to rsync) the fsck would use up all available memory. Even
> if I added mega swap space, the program would use it all up. I'm not super
> techie on this, so I can add to my test a temporary removal of all the
> hardlinks after I resolve my own issue of getting a chassis to house all
> the harddrives. I don't expect to have much time to resolve my chassis
> problem within a  month, but now that I know there are some possible
> workarounds, and path moving forward, I can try to raise the priority back
> on this issue. Don't expect quick turn around on my part, but I do need to
> get that file system back operational to get data off the drives so I
> eventually will need to get it to a state where I can do the fsck test.
>
> oh btw, when initially the file system was clean, fsck would work no
> problem as I tested that before using the system. It was after there was
> some file system problems that I later ran fsck and ran into problems. I
> don't know if these people who are successfully running fsck are doing so
> from a non-clean file system.
>
> On Fri, Nov 13, 2015 at 12:09 PM, Theodore Ts'o <tytso@xxxxxxx> wrote:
>
>> There are plenty of people who *are* able to check file systems larger
>> than 16TB.  So talking about it in terms "that problem" doesn't make much
>> sense.
>>
>> There is a known issue where if you are using a huge number of hard links
>> (if you are using a backup solution which uses hard links and a huge
>> number
>> of parallel directory hierarchies) e2fsck uses a large amount of memory.
>> I
>> don't know if that's what you are doing, but if that's what you are
>> doing,
>> it wouldn't surprise me if you are running out of memory.  You should
>> have
>> gotten an explicit "you ran out of memory" error instead of a seg fault,
>> though.
>>
>> There is a way you can use scratch files for some of the data structures
>> that e2fsck needs to track all of the hard links; see the
>> "The [scratch_files] Stanza" section in the e2fsck.conf.  However, this
>> is
>> going to be *SLOW*.    The primary use of this was for people who were
>> using 32-bit systems and who didn't have the address space for the memory
>> requirements for large file systems with a large number of hard links.
>>  You may be better off just trying to configure our swap space.  (All of
>> this is assuming the problem is that the file system has a very large
>> number of hard link farms.  Have you confirmed that when you configured
>> 25GB of swap, that the system actually used that amount of swap)?
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/1345682
>>
>> Title:
>>   fsck on 24TB ext4 keeps crashing
>>
>> Status in e2fsprogs package in Ubuntu:
>>   Incomplete
>>
>> Bug description:
>>   date; fsck -vy /dev/mapper/raid61p1 ;date
>>   Thu Jul 17 11:54:21 PDT 2014
>>   fsck from util-linux 2.20.1
>>   I can't get my ext4 24TB to fsck clean. there came a whole bunch of
>> file
>> stat problems after a clean reboot.
>>   I tried 3 times so far, and each does something like segfault. I looked
>> twice, and the errors were different,
>>   but was able to capture one output. The system has 6GB ram, and it used
>> up all the memory. ended up adding 25GB
>>   of swap as fsck seems to use up a huge amount of memory. After a long
>> while, usually after mem use is >90%,
>>   I come back to see the fsck has crashed and file system is still not
>> clean.
>>
>>   The reboot was clean and should not  have caused any corruption. the
>>   system is using ubuntu-14.04
>>
>>   # uname -a
>>   Linux gigabyte133 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08
>> UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
>>   root@gigabyte133:~#
>>
>>   I had to use the 64 bit version since the 32 bit version I couldn't get
>> more than 15TB file system built and read
>>   that there were compatibility problems between 32bit version with the
>> -O
>> 64bit and the 64 bit version so just
>>   built the whole thing in 64 bit ubuntu..
>>
>>
>>   e2fsck 1.42.9 (4-Feb-2014)
>>
>>
>>   /dev/mapper/raid61p1 contains a file system with errors, check forced.
>>   Pass 1: Checking inodes, blocks, and sizes
>>   Inode 203167820 has compression flag set on filesystem without
>> compression support.  Clear? yes
>>
>>   Signal (11) SIGSEGV si_code=SI_KERNEL fault addr=(nil)
>>   fsck.ext4[0x4266f1]
>>   /lib/x86_64-linux-gnu/libc.so.6(+0x36ff0)[0x7fd38d087ff0]
>>
>> /lib/x86_64-linux-gnu/libext2fs.so.2(ext2fs_rb_next+0x23)[0x7fd38dc7bc43]
>>   /lib/x86_64-linux-gnu/libext2fs.so.2(+0x10670)[0x7fd38dc5e670]
>>   fsck.ext4[0x4100f6]# lsb_release -va
>>   No LSB modules are available.
>>   Distributor ID: Ubuntu
>>   Description:    Ubuntu 14.04 LTS
>>   Release:        14.04
>>   Codename:       trusty
>>
>>   /lib/x86_64-linux-gnu/libext2fs.so.2(+0x11c3d)[0x7fd38dc5fc3d]
>>   /lib/x86_64-linux-gnu/libext2fs.so.2(+0x11f78)[0x7fd38dc5ff78]
>>
>> /lib/x86_64-linux-gnu/libext2fs.so.2(ext2fs_block_iterate3+0xa13)[0x7fd38dc60b13]
>>   fsck.ext4[0x4115c4]
>>   fsck.ext4[0x412699]
>>   fsck.ext4[0x412761]
>>
>> /lib/x86_64-linux-gnu/libext2fs.so.2(ext2fs_get_next_inode_full+0x59)[0x7fd38dc6c9a9]
>>   fsck.ext4(e2fsck_pass1+0x8d8)[0x4130c8]
>>   fsck.ext4(e2fsck_run+0x52)[0x40deb2]
>>   fsck.ext4(main+0xd27)[0x40a0e7]
>>   /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7fd38d072ec5]
>>   fsck.ext4[0x40bde6]
>>   Thu Jul 17 15:55:01 PDT 2014
>>
>>   # lsb_release -va
>>   No LSB modules are available.
>>   Distributor ID: Ubuntu
>>   Description:    Ubuntu 14.04 LTS
>>   Release:        14.04
>>   Codename:       trusty
>>
>>   ProblemType: Bug
>>   DistroRelease: Ubuntu 14.04
>>   Package: util-linux 2.20.1-5.1ubuntu20.1
>>   ProcVersionSignature: Ubuntu 3.13.0-32.57-generic 3.13.11.4
>>   Uname: Linux 3.13.0-32-generic x86_64
>>   NonfreeKernelModules: zfs zunicode zavl zcommon znvpair
>>   ApportVersion: 2.14.1-0ubuntu3.2
>>   Architecture: amd64
>>   Date: Sat Jul 19 18:49:32 2014
>>   InstallationDate: Installed on 2014-07-11 (8 days ago)
>>   InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64
>> (20140417)
>>   ProcEnviron:
>>    TERM=xterm
>>    PATH=(custom, no user)
>>    LANG=en_US.UTF-8
>>    SHELL=/bin/bash
>>   SourcePackage: util-linux
>>   UpgradeStatus: No upgrade log present (probably fresh install)
>>
>> To manage notifications about this bug go to:
>>
>> https://bugs.launchpad.net/ubuntu/+source/e2fsprogs/+bug/1345682/+subscriptions
>>
>

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to e2fsprogs in Ubuntu.
https://bugs.launchpad.net/bugs/1345682

Title:
  fsck on 24TB ext4 keeps crashing

Status in e2fsprogs package in Ubuntu:
  Incomplete

Bug description:
  date; fsck -vy /dev/mapper/raid61p1 ;date
  Thu Jul 17 11:54:21 PDT 2014
  fsck from util-linux 2.20.1
  I can't get my ext4 24TB to fsck clean. there came a whole bunch of file stat problems after a clean reboot.
  I tried 3 times so far, and each does something like segfault. I looked twice, and the errors were different,
  but was able to capture one output. The system has 6GB ram, and it used up all the memory. ended up adding 25GB
  of swap as fsck seems to use up a huge amount of memory. After a long while, usually after mem use is >90%,
  I come back to see the fsck has crashed and file system is still not clean.

  The reboot was clean and should not  have caused any corruption. the
  system is using ubuntu-14.04

  # uname -a
  Linux gigabyte133 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
  root@gigabyte133:~# 

  I had to use the 64 bit version since the 32 bit version I couldn't get more than 15TB file system built and read 
  that there were compatibility problems between 32bit version with the -O 64bit and the 64 bit version so just
  built the whole thing in 64 bit ubuntu..

  
  e2fsck 1.42.9 (4-Feb-2014)

  
  /dev/mapper/raid61p1 contains a file system with errors, check forced.
  Pass 1: Checking inodes, blocks, and sizes
  Inode 203167820 has compression flag set on filesystem without compression support.  Clear? yes

  Signal (11) SIGSEGV si_code=SI_KERNEL fault addr=(nil)
  fsck.ext4[0x4266f1]
  /lib/x86_64-linux-gnu/libc.so.6(+0x36ff0)[0x7fd38d087ff0]
  /lib/x86_64-linux-gnu/libext2fs.so.2(ext2fs_rb_next+0x23)[0x7fd38dc7bc43]
  /lib/x86_64-linux-gnu/libext2fs.so.2(+0x10670)[0x7fd38dc5e670]
  fsck.ext4[0x4100f6]# lsb_release -va
  No LSB modules are available.
  Distributor ID: Ubuntu
  Description:    Ubuntu 14.04 LTS
  Release:        14.04
  Codename:       trusty

  /lib/x86_64-linux-gnu/libext2fs.so.2(+0x11c3d)[0x7fd38dc5fc3d]
  /lib/x86_64-linux-gnu/libext2fs.so.2(+0x11f78)[0x7fd38dc5ff78]
  /lib/x86_64-linux-gnu/libext2fs.so.2(ext2fs_block_iterate3+0xa13)[0x7fd38dc60b13]
  fsck.ext4[0x4115c4]
  fsck.ext4[0x412699]
  fsck.ext4[0x412761]
  /lib/x86_64-linux-gnu/libext2fs.so.2(ext2fs_get_next_inode_full+0x59)[0x7fd38dc6c9a9]
  fsck.ext4(e2fsck_pass1+0x8d8)[0x4130c8]
  fsck.ext4(e2fsck_run+0x52)[0x40deb2]
  fsck.ext4(main+0xd27)[0x40a0e7]
  /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7fd38d072ec5]
  fsck.ext4[0x40bde6]
  Thu Jul 17 15:55:01 PDT 2014

  # lsb_release -va
  No LSB modules are available.
  Distributor ID: Ubuntu
  Description:    Ubuntu 14.04 LTS
  Release:        14.04
  Codename:       trusty

  ProblemType: Bug
  DistroRelease: Ubuntu 14.04
  Package: util-linux 2.20.1-5.1ubuntu20.1
  ProcVersionSignature: Ubuntu 3.13.0-32.57-generic 3.13.11.4
  Uname: Linux 3.13.0-32-generic x86_64
  NonfreeKernelModules: zfs zunicode zavl zcommon znvpair
  ApportVersion: 2.14.1-0ubuntu3.2
  Architecture: amd64
  Date: Sat Jul 19 18:49:32 2014
  InstallationDate: Installed on 2014-07-11 (8 days ago)
  InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
  ProcEnviron:
   TERM=xterm
   PATH=(custom, no user)
   LANG=en_US.UTF-8
   SHELL=/bin/bash
  SourcePackage: util-linux
  UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/e2fsprogs/+bug/1345682/+subscriptions