linuxdcpp-team team mailing list archive

[maksis <1909861@xxxxxxxxxxxxxxxxxx>]

I ran some additional tests with different read buffer sizes on
different platforms. Cached hashing speed on macOS is CPU-bound and the
same most likely applies to Windows results as well (both disks are able
to read 1000 MB/s+). 1024 KB buffer size looks quite good based on the
tests.


macOS 11 (MacBook Pro, SSD)

256 KB:

Cached: 2 files (20.76 GiB) have been hashed in 59 seconds (357.97 MiB/s)
F_NOCACHE: 2 files (20.76 GiB) have been hashed in 1 minute 54 seconds (186.08 MiB/s)

1024 KB:

Cached: 2 files (20.76 GiB) have been hashed in 1 minute 0 second (354.01 MiB/s)
F_NOCACHE: 2 files (20.76 GiB) have been hashed in 1 minute 17 seconds (275.21 MiB/s)

4096 KB:

Cached: 2 files (20.76 GiB) have been hashed in 59 seconds (355.90 MiB/s)
F_NOCACHE: 2 files (20.76 GiB) have been hashed in 1 minute 6 seconds (320.85 MiB/s)


Ubuntu 20.04 (Celeron N3150, internal HDD)

256 KB:

Cached: 671 files (21.70 GiB) in 9 directories have been hashed in 3 minutes 58 seconds (93.08 MiB/s)
O_DIRECT: 671 files (21.70 GiB) in 9 directories have been hashed in 4 minutes 10 seconds (88.62 MiB/s)

1024 KB:

POSIX_FADV_DONTNEED: 671 files (21.70 GiB) in 9 directories have been hashed in 3 minutes 55 seconds (94.55 MiB/s)
O_DIRECT: 671 files (21.70 GiB) in 9 directories have been hashed in 4 minutes 1 second (91.95 MiB/s)

4096 KB:

POSIX_FADV_DONTNEED: Hashing finished: 671 files (21.70 GiB) in 9 directories have been hashed in 4 minutes 58 seconds (74.42 MiB/s)
O_DIRECT: 671 files (21.70 GiB) in 9 directories have been hashed in 4 minutes 55 seconds (75.24 MiB/s)


Windows 10 (AMD 2700X, SSD)

OVERLAPPED, 256 KB: 191 files (21,36 GiB) have been hashed in 39 seconds (551,41 MiB/s) 
OVERLAPPED, 1024 KB: 191 files (21,36 GiB) have been hashed in 37 seconds (577,62 MiB/s) 
OVERLAPPED, 4096 KB: 191 files (21,36 GiB) have been hashed in 37 seconds (581,33 MiB/s)

-- 
You received this bug notification because you are a member of
Dcplusplus-team, which is subscribed to DC++.
https://bugs.launchpad.net/bugs/1909861

Title:
  FileReader is not thread safe on Linux

Status in DC++:
  New

Bug description:
  FileReader::readMapped currently modifies the global SIGBUS handler in
  order to catch read errors:

  https://sourceforge.net/p/dcplusplus/code/ci/default/tree/dcpp/FileReader.cpp#l289

  Since the function can be called concurrently from different threads
  (currently hashing/queue recheck/sfv check in DC++) and each of them
  sets and resets the SIGBUS handler, there's a high risk that the
  application will crash in case of read errors as they aren't being
  handler properly.

  More information about the caveats:
  https://www.sublimetext.com/blog/articles/use-mmap-with-care

  These issues are much more likely to happen with AirDC++ as it uses
  multiple threads for hashing. Read errors caused rather nasty crashes
  with corrupted stack traces for one user, but luckily he was able to
  catch the SIGBUS signal with gdb.

  I didn't even spend time in trying to figure out how to make the
  mapped reads work properly, as based on my testing the basic
  FileReader::readCached function is noticeably faster:

  readMapped: 671 files (21.70 GiB) in 9 directories have been hashed in 4 minutes 21 seconds (84.87 MiB/s)
  readCached: 671 files (21.70 GiB) in 9 directories have been hashed in 3 minutes 58 seconds (93.08 MiB/s)

  FileReader::readMapped is now disabled in AirDC++, as I can't see any
  benefits from using it. The included setjmp.h header is even causing
  issues when using clang for compiling on Linux:
  https://bugs.gentoo.org/731676

To manage notifications about this bug go to:
https://bugs.launchpad.net/dcplusplus/+bug/1909861/+subscriptions