linuxdcpp-team team mailing list archive

[maksis <1909861@xxxxxxxxxxxxxxxxxx>]

AirDC++ actually has some kind of support for opening files O_DIRECT:
https://github.com/airdcpp/airdcpp-
windows/blob/b863d8626d95d0ee483572a5139f8f569b558c3f/airdcpp/airdcpp/File.cpp#L380-L394
(BUFFER_NONE isn't currently being used anywhere though when opening
files)

FileReader::readCached currently uses the fadvise code path with
POSIX_FADV_SEQUENTIAL.

I compiled AirDC++ with a modified FileReader that opens the file with
BUFFER_NONE (O_DIRECT) and did a quick benchmark:

O_DIRECT: Hashing finished: 671 files (21.70 GiB) in 9 directories have been hashed in 4 minutes 10 seconds (88.62 MiB/s)
Normal caching (POSIX_FADV_SEQUENTIAL): Hashing finished: 671 files (21.70 GiB) in 9 directories have been hashed in 4 minutes 1 second (92.07 MiB/s)

Test builds for Linux: https://web-builds.airdcpp.net/develop/

So O_DIRECT is slower, but not by much. I still wonder whether the
difference would be bigger with different disk setups (e.g. with network
disks).

-- 
You received this bug notification because you are a member of
Dcplusplus-team, which is subscribed to DC++.
https://bugs.launchpad.net/bugs/1909861

Title:
  FileReader is not thread safe on Linux

Status in DC++:
  New

Bug description:
  FileReader::readMapped currently modifies the global SIGBUS handler in
  order to catch read errors:

  https://sourceforge.net/p/dcplusplus/code/ci/default/tree/dcpp/FileReader.cpp#l289

  Since the function can be called concurrently from different threads
  (currently hashing/queue recheck/sfv check in DC++) and each of them
  sets and resets the SIGBUS handler, there's a high risk that the
  application will crash in case of read errors as they aren't being
  handler properly.

  More information about the caveats:
  https://www.sublimetext.com/blog/articles/use-mmap-with-care

  These issues are much more likely to happen with AirDC++ as it uses
  multiple threads for hashing. Read errors caused rather nasty crashes
  with corrupted stack traces for one user, but luckily he was able to
  catch the SIGBUS signal with gdb.

  I didn't even spend time in trying to figure out how to make the
  mapped reads work properly, as based on my testing the basic
  FileReader::readCached function is noticeably faster:

  readMapped: 671 files (21.70 GiB) in 9 directories have been hashed in 4 minutes 21 seconds (84.87 MiB/s)
  readCached: 671 files (21.70 GiB) in 9 directories have been hashed in 3 minutes 58 seconds (93.08 MiB/s)

  FileReader::readMapped is now disabled in AirDC++, as I can't see any
  benefits from using it. The included setjmp.h header is even causing
  issues when using clang for compiling on Linux:
  https://bugs.gentoo.org/731676

To manage notifications about this bug go to:
https://bugs.launchpad.net/dcplusplus/+bug/1909861/+subscriptions