sts-sponsors team mailing list archive

[Matthew Ruffell <1876230@xxxxxxxxxxxxxxxxxx>]

To answer question 1, I went and checked every rdepends package:

gdnsd: dynamically links to -lurcu-qsbr
$ ldd /usr/sbin/gdnsd
liburcu-qsbr.so.6 => /usr/lib/x86_64-linux-gnu/liburcu-qsbr.so.6 (0x00007f33bfde5000)

glusterfs: The only package I am not entirely sure about. Only glusterd uses urcu:
ubuntu@ubuntu:~/glusterfs-3.13.2$ grep -Rin "<urcu" .
./config.h.in:200:/* Define to 1 if you have the <urcu-bp.h> header file. */
./config.h.in:203:/* Define to 1 if you have the <urcu/cds.h> header file. */
./xlators/mgmt/glusterd/src/glusterd-rcu.h:14:#include <urcu-bp.h>
./xlators/mgmt/glusterd/src/glusterd-rcu.h:15:#include <urcu/rculist.h>
./xlators/mgmt/glusterd/src/glusterd-rcu.h:16:#include <urcu/compiler.h>
./xlators/mgmt/glusterd/src/glusterd-rcu.h:17:#include <urcu/uatomic.h>
./xlators/mgmt/glusterd/src/glusterd-rcu.h:18:#include <urcu-call-rcu.h>
./xlators/mgmt/glusterd/src/glusterd-conn-helper.c:15:#include <urcu/rculist.h>
$ ldd /usr/sbin/glusterd | grep urcu
<blank>
No mention of static linking either:
ubuntu@ubuntu:~/glusterfs-3.13.2$ grep -Rin "\.a" . | grep urcu
<blank>
The library linker settings are in ./configure.ac:
dnl Check for userspace-rcu
PKG_CHECK_MODULES([URCU], [liburcu-bp], [],
  [AC_CHECK_HEADERS([urcu-bp.h],
     [URCU_LIBS='-lurcu-bp'],
     AC_MSG_ERROR([liburcu-bp not found]))])
PKG_CHECK_MODULES([URCU_CDS], [liburcu-cds >= 0.8], [],
  [PKG_CHECK_MODULES([URCU_CDS], [liburcu-cds >= 0.7],
    [AC_DEFINE(URCU_OLD, 1, [Define if liburcu 0.6 or 0.7 is found])],
    [AC_CHECK_HEADERS([urcu/cds.h],
      [AC_DEFINE(URCU_OLD, 1, [Define if liburcu 0.6 or 0.7 is found])
       URCU_CDS_LIBS='-lurcu-cds'],
      [AC_MSG_ERROR([liburcu-cds not found])])])])
I ran ldd over all glusterfs binaries which are listed by dpkg -L, but they all came back negative for urcu. From what I can tell, glusterd from the xlators directory is either not built, or does not link against urcu in Ubuntu.

knot: dynamically links to -lurcu
ubuntu@ubuntu:~$ ldd /usr/sbin/knotd
liburcu.so.6 => /usr/lib/x86_64-linux-gnu/liburcu.so.6 (0x00007fc9d1e8b000)

lttng: dynamically links to various urcu libraries
$ ldd /usr/bin/lttng
liburcu-common.so.6 => /usr/lib/x86_64-linux-gnu/liburcu-common.so.6 (0x00007f6711a2f000)
liburcu.so.6 => /usr/lib/x86_64-linux-gnu/liburcu.so.6 (0x00007f6711827000)
liburcu-cds.so.6 => /usr/lib/x86_64-linux-gnu/liburcu-cds.so.6 (0x00007f671161d000)

multipath-tools: dynamically links to -lurcu
$ ldd /sbin/multipath
liburcu.so.6 => /usr/lib/x86_64-linux-gnu/liburcu.so.6 (0x00007fe63ace1000)

netsniff-ng: dynamically links to -lurcu
~/netsniff-ng-0.6.4$ grep -Rin "urcu"
flowtop/Makefile:1:flowtop-libs =	-lurcu \
ui.h:7:#include <urcu/list.h>
flowtop.c:28:#include <urcu.h>
flowtop.c:29:#include <urcu/list.h>
flowtop.c:30:#include <urcu/rculist.h>
INSTALL:28: - liburcu:                flowtop
$ ldd /usr/sbin/flowtop
liburcu.so.6 => /usr/lib/x86_64-linux-gnu/liburcu.so.6 (0x00007f7561a5a000)

sheepdog: this program doesn't link against urcu at all! It only uses the <urcu/uatomic.h> header file and not anything more.
search for urcu in sheepdog: https://paste.ubuntu.com/p/VPKr4pWtQg/
explanation found in d/changelog: https://paste.ubuntu.com/p/K8XwjK2czD/

ust: dynamically links to various urcu libraries
$ ldd /usr/lib/x86_64-linux-gnu/liblttng-ust-cyg-profile-fast.so.0.0.0
liburcu-bp.so.6 => /usr/lib/x86_64-linux-gnu/liburcu-bp.so.6 (0x00007f91b0c13000)
liburcu-cds.so.6 => /usr/lib/x86_64-linux-gnu/liburcu-cds.so.6 (0x00007f91b01f4000)

>From what I can tell, no rdepends package statically links to liburcu,
and they all use dynamic linking, meaning no packages will need to be
rebuilt for this SRU.

-- 
You received this bug notification because you are a member of STS
Sponsors, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1876230

Title:
  liburcu: Enable MEMBARRIER_CMD_PRIVATE_EXPEDITED to address
  performance problems with MEMBARRIER_CMD_SHARED

Status in liburcu package in Ubuntu:
  Fix Released
Status in liburcu source package in Bionic:
  In Progress

Bug description:
  [Impact]

  In Linux 4.3, a new syscall was defined, called "membarrier". This
  systemcall was defined specifically for use in userspace-rcu (liburcu)
  to speed up the fast path / reader side of the library. The original
  implementation in Linux 4.3 only supported the MEMBARRIER_CMD_SHARED
  subcommand of the membarrier syscall.

  MEMBARRIER_CMD_SHARED executes a memory barrier on all threads from
  all processes running on the system. When it exits, the userspace
  thread which called it is guaranteed that all running threads share
  the same world view in regards to userspace addresses which are
  consumed by readers and writers.

  The problem with MEMBARRIER_CMD_SHARED is system calls made in this
  fashion can block, since it deploys a barrier across all threads in a
  system, and some other threads can be waiting on blocking operations,
  and take time to reach the barrier.

  In Linux 4.14, this was addressed by adding the
  MEMBARRIER_CMD_PRIVATE_EXPEDITED command to the membarrier syscall. It
  only targets threads which share the same mm as the thread calling the
  membarrier syscall, aka, threads in the current process, and not all
  threads / processes in the system.

  Calls to membarrier with the MEMBARRIER_CMD_PRIVATE_EXPEDITED command
  are guaranteed non-blocking, due to using inter-processor interrupts
  to implement memory barriers.

  Because of this, membarrier calls that use
  MEMBARRIER_CMD_PRIVATE_EXPEDITED are much faster than those that use
  MEMBARRIER_CMD_SHARED.

  Since Bionic uses a 4.15 kernel, all kernel requirements are met, and
  this SRU is to enable support for MEMBARRIER_CMD_PRIVATE_EXPEDITED in
  the liburcu package.

  This brings the performance of the liburcu library back in line to
  where it was in Trusty, as this particular user has performance
  problems upon upgrading from Trusty to Bionic.

  [Test]

  Testing performance is heavily dependant on the application which
  links against liburcu, and the workload which it executes.

  A test package is available in the following ppa:
  https://launchpad.net/~mruffell/+archive/ubuntu/sf276198-test

  For the sake of testing, we can use the benchmarks provided in the
  liburcu source code. Download a copy of the source code for liburcu
  either from the repos or from github:

  $ pull-lp-source liburcu bionic
  # OR
  $ git clone https://github.com/urcu/userspace-rcu.git
  $ git checkout v0.10.1 # version in bionic

  Build the code:

  $ ./bootstrap
  $ ./configure
  $ make

  Go into the tests/benchmark directory

  $ cd tests/benchmark

  From there, you can run benchmarks for the four main usages of
  liburcu: urcu, urcu-bp, urcu-signal and urcu-mb.

  On a 8 core machine, 6 threads for readers and 2 threads for writers,
  with a 10 second runtime, execute:

  $ ./test_urcu 6 2 10
  $ ./test_urcu_bp 6 2 10
  $ ./test_urcu_signal 6 2 10
  $ ./test_urcu_mb 6 2 10

  Results:

  ./test_urcu 6 2 10
  0.10.1-1: 17612527667 reads, 268 writes, 17612527935 ops
  0.10.1-1ubuntu1: 14988437247 reads, 810069 writes, 14989247316 ops

  $ ./test_urcu_bp 6 2 10
  0.10.1-1: 1177891079 reads, 1699523 writes, 1179590602 ops
  0.10.1-1ubuntu1: 13230354737 reads, 575314 writes, 13230930051 ops

  $ ./test_urcu_signal 6 2 10
  0.10.1-1: 20128392417 reads, 6859 writes, 20128399276 ops
  0.10.1-1ubuntu1: 20501430707 reads, 6890 writes, 20501437597 ops

  $ ./test_urcu_mb 6 2 10
  0.10.1-1: 627996563 reads, 5409563 writes, 633406126 ops
  0.10.1-1ubuntu1: 653194752 reads, 4590020 writes, 657784772 ops

  The SRU only changes behaviour for urcu and urcu-bp, since they are
  the only "flavours" of liburcu which the patches change. From a pure
  ops standpoint:

  $ ./test_urcu 6 2 10
  17612527935 ops
  14989247316 ops

  $ ./test_urcu_bp 6 2 10
  1179590602 ops
  13230930051 ops

  We see that this particular benchmark workload, test_urcu sees extra
  performance overhead with MEMBARRIER_CMD_PRIVATE_EXPEDITED, which is
  explained by the extra impact that it has on the slowpath, and the
  extra amount of writes it did during my benchmark.

  The real winner in this benchmark workload is test_urcu_bp, which sees
  a 10x performance increase with MEMBARRIER_CMD_PRIVATE_EXPEDITED. Some
  of this may be down to the 3x less writes it did during my benchmark.

  Again, these benchmarks are indicative only are very "random".
  Performance is really dependant on the application which links against
  liburcu and its workload.

  [Regression Potential]

  This SRU changes the behaviour of the following libraries which
  applications link against: -lurcu and -lurcu-bp. Behaviour is not
  changed in the rest: -lurcu-qsbr, -lucru-signal and -lucru-mb.

  On Bionic, liburcu will call the membarrier syscall in urcu and urcu-
  bp. This does not change. What is changing is the semantics of that
  syscall, from MEMBARRIER_CMD_SHARED to
  MEMBARRIER_CMD_PRIVATE_EXPEDITED. The changed code is all run in
  kernel space and resides in the kernel. These commits simply change
  the parameters which are supplied to the membarrier syscall from
  liburcu.

  I have run the testsuite that comes with the Bionic source code, and
  "make regtest", "make short_bench" and "make long_bench" pass. You
  want to run these on a cloud instance somewhere since they take
  multiple hours.

  If a regression were to occur, applications linked against -lurcu and
  -lurcu-bp would be affected. The homepage: https://liburcu.org/ offers
  a list of the major applications that use liburcu: Knot DNS, Netsniff-
  ng, Sheepdog, GlusterFS, gdnsd and LTTng.

  [Scope]

  The two commits which are being SRU'd are:

  commit c0bb9f693f926595a7cb8b4ce712cef08d9f5d49
  Author: Mathieu Desnoyers <mathieu.desnoyers@xxxxxxxxxxxx>
  Date: Thu Dec 21 13:42:23 2017 -0500
  Subject: liburcu: Use membarrier private expedited when available
  Link: https://github.com/urcu/userspace-rcu/commit/c0bb9f693f926595a7cb8b4ce712cef08d9f5d49

  commit 3745305bf09e7825e75ee5b5490347ee67c6efdd
  Author: Mathieu Desnoyers <mathieu.desnoyers@xxxxxxxxxxxx>
  Date: Fri Dec 22 10:57:59 2017 -0500
  Subject: liburcu-bp: Use membarrier private expedited when available
  Link: https://github.com/urcu/userspace-rcu/commit/3745305bf09e7825e75ee5b5490347ee67c6efdd

  Both cherry pick directly onto 0.10.1 in Bionic, and are originally
  from 0.11.0, meaning that Eoan, Focal and Groovy already have the
  patch.

  [Other]

  If you are interested in how the membarrier syscall works, you can
  read their commits in the Linux kernel:

  commit 5b25b13ab08f616efd566347d809b4ece54570d1
  Author: Mathieu Desnoyers <mathieu.desnoyers@xxxxxxxxxxxx>
  Date:   Fri Sep 11 13:07:39 2015 -0700
  Subject: sys_membarrier(): system-wide memory barrier (generic, x86)
  Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5b25b13ab08f616efd566347d809b4ece54570d1

  commit 22e4ebb975822833b083533035233d128b30e98f
  Author: Mathieu Desnoyers <mathieu.desnoyers@xxxxxxxxxxxx>
  Date:   Fri Jul 28 16:40:40 2017 -0400
  Subject: membarrier: Provide expedited private command
  Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=22e4ebb975822833b083533035233d128b30e98f

  Additionally, blog posts from LTTng:
  https://lttng.org/blog/2018/01/15/membarrier-system-call-performance-and-userspace-rcu/

  And Phoronix:
  https://www.phoronix.com/scan.php?page=news_item&px=URCU-Membarrier-Performance

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/liburcu/+bug/1876230/+subscriptions