kernel-packages team mailing list archive

[Chris J Arges <1328088@xxxxxxxxxxxxxxxxxx>]

** Also affects: linux (Ubuntu)
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1328088

Title:
  Kernel network namespace performance regression during rcu development
  on kernels above 3.8

Status in The Linux Kernel:
  In Progress
Status in “linux” package in Ubuntu:
  New

Bug description:
  Please, follow this in:
  http://people.canonical.com/~inaddy/lp1328088/. Same description on
  daily-basis updated text.

  --

  It was brought to my attention that "fake router creation" scalability
  was affected during kernel development.

  The following script was used for all the tests and charts generation:

  http://people.canonical.com/~inaddy/lp1328088/make_fake_routers.sh
  http://people.canonical.com/~inaddy/lp1328088/parse.py

  I measured how many "fake routers" (above script) could be added per
  second from 0 to 4000 created routers mark. Using this script and a
  git bisect on kernel tree I was led to one specific commit causing
  regression: #911af505 "rcu: Provide compile-time control for no-CBs
  CPUs".

  It appeared that rcu, rcu callbacks and no-cb cpus were causing the
  issue so every commit that changed any of this files:
  "kernel/rcutree.c kernel/rcutree.h kernel/rcutree_plugin.h
  include/trace/events/rcu.h include/linux/rcupdate.h" was tested. The
  idea was to check performance regression during rcu development. In
  the worst case I would have data for performance regression during
  kernel development (since we have rcu commits from 3.8 to 3.14).

  All text below this refer to 2 groups of charts, generated during the
  study:

  1) Kernel git tags from 3.8 to 3.14.
  http://people.canonical.com/~inaddy/lp1328088/charts/250-tag.html

  2) Kernel git commits for rcu development (111 commits).
  http://people.canonical.com/~inaddy/lp1328088/charts/250.html

  Since there was difference in results depending on how many cpus or
  how the no-cb cpus were configured, 3 kernel config options were used
  on every measure:

  - CONFIG_RCU_NOCB_CPU (disabled: nocbno)
  - CONFIG_RCU_NOCB_CPU_ALL (enabled: nocball)
  - CONFIG_RCU_NOCB_CPU_NONE (enabled: nocbnone)

  After charts generation and study it was clear that NOCB_CPU_ALL (4
  cpus) affected the "fake routers" creation process performance and
  this regression continues up to upstream version. It was also clear
  that, after this commit, there is no scalability executing this test
  with more than 1 cpu.

  Comparing standing out points (see charts):

  #81e5949 - good
  #911af50 - bad
  #6faf728 - not good enough

  I was able to see that from the script above the following lines were
  affected:

  1) ip netns add -> huge performance regression
  2) ip netns exec -> some performance regression

  #
  # Assumption
  #

  rcu callbacks being offloaded to other cpus caused regression in
  unshare(CLONE_NEWNET) code.

  # Specific kernel entry being investigated:

  unshare(CLONE_NEWNET)

To manage notifications about this bug go to:
https://bugs.launchpad.net/linux/+bug/1328088/+subscriptions