group.of.nepali.translators team mailing list archive

[Matthew Ruffell <1882478@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

BugLink: https://bugs.launchpad.net/bugs/1882478

[Impact]

Intel Skylake server processors and onward have a different Last Level
Cache (LLC) topology than earlier processors, and such processors have a
new feature called Sub-NUMA-Clustering (SNC) which is similar to the
existing Cluster-On-Die (CoD) feature earlier server processors has.

Sub-NUMA-Clustering divides the system into two "slices", each of which
are allocated half the CPU cores, half the Last Level Cache and one
memory controller. Each slice is enumerated as a NUMA node.

The difference between Sub-NUMA-Clustering and Cluster-On-Die is how the
Last Level Cache is exposed to each NUMA node. CoD had the same cache
line present in each half of the LLC. In SNC, each cache line is only
present in its respective slice. Because of this, the semantics around
accessing LLC changes, with a process accessing NUMA-local memory only
seeing half the LLC capacity.

On systems with Sub-NUMA-Clustering enabled, on the Xenial 4.4 and
Bionic 4.15 kernels we see the following oops during NUMA node
enumeration:

.... node #0, CPUs: #1 #2 #3 #4 #5 #6
.... node #1, CPUs: #7
sched: CPU #7's llc-sibling CPU #0 is not on the same node! [node: 1 != 0]. Ignoring dependency.
WARNING: CPU: 7 PID: 0 at /build/linux-hwe-F5opqf/linux-hwe-4.15.0/arch/x86/kernel/smpboot.c:375 topology_sane.isra.4+0x6c/0x70
Modules linked in:
CPU: 7 PID: 0 Comm: swapper/7 Not tainted 4.15.0-47-generic #50~16.04.1-Ubuntu
Hardware name: HPE ProLiant DL360 Gen10/ProLiant DL360 Gen10, BIOS U32 10/02/2018
RIP: 0010:topology_sane.isra.4+0x6c/0x70
Call Trace:
set_cpu_sibling_map+0x153/0x540
start_secondary+0xb2/0x200
secondary_startup_64+0xa5/0xb0
#8 #9 #10 #11 #12 #13
.... node #0, CPUs: #14 #15 #16 #17 #18 #19 #20
.... node #1, CPUs: #21 #22 #23 #24 #25 #26 #27
smp: Brought up 2 nodes, 28 CPUs

This was with a Intel Xeon Gold 5120 CPU on a HP DL360 Gen10.

The oops happens because topology_sane() checks to see if the Last Level
Cache line matches across different CPUs, which it no longer does.

[Fix]

The fix comes in the form of the following upstream commit, which landed
in Linux 4.17:

commit 1340ccfa9a9afefdbab90d7935d4ed19817e37c2
Author: Alison Schofield <alison.schofield@xxxxxxxxx>
Date: Fri Apr 6 17:21:30 2018 -0700
Subject: x86,sched: Allow topologies where NUMA nodes share an LLC
Link: https://github.com/torvalds/linux/commit/1340ccfa9a9afefdbab90d7935d4ed19817e37c2

The commit adds a check for this particular family of Intel processors,
and if the CPU family matches, it simply skips the check to
topology_sane().

The commit needs minor backports to Xenial 4.4 and Bionic 4.15, with the
only remarks being re-arranging #includes and small context fixups.

[Testcase]

Unfortunately, this is hardware specific. To test this, you need a Intel
Skylake server processor which supports Sub-NUMA-Clustering.

We have a customer with a Intel Xeon Gold 5120 CPU on a HP DL360 Gen10
that has successfully tested the below test kernels, with good results.

Xenial 4.4 ppa:
https://launchpad.net/~mruffell/+archive/ubuntu/sf280048-test-ga

Xenial 4.15 HWE ppa:
https://launchpad.net/~mruffell/+archive/ubuntu/sf280048-test-hwe

Running the test kernel, the oops does not reproduce:

smp: Bringing up secondary CPUs ...
x86: Booting SMP configuration:
.... node #0, CPUs: #1
NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
#2 #3 #4 #5 #6
.... node #1, CPUs: #7 #8 #9 #10 #11 #12 #13
.... node #0, CPUs: #14 #15 #16 #17 #18 #19 #20
.... node #1, CPUs: #21 #22 #23 #24 #25 #26 #27
smp: Brought up 2 nodes, 28 CPUs
smpboot: Max logical packages: 1
smpboot: Total of 28 processors activated

[Regression Potential]

The commit modifies a small section of smpboot code, which every machine
will execute on boot. The majority of the commit breaks up a large if
statement into smaller blocks than it was previously, and adds an extra
if statement to check for a specific processor family.

If a regression were to occur, some machines would or would not make
their calls to topology_sane(), which in the worst case, would result in
a oops message and slightly degraded performance. The system would still
function normally.

The commit has been present since 4.17-rc2 and is present in Eoan and
Focal. There are no fixup commits, and no additional processor families
have been added since.

Because of the small re-arrangement in logic, and the addition of a
processor family check, these changes a fairly minor, and I don't think
it will cause any regressions.

** Affects: linux (Ubuntu)
     Importance: Undecided
         Status: Fix Released

** Affects: linux (Ubuntu Xenial)
     Importance: Medium
     Assignee: Matthew Ruffell (mruffell)
         Status: In Progress

** Affects: linux (Ubuntu Bionic)
     Importance: Medium
     Assignee: Matthew Ruffell (mruffell)
         Status: In Progress


** Tags: sts

** Also affects: linux (Ubuntu Xenial)
   Importance: Undecided
       Status: New

** Also affects: linux (Ubuntu Bionic)
   Importance: Undecided
       Status: New

** Changed in: linux (Ubuntu)
       Status: New => Fix Released

** Changed in: linux (Ubuntu Xenial)
       Status: New => In Progress

** Changed in: linux (Ubuntu Bionic)
       Status: New => In Progress

** Changed in: linux (Ubuntu Xenial)
   Importance: Undecided => Medium

** Changed in: linux (Ubuntu Bionic)
   Importance: Undecided => Medium

** Changed in: linux (Ubuntu Xenial)
     Assignee: (unassigned) => Matthew Ruffell (mruffell)

** Changed in: linux (Ubuntu Bionic)
     Assignee: (unassigned) => Matthew Ruffell (mruffell)

** Tags added: sts

** Description changed:

- BugLink: https://bugs.launchpad.net/bugs/
+ BugLink: https://bugs.launchpad.net/bugs/1882478
  
  [Impact]
  
  Intel Skylake server processors and onward have a different Last Level
  Cache (LLC) topology than earlier processors, and such processors have a
  new feature called Sub-NUMA-Clustering (SNC) which is similar to the
  existing Cluster-On-Die (CoD) feature earlier server processors has.
  
  Sub-NUMA-Clustering divides the system into two "slices", each of which
  are allocated half the CPU cores, half the Last Level Cache and one
  memory controller. Each slice is enumerated as a NUMA node.
  
  The difference between Sub-NUMA-Clustering and Cluster-On-Die is how the
  Last Level Cache is exposed to each NUMA node. CoD had the same cache
  line present in each half of the LLC. In SNC, each cache line is only
  present in its respective slice. Because of this, the semantics around
  accessing LLC changes, with a process accessing NUMA-local memory only
  seeing half the LLC capacity.
  
  On systems with Sub-NUMA-Clustering enabled, on the Xenial 4.4 and
  Bionic 4.15 kernels we see the following oops during NUMA node
  enumeration:
  
  .... node #0, CPUs: #1 #2 #3 #4 #5 #6
  .... node #1, CPUs: #7
  sched: CPU #7's llc-sibling CPU #0 is not on the same node! [node: 1 != 0]. Ignoring dependency.
  WARNING: CPU: 7 PID: 0 at /build/linux-hwe-F5opqf/linux-hwe-4.15.0/arch/x86/kernel/smpboot.c:375 topology_sane.isra.4+0x6c/0x70
  Modules linked in:
  CPU: 7 PID: 0 Comm: swapper/7 Not tainted 4.15.0-47-generic #50~16.04.1-Ubuntu
  Hardware name: HPE ProLiant DL360 Gen10/ProLiant DL360 Gen10, BIOS U32 10/02/2018
  RIP: 0010:topology_sane.isra.4+0x6c/0x70
  Call Trace:
  set_cpu_sibling_map+0x153/0x540
  start_secondary+0xb2/0x200
  secondary_startup_64+0xa5/0xb0
  #8 #9 #10 #11 #12 #13
  .... node #0, CPUs: #14 #15 #16 #17 #18 #19 #20
  .... node #1, CPUs: #21 #22 #23 #24 #25 #26 #27
- smp: Brought up 2 nodes, 28 CPUs 
+ smp: Brought up 2 nodes, 28 CPUs
  
  This was with a Intel Xeon Gold 5120 CPU on a HP DL360 Gen10.
  
  The oops happens because topology_sane() checks to see if the Last Level
  Cache line matches across different CPUs, which it no longer does.
  
  [Fix]
  
  The fix comes in the form of the following upstream commit, which landed in
  Linux 4.17:
  
  commit 1340ccfa9a9afefdbab90d7935d4ed19817e37c2
  Author: Alison Schofield <alison.schofield@xxxxxxxxx>
  Date: Fri Apr 6 17:21:30 2018 -0700
  Subject: x86,sched: Allow topologies where NUMA nodes share an LLC
- Link: https://github.com/torvalds/linux/commit/1340ccfa9a9afefdbab90d7935d4ed19817e37c2 
+ Link: https://github.com/torvalds/linux/commit/1340ccfa9a9afefdbab90d7935d4ed19817e37c2
  
  The commit adds a check for this particular family of Intel processors,
  and if the CPU family matches, it simply skips the check to
  topology_sane().
  
  The commit needs minor backports to Xenial 4.4 and Bionic 4.15, with the
  only remarks being re-arranging #includes and small context fixups.
  
  [Testcase]
  
  Unfortunately, this is hardware specific. To test this, you need a Intel
  Skylake server processor which supports Sub-NUMA-Clustering.
  
  We have a customer with a Intel Xeon Gold 5120 CPU on a HP DL360 Gen10
  that has successfully tested the below test kernels, with good results.
  
  Xenial 4.4 ppa:
  https://launchpad.net/~mruffell/+archive/ubuntu/sf280048-test-ga
  
  Xenial 4.15 HWE ppa:
  https://launchpad.net/~mruffell/+archive/ubuntu/sf280048-test-hwe
  
  Running the test kernel, the oops does not reproduce:
  
  smp: Bringing up secondary CPUs ...
  x86: Booting SMP configuration:
  .... node #0, CPUs: #1
  NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
  #2 #3 #4 #5 #6
  .... node #1, CPUs: #7 #8 #9 #10 #11 #12 #13
  .... node #0, CPUs: #14 #15 #16 #17 #18 #19 #20
  .... node #1, CPUs: #21 #22 #23 #24 #25 #26 #27
  smp: Brought up 2 nodes, 28 CPUs
  smpboot: Max logical packages: 1
  smpboot: Total of 28 processors activated
  
  [Regression Potential]
  
  The commit modifies a small section of smpboot code, which every machine
  will execute on boot. The majority of the commit breaks up a large if
  statement into smaller blocks than it was previously, and adds an extra
  if statement to check for a specific processor family.
  
  If a regression were to occur, some machines would or would not make
  their calls to topology_sane(), which in the worst case, would result in
  a oops message and slightly degraded performance. The system would still
  function normally.
  
  The commit has been present since 4.17-rc2 and is present in Eoan and
  Focal. There are no fixup commits, and no additional processor families
  have been added since.
  
  Because of the small re-arrangement in logic, and the addition of a
  processor family check, these changes a fairly minor, and I don't think
  it will cause any regressions.

** Description changed:

  BugLink: https://bugs.launchpad.net/bugs/1882478
  
  [Impact]
  
  Intel Skylake server processors and onward have a different Last Level
  Cache (LLC) topology than earlier processors, and such processors have a
  new feature called Sub-NUMA-Clustering (SNC) which is similar to the
  existing Cluster-On-Die (CoD) feature earlier server processors has.
  
  Sub-NUMA-Clustering divides the system into two "slices", each of which
  are allocated half the CPU cores, half the Last Level Cache and one
  memory controller. Each slice is enumerated as a NUMA node.
  
  The difference between Sub-NUMA-Clustering and Cluster-On-Die is how the
  Last Level Cache is exposed to each NUMA node. CoD had the same cache
  line present in each half of the LLC. In SNC, each cache line is only
  present in its respective slice. Because of this, the semantics around
  accessing LLC changes, with a process accessing NUMA-local memory only
  seeing half the LLC capacity.
  
  On systems with Sub-NUMA-Clustering enabled, on the Xenial 4.4 and
  Bionic 4.15 kernels we see the following oops during NUMA node
  enumeration:
  
  .... node #0, CPUs: #1 #2 #3 #4 #5 #6
  .... node #1, CPUs: #7
  sched: CPU #7's llc-sibling CPU #0 is not on the same node! [node: 1 != 0]. Ignoring dependency.
  WARNING: CPU: 7 PID: 0 at /build/linux-hwe-F5opqf/linux-hwe-4.15.0/arch/x86/kernel/smpboot.c:375 topology_sane.isra.4+0x6c/0x70
  Modules linked in:
  CPU: 7 PID: 0 Comm: swapper/7 Not tainted 4.15.0-47-generic #50~16.04.1-Ubuntu
  Hardware name: HPE ProLiant DL360 Gen10/ProLiant DL360 Gen10, BIOS U32 10/02/2018
  RIP: 0010:topology_sane.isra.4+0x6c/0x70
  Call Trace:
  set_cpu_sibling_map+0x153/0x540
  start_secondary+0xb2/0x200
  secondary_startup_64+0xa5/0xb0
  #8 #9 #10 #11 #12 #13
  .... node #0, CPUs: #14 #15 #16 #17 #18 #19 #20
  .... node #1, CPUs: #21 #22 #23 #24 #25 #26 #27
  smp: Brought up 2 nodes, 28 CPUs
  
  This was with a Intel Xeon Gold 5120 CPU on a HP DL360 Gen10.
  
  The oops happens because topology_sane() checks to see if the Last Level
  Cache line matches across different CPUs, which it no longer does.
  
  [Fix]
  
- The fix comes in the form of the following upstream commit, which landed in
- Linux 4.17:
+ The fix comes in the form of the following upstream commit, which landed
+ in Linux 4.17:
  
  commit 1340ccfa9a9afefdbab90d7935d4ed19817e37c2
  Author: Alison Schofield <alison.schofield@xxxxxxxxx>
  Date: Fri Apr 6 17:21:30 2018 -0700
  Subject: x86,sched: Allow topologies where NUMA nodes share an LLC
  Link: https://github.com/torvalds/linux/commit/1340ccfa9a9afefdbab90d7935d4ed19817e37c2
  
  The commit adds a check for this particular family of Intel processors,
  and if the CPU family matches, it simply skips the check to
  topology_sane().
  
  The commit needs minor backports to Xenial 4.4 and Bionic 4.15, with the
  only remarks being re-arranging #includes and small context fixups.
  
  [Testcase]
  
  Unfortunately, this is hardware specific. To test this, you need a Intel
  Skylake server processor which supports Sub-NUMA-Clustering.
  
  We have a customer with a Intel Xeon Gold 5120 CPU on a HP DL360 Gen10
  that has successfully tested the below test kernels, with good results.
  
  Xenial 4.4 ppa:
  https://launchpad.net/~mruffell/+archive/ubuntu/sf280048-test-ga
  
  Xenial 4.15 HWE ppa:
  https://launchpad.net/~mruffell/+archive/ubuntu/sf280048-test-hwe
  
  Running the test kernel, the oops does not reproduce:
  
  smp: Bringing up secondary CPUs ...
  x86: Booting SMP configuration:
  .... node #0, CPUs: #1
  NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
  #2 #3 #4 #5 #6
  .... node #1, CPUs: #7 #8 #9 #10 #11 #12 #13
  .... node #0, CPUs: #14 #15 #16 #17 #18 #19 #20
  .... node #1, CPUs: #21 #22 #23 #24 #25 #26 #27
  smp: Brought up 2 nodes, 28 CPUs
  smpboot: Max logical packages: 1
  smpboot: Total of 28 processors activated
  
  [Regression Potential]
  
  The commit modifies a small section of smpboot code, which every machine
  will execute on boot. The majority of the commit breaks up a large if
  statement into smaller blocks than it was previously, and adds an extra
  if statement to check for a specific processor family.
  
  If a regression were to occur, some machines would or would not make
  their calls to topology_sane(), which in the worst case, would result in
  a oops message and slightly degraded performance. The system would still
  function normally.
  
  The commit has been present since 4.17-rc2 and is present in Eoan and
  Focal. There are no fixup commits, and no additional processor families
  have been added since.
  
  Because of the small re-arrangement in logic, and the addition of a
  processor family check, these changes a fairly minor, and I don't think
  it will cause any regressions.

-- 
You received this bug notification because you are a member of नेपाली
भाषा समायोजकहरुको समूह, which is subscribed to Xenial.
Matching subscriptions: Ubuntu 16.04 Bugs
https://bugs.launchpad.net/bugs/1882478

Title:
  smpboot: don't call topology_sane() when Sub-NUMA-Clustering is
  enabled

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Xenial:
  In Progress
Status in linux source package in Bionic:
  In Progress

Bug description:
  BugLink: https://bugs.launchpad.net/bugs/1882478

  [Impact]

  Intel Skylake server processors and onward have a different Last Level
  Cache (LLC) topology than earlier processors, and such processors have
  a new feature called Sub-NUMA-Clustering (SNC) which is similar to the
  existing Cluster-On-Die (CoD) feature earlier server processors has.

  Sub-NUMA-Clustering divides the system into two "slices", each of
  which are allocated half the CPU cores, half the Last Level Cache and
  one memory controller. Each slice is enumerated as a NUMA node.

  The difference between Sub-NUMA-Clustering and Cluster-On-Die is how
  the Last Level Cache is exposed to each NUMA node. CoD had the same
  cache line present in each half of the LLC. In SNC, each cache line is
  only present in its respective slice. Because of this, the semantics
  around accessing LLC changes, with a process accessing NUMA-local
  memory only seeing half the LLC capacity.

  On systems with Sub-NUMA-Clustering enabled, on the Xenial 4.4 and
  Bionic 4.15 kernels we see the following oops during NUMA node
  enumeration:

  .... node #0, CPUs: #1 #2 #3 #4 #5 #6
  .... node #1, CPUs: #7
  sched: CPU #7's llc-sibling CPU #0 is not on the same node! [node: 1 != 0]. Ignoring dependency.
  WARNING: CPU: 7 PID: 0 at /build/linux-hwe-F5opqf/linux-hwe-4.15.0/arch/x86/kernel/smpboot.c:375 topology_sane.isra.4+0x6c/0x70
  Modules linked in:
  CPU: 7 PID: 0 Comm: swapper/7 Not tainted 4.15.0-47-generic #50~16.04.1-Ubuntu
  Hardware name: HPE ProLiant DL360 Gen10/ProLiant DL360 Gen10, BIOS U32 10/02/2018
  RIP: 0010:topology_sane.isra.4+0x6c/0x70
  Call Trace:
  set_cpu_sibling_map+0x153/0x540
  start_secondary+0xb2/0x200
  secondary_startup_64+0xa5/0xb0
  #8 #9 #10 #11 #12 #13
  .... node #0, CPUs: #14 #15 #16 #17 #18 #19 #20
  .... node #1, CPUs: #21 #22 #23 #24 #25 #26 #27
  smp: Brought up 2 nodes, 28 CPUs

  This was with a Intel Xeon Gold 5120 CPU on a HP DL360 Gen10.

  The oops happens because topology_sane() checks to see if the Last
  Level Cache line matches across different CPUs, which it no longer
  does.

  [Fix]

  The fix comes in the form of the following upstream commit, which
  landed in Linux 4.17:

  commit 1340ccfa9a9afefdbab90d7935d4ed19817e37c2
  Author: Alison Schofield <alison.schofield@xxxxxxxxx>
  Date: Fri Apr 6 17:21:30 2018 -0700
  Subject: x86,sched: Allow topologies where NUMA nodes share an LLC
  Link: https://github.com/torvalds/linux/commit/1340ccfa9a9afefdbab90d7935d4ed19817e37c2

  The commit adds a check for this particular family of Intel
  processors, and if the CPU family matches, it simply skips the check
  to topology_sane().

  The commit needs minor backports to Xenial 4.4 and Bionic 4.15, with
  the only remarks being re-arranging #includes and small context
  fixups.

  [Testcase]

  Unfortunately, this is hardware specific. To test this, you need a
  Intel Skylake server processor which supports Sub-NUMA-Clustering.

  We have a customer with a Intel Xeon Gold 5120 CPU on a HP DL360 Gen10
  that has successfully tested the below test kernels, with good
  results.

  Xenial 4.4 ppa:
  https://launchpad.net/~mruffell/+archive/ubuntu/sf280048-test-ga

  Xenial 4.15 HWE ppa:
  https://launchpad.net/~mruffell/+archive/ubuntu/sf280048-test-hwe

  Running the test kernel, the oops does not reproduce:

  smp: Bringing up secondary CPUs ...
  x86: Booting SMP configuration:
  .... node #0, CPUs: #1
  NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
  #2 #3 #4 #5 #6
  .... node #1, CPUs: #7 #8 #9 #10 #11 #12 #13
  .... node #0, CPUs: #14 #15 #16 #17 #18 #19 #20
  .... node #1, CPUs: #21 #22 #23 #24 #25 #26 #27
  smp: Brought up 2 nodes, 28 CPUs
  smpboot: Max logical packages: 1
  smpboot: Total of 28 processors activated

  [Regression Potential]

  The commit modifies a small section of smpboot code, which every
  machine will execute on boot. The majority of the commit breaks up a
  large if statement into smaller blocks than it was previously, and
  adds an extra if statement to check for a specific processor family.

  If a regression were to occur, some machines would or would not make
  their calls to topology_sane(), which in the worst case, would result
  in a oops message and slightly degraded performance. The system would
  still function normally.

  The commit has been present since 4.17-rc2 and is present in Eoan and
  Focal. There are no fixup commits, and no additional processor
  families have been added since.

  Because of the small re-arrangement in logic, and the addition of a
  processor family check, these changes a fairly minor, and I don't
  think it will cause any regressions.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1882478/+subscriptions