yahoo-eng-team team mailing list archive

[Stephen Finucane <stephen.finucane@xxxxxxxxx>]

Public bug reported:

Using a CPU policy of dedicated ('hw:cpu_policy=dedicated') results in
vCPUs being pinned to pCPUs, per the original blueprint:

    http://specs.openstack.org/openstack/nova-
specs/specs/kilo/implemented/virt-driver-cpu-pinning.html

When scheduling instance with this extra spec there appears to be an
implicit use of the 'prefer' threads policy, i.e. where possible vCPUs
are pinned to thread siblings first. This is "implicit" because the
threads policy aspect of this spec has not yet been implemented.

However, this implicit 'prefer' policy breaks when a VM with an odd
number of vCPUs is booted. This has been seen on a Hyper-Threading-
enabled host where "sibling sets" are two long, but it would presumably
happen on any host where the number of siblings (or any number between
this value and one) is not an factor of the number of vCPUs (i.e. vCPUs
% n != 0, for siblings <= n > 0).

It is reasonable to assume that a three vCPU VM, for example, should try
best effort and use siblings for at the first two vCPUs of the VM
(assuming you're on a host system with HyperThreading and sibling sets
are of length two). This would give us a true best effort
implementation.

---

# Testing Configuration

Testing was conducted on a single-node, Fedora 21-based
(3.17.8-300.fc21.x86_64) OpenStack instance (built with devstack). The
system is a dual-socket, 10 core, HT-enabled system (2 sockets * 10
cores * 2 threads = 40 "pCPUs". 0-9,20-29 = node0, 10-19,30-39 = node1).
Two flavors were used:

    openstack flavor create --ram 4096 --disk 20 --vcpus 3 demo.odd
    nova flavor-key demo.odd set hw:cpu_policy=dedicated

    openstack flavor create --ram 4096 --disk 20 --vcpus 4 demo.even
    nova flavor-key demo.even set hw:cpu_policy=dedicated

# Results

Correct case ("even" number of vCPUs)
=====================================

The output from 'virsh dumpxml [ID]' for the four vCPU VM is given
below. Similar results can be seen for varying "even" numbers of vCPUs
(2, 4, 10 tested):

    <cputune>
        <shares>4096</shares>
        <vcpupin vcpu='0' cpuset='3'/>
        <vcpupin vcpu='1' cpuset='23'/>
        <vcpupin vcpu='2' cpuset='26'/>
        <vcpupin vcpu='3' cpuset='6'/>
        <emulatorpin cpuset='3,6,23,26'/>
    </cputune>

Incorrect case ("odd" number of vCPUs)
======================================

The output from 'virsh dumpxml [ID]' for the three vCPU VM is given
below. Similar results can be seen for varying "odd" numbers of vCPUs
(3, 5 tested):

    <cputune>
        <shares>3072</shares>
        <vcpupin vcpu='0' cpuset='1'/>
        <vcpupin vcpu='1' cpuset='0'/>
        <vcpupin vcpu='2' cpuset='25'/>
        <emulatorpin cpuset='0-1,25'/>
    </cputune>

This isn't correct. We would expect something closer to this:

    <cputune>
        <shares>3072</shares>
        <vcpupin vcpu='0' cpuset='0'/>
        <vcpupin vcpu='1' cpuset='20'/>
        <vcpupin vcpu='2' cpuset='1'/>
        <emulatorpin cpuset='0-1,20'/>
    </cputune>

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: libvirt

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1467927

Title:
  Odd number of vCPUs breaks 'prefer' threads policy

Status in OpenStack Compute (Nova):
  New

Bug description:
  Using a CPU policy of dedicated ('hw:cpu_policy=dedicated') results in
  vCPUs being pinned to pCPUs, per the original blueprint:

      http://specs.openstack.org/openstack/nova-
  specs/specs/kilo/implemented/virt-driver-cpu-pinning.html

  When scheduling instance with this extra spec there appears to be an
  implicit use of the 'prefer' threads policy, i.e. where possible vCPUs
  are pinned to thread siblings first. This is "implicit" because the
  threads policy aspect of this spec has not yet been implemented.

  However, this implicit 'prefer' policy breaks when a VM with an odd
  number of vCPUs is booted. This has been seen on a Hyper-Threading-
  enabled host where "sibling sets" are two long, but it would
  presumably happen on any host where the number of siblings (or any
  number between this value and one) is not an factor of the number of
  vCPUs (i.e. vCPUs % n != 0, for siblings <= n > 0).

  It is reasonable to assume that a three vCPU VM, for example, should
  try best effort and use siblings for at the first two vCPUs of the VM
  (assuming you're on a host system with HyperThreading and sibling sets
  are of length two). This would give us a true best effort
  implementation.

  ---

  # Testing Configuration

  Testing was conducted on a single-node, Fedora 21-based
  (3.17.8-300.fc21.x86_64) OpenStack instance (built with devstack). The
  system is a dual-socket, 10 core, HT-enabled system (2 sockets * 10
  cores * 2 threads = 40 "pCPUs". 0-9,20-29 = node0, 10-19,30-39 =
  node1). Two flavors were used:

      openstack flavor create --ram 4096 --disk 20 --vcpus 3 demo.odd
      nova flavor-key demo.odd set hw:cpu_policy=dedicated

      openstack flavor create --ram 4096 --disk 20 --vcpus 4 demo.even
      nova flavor-key demo.even set hw:cpu_policy=dedicated

  # Results

  Correct case ("even" number of vCPUs)
  =====================================

  The output from 'virsh dumpxml [ID]' for the four vCPU VM is given
  below. Similar results can be seen for varying "even" numbers of vCPUs
  (2, 4, 10 tested):

      <cputune>
          <shares>4096</shares>
          <vcpupin vcpu='0' cpuset='3'/>
          <vcpupin vcpu='1' cpuset='23'/>
          <vcpupin vcpu='2' cpuset='26'/>
          <vcpupin vcpu='3' cpuset='6'/>
          <emulatorpin cpuset='3,6,23,26'/>
      </cputune>

  Incorrect case ("odd" number of vCPUs)
  ======================================

  The output from 'virsh dumpxml [ID]' for the three vCPU VM is given
  below. Similar results can be seen for varying "odd" numbers of vCPUs
  (3, 5 tested):

      <cputune>
          <shares>3072</shares>
          <vcpupin vcpu='0' cpuset='1'/>
          <vcpupin vcpu='1' cpuset='0'/>
          <vcpupin vcpu='2' cpuset='25'/>
          <emulatorpin cpuset='0-1,25'/>
      </cputune>

  This isn't correct. We would expect something closer to this:

      <cputune>
          <shares>3072</shares>
          <vcpupin vcpu='0' cpuset='0'/>
          <vcpupin vcpu='1' cpuset='20'/>
          <vcpupin vcpu='2' cpuset='1'/>
          <emulatorpin cpuset='0-1,20'/>
      </cputune>

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1467927/+subscriptions