← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #30670

[Bug 1438253] [NEW] nova libvirt pinning won't work across numa nodes

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Using a CPU policy of dedicated ('hw:cpu_policy=dedicated') results in
vCPUs being pinned to pCPUs, per the original blueprint:

    http://specs.openstack.org/openstack/nova-
specs/specs/kilo/implemented/virt-driver-cpu-pinning.html

When combined with the 'vcpu_pin_set' nova configuration, it should be
possible to get very good control over which pCPUs are used by
instances. This works as expected when creating a multi-vCPU instance
pinned to pCPUs on a single NUMA node. However, if these pCPUs are
spread across multiple NUMA nodes then it fails. This behavior is not
documented in the original blueprint and therefore should not function
like so.

---

# Testing Configuration

Testing was conducted on a single-node, Fedora 21-based
(3.17.8-300.fc21.x86_64) OpenStack instance (built with devstack). Two
flavors were used:

    openstack flavor create --ram 4096 --disk 20 --vcpus 10 demo.no-
pinning

    openstack flavor create --ram 4096 --disk 20 --vcpus 10 demo.pinning
    nova flavor-key demo.pinning set hw:cpu_policy=dedicated hw:cpu_threads_policy=separate

# Results

Passing (same NUMA socket, contiguous)
======================================

        vcpu_pin_set = 10-19

Pinned
------

	$ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.pinning test1 --wait
	$ ps aux | grep qemu | awk '{print }' | head -1 | xargs taskset -pc
	pid 68150's current affinity list: 10-19
	$ sudo -E virsh vcpucount 13
	maximum      config        10
	maximum      live          10
	current      config        10
	current      live          10

Unpinned
--------

	$ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.no-pinning test1 --wait
	$ ps aux | grep qemu | awk '{print }' | head -1 | xargs taskset -pc
	pid 70249's current affinity list: 10-19

Passing (same NUMA socket, non-contiguous)
==========================================

        vcpu_pin_set = 5-9,25-29

Pinned
------

	$ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.pinning test1 --wait
	$ ps aux | grep qemu | awk '{print }' | head -1 | xargs taskset -pc
	pid 91186's current affinity list: 5-9,25-29

Unpinned
--------

	$ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.no-pinning test1 --wait
	$ ps aux | grep qemu | awk '{print }' | head -1 | xargs taskset -pc
	pid 92212's current affinity list: 5-9,25-29

Passing (multiple NUMA sockets, non-contiguous)
===============================================

        vcpu_pin_set = 5-10,25-29

Pinned
------

	$ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.pinning test1 --wait
	$ ps aux | grep qemu | awk '{print }' | head -1 | xargs taskset -pc
	pid 93884's current affinity list: 5-9,25-29

Unpinned
--------

	$ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.no-pinning test1 --wait
	$ ps aux | grep qemu | awk '{print $2}' | head -1 | xargs taskset -pc
	pid 94468's current affinity list: 5-9,25-29

Additional combinations
-----------------------

        vcpu_pin_set = 5-14,25-29


Failing (different NUMA socket, contiguous)
===========================================

        vcpu_pin_set = 9-18

Pinned
------

	$ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.pinning test1 --wait
	ERROR: openstackclient.compute.v2.server.CreateServer Error creating server: test1

Unpinned
--------

	$ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.no-pinning test1 --wait
	$ ps aux | grep qemu | awk '{print }' | head -1 | xargs taskset -pc
	pid 71194's current affinity list: 9-18

Failing (different NUMA socket, non-contiguous)
===============================================

        vcpu_pin_set = 5-9,15-19

Pinned
------

	$ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.no-pinning test1 --wait
	ERROR: openstackclient.compute.v2.server.CreateServer Error creating server: test1
	$ openstack server show test1 | grep fault
	| fault                                | {u'message': u'No valid host was found. There are not enough hosts available.', u'code': 500,

Unpinned
--------

	$ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.no-pinning test1 --wait
	$ ps aux | grep qemu | awk '{print }' | head -1 | xargs taskset -pc
	pid 88703's current affinity list: 5-9,15-19

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: libvirt

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1438253

Title:
  nova libvirt pinning won't work across numa nodes

Status in OpenStack Compute (Nova):
  New

Bug description:
  Using a CPU policy of dedicated ('hw:cpu_policy=dedicated') results in
  vCPUs being pinned to pCPUs, per the original blueprint:

      http://specs.openstack.org/openstack/nova-
  specs/specs/kilo/implemented/virt-driver-cpu-pinning.html

  When combined with the 'vcpu_pin_set' nova configuration, it should be
  possible to get very good control over which pCPUs are used by
  instances. This works as expected when creating a multi-vCPU instance
  pinned to pCPUs on a single NUMA node. However, if these pCPUs are
  spread across multiple NUMA nodes then it fails. This behavior is not
  documented in the original blueprint and therefore should not function
  like so.

  ---

  # Testing Configuration

  Testing was conducted on a single-node, Fedora 21-based
  (3.17.8-300.fc21.x86_64) OpenStack instance (built with devstack). Two
  flavors were used:

      openstack flavor create --ram 4096 --disk 20 --vcpus 10 demo.no-
  pinning

      openstack flavor create --ram 4096 --disk 20 --vcpus 10 demo.pinning
      nova flavor-key demo.pinning set hw:cpu_policy=dedicated hw:cpu_threads_policy=separate

  # Results

  Passing (same NUMA socket, contiguous)
  ======================================

          vcpu_pin_set = 10-19

  Pinned
  ------

  	$ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.pinning test1 --wait
  	$ ps aux | grep qemu | awk '{print }' | head -1 | xargs taskset -pc
  	pid 68150's current affinity list: 10-19
  	$ sudo -E virsh vcpucount 13
  	maximum      config        10
  	maximum      live          10
  	current      config        10
  	current      live          10

  Unpinned
  --------

  	$ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.no-pinning test1 --wait
  	$ ps aux | grep qemu | awk '{print }' | head -1 | xargs taskset -pc
  	pid 70249's current affinity list: 10-19

  Passing (same NUMA socket, non-contiguous)
  ==========================================

          vcpu_pin_set = 5-9,25-29

  Pinned
  ------

  	$ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.pinning test1 --wait
  	$ ps aux | grep qemu | awk '{print }' | head -1 | xargs taskset -pc
  	pid 91186's current affinity list: 5-9,25-29

  Unpinned
  --------

  	$ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.no-pinning test1 --wait
  	$ ps aux | grep qemu | awk '{print }' | head -1 | xargs taskset -pc
  	pid 92212's current affinity list: 5-9,25-29

  Passing (multiple NUMA sockets, non-contiguous)
  ===============================================

          vcpu_pin_set = 5-10,25-29

  Pinned
  ------

  	$ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.pinning test1 --wait
  	$ ps aux | grep qemu | awk '{print }' | head -1 | xargs taskset -pc
  	pid 93884's current affinity list: 5-9,25-29

  Unpinned
  --------

  	$ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.no-pinning test1 --wait
  	$ ps aux | grep qemu | awk '{print $2}' | head -1 | xargs taskset -pc
  	pid 94468's current affinity list: 5-9,25-29

  Additional combinations
  -----------------------

          vcpu_pin_set = 5-14,25-29

  
  Failing (different NUMA socket, contiguous)
  ===========================================

          vcpu_pin_set = 9-18

  Pinned
  ------

  	$ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.pinning test1 --wait
  	ERROR: openstackclient.compute.v2.server.CreateServer Error creating server: test1

  Unpinned
  --------

  	$ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.no-pinning test1 --wait
  	$ ps aux | grep qemu | awk '{print }' | head -1 | xargs taskset -pc
  	pid 71194's current affinity list: 9-18

  Failing (different NUMA socket, non-contiguous)
  ===============================================

          vcpu_pin_set = 5-9,15-19

  Pinned
  ------

  	$ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.no-pinning test1 --wait
  	ERROR: openstackclient.compute.v2.server.CreateServer Error creating server: test1
  	$ openstack server show test1 | grep fault
  	| fault                                | {u'message': u'No valid host was found. There are not enough hosts available.', u'code': 500,

  Unpinned
  --------

  	$ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.no-pinning test1 --wait
  	$ ps aux | grep qemu | awk '{print }' | head -1 | xargs taskset -pc
  	pid 88703's current affinity list: 5-9,15-19

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1438253/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next