yahoo-eng-team team mailing list archive

[sean mooney <sean.k.mooney@xxxxxxxxx>]

This has been discused several times in the last few releases.
the original spec did not mandate that we invent a numa toploygy
of 1 when we request cpu pinning however it did not forbid it.

The current behavior is specific to the libvirt dirver and while it could
legally be changed without violating the api, there has been push back
to doing so as some may have built workflows that assume this behavior.

there are several specs touching on this topic, since this requires broader
agreement then a simple bugfix to address i am closeing as invalid as any
change would require a blueprint and or a spec to address.

** Changed in: nova
       Status: Confirmed => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1438253

Title:
  nova libvirt pinning won't work across numa nodes

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Using a CPU policy of dedicated ('hw:cpu_policy=dedicated') results in
  vCPUs being pinned to pCPUs, per the original blueprint:

      http://specs.openstack.org/openstack/nova-
  specs/specs/kilo/implemented/virt-driver-cpu-pinning.html

  When combined with the 'vcpu_pin_set' nova configuration, it should be
  possible to get very good control over which pCPUs are used by
  instances. This works as expected when creating a multi-vCPU instance
  pinned to pCPUs on a single NUMA node. However, if these pCPUs are
  spread across multiple NUMA nodes then it fails. This behavior is not
  documented in the original blueprint and therefore should not function
  like so.

  ---

  # Testing Configuration

  Testing was conducted on a single-node, Fedora 21-based
  (3.17.8-300.fc21.x86_64) OpenStack instance (built with devstack). The
  system is a dual-socket, 10 core, HT-enabled system (2 sockets * 10
  cores * 2 threads = 40 "pCPUs". 0-9,20-29 = node0, 10-19,30-39 =
  node1). Two flavors were used:

      openstack flavor create --ram 4096 --disk 20 --vcpus 10 demo.no-
  pinning

      openstack flavor create --ram 4096 --disk 20 --vcpus 10 demo.pinning
      nova flavor-key demo.pinning set hw:cpu_policy=dedicated hw:cpu_thread_policy=isolate

  # Results

  Passing (same NUMA socket, contiguous)
  ======================================

   vcpu_pin_set = 10-19

  Pinned
  ------

   $ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.pinning test1 --wait
   $ ps aux | grep qemu | awk '{print }' | head -1 | xargs taskset -pc
   pid 68150's current affinity list: 10-19
   $ sudo -E virsh vcpucount 13
   maximum      config        10
   maximum      live          10
   current      config        10
   current      live          10

  Unpinned
  --------

   $ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.no-pinning test1 --wait
   $ ps aux | grep qemu | awk '{print }' | head -1 | xargs taskset -pc
   pid 70249's current affinity list: 10-19

  Passing (same NUMA socket, non-contiguous)
  ==========================================

   vcpu_pin_set = 5-9,25-29

  Pinned
  ------

   $ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.pinning test1 --wait
   $ ps aux | grep qemu | awk '{print }' | head -1 | xargs taskset -pc
   pid 91186's current affinity list: 5-9,25-29

  Unpinned
  --------

   $ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.no-pinning test1 --wait
   $ ps aux | grep qemu | awk '{print }' | head -1 | xargs taskset -pc
   pid 92212's current affinity list: 5-9,25-29

  Passing (multiple NUMA sockets, non-contiguous)
  ===============================================

   vcpu_pin_set = 5-10,25-29

  Pinned
  ------

   $ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.pinning test1 --wait
   $ ps aux | grep qemu | awk '{print }' | head -1 | xargs taskset -pc
   pid 93884's current affinity list: 5-9,25-29

  Unpinned
  --------

   $ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.no-pinning test1 --wait
   $ ps aux | grep qemu | awk '{print $2}' | head -1 | xargs taskset -pc
   pid 94468's current affinity list: 5-9,25-29

  Additional combinations
  -----------------------

   vcpu_pin_set = 5-14,25-29

  Failing (different NUMA socket, contiguous)
  ===========================================

   vcpu_pin_set = 9-18

  Pinned
  ------

   $ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.pinning test1 --wait
   ERROR: openstackclient.compute.v2.server.CreateServer Error creating server: test1

  Unpinned
  --------

   $ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.no-pinning test1 --wait
   $ ps aux | grep qemu | awk '{print }' | head -1 | xargs taskset -pc
   pid 71194's current affinity list: 9-18

  Failing (different NUMA socket, non-contiguous)
  ===============================================

   vcpu_pin_set = 5-9,15-19

  Pinned
  ------

   $ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.no-pinning test1 --wait
   ERROR: openstackclient.compute.v2.server.CreateServer Error creating server: test1
   $ openstack server show test1 | grep fault
   | fault                                | {u'message': u'No valid host was found. There are not enough hosts available.', u'code': 500,

  Unpinned
  --------

   $ openstack server create --image Fedora-x86_64-20-20140618-sda --flavor demo.no-pinning test1 --wait
   $ ps aux | grep qemu | awk '{print }' | head -1 | xargs taskset -pc
   pid 88703's current affinity list: 5-9,15-19

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1438253/+subscriptions