yahoo-eng-team team mailing list archive

[Chris Friesen <chris.friesen@xxxxxxxxxxxxx>]

Public bug reported:

I'm running nova trunk, commit 752954a.

I configured a flavor with two vcpus and extra specs
"hw:cpu_policy=dedicated" in order to enable vcpu pinning.

I booted up an instance with this flavor, and "virsh dumpxml" shows that
the two vcpus were affined suitably to host cpus, but the emulator
thread was left to float across the available host cores on that numa
node.

  <cputune>
    <shares>2048</shares>
        <vcpupin vcpu='0' policy='other' priority='0' cpuset='4'/>
        <vcpupin vcpu='1' policy='other' priority='0' cpuset='5'/>
    <emulatorpin cpuset='3-11'/>
  </cputune>


Looking at the kvm process shortly after creation, we see quite a few
emulator threads running with the emulatorpin affinity:

compute-2:~$ taskset -apc 136143
pid 136143's current affinity list: 3-11
pid 136144's current affinity list: 0,3-24,27-47
pid 136146's current affinity list: 4
pid 136147's current affinity list: 5
pid 136149's current affinity list: 0
pid 136433's current affinity list: 3-11
pid 136434's current affinity list: 3-11
pid 136435's current affinity list: 3-11
pid 136436's current affinity list: 3-11
pid 136437's current affinity list: 3-11
pid 136438's current affinity list: 3-11
pid 136439's current affinity list: 3-11
pid 136440's current affinity list: 3-11
pid 136441's current affinity list: 3-11
pid 136442's current affinity list: 3-11
pid 136443's current affinity list: 3-11
pid 136444's current affinity list: 3-11
pid 136445's current affinity list: 3-11
pid 136446's current affinity list: 3-11
pid 136447's current affinity list: 3-11
pid 136448's current affinity list: 3-11
pid 136449's current affinity list: 3-11
pid 136450's current affinity list: 3-11
pid 136451's current affinity list: 3-11
pid 136452's current affinity list: 3-11
pid 136453's current affinity list: 3-11
pid 136454's current affinity list: 3-11


Since the purpose of "hw:cpu_policy=dedicated" is to provide a dedicated host CPU for each guest CPU, the libvirt emulatorpin cpuset for a given guest should be set to one (or possibly more) of the CPUs specified for that guest.  Otherwise, any work done by the emulator threads could rob CPU time from another guest instance.

Personally I'd like to see the emulator thread affined the same as guest
vCPU 0 (we use guest vCPU0 as a maintenance processor while doing the
"real work" on the other vCPUs), but an argument could be made that it
should be affined to the logical OR of all the guest vCPU cpusets.

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: compute

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1417671

Title:
  when using dedicated cpus, the emulator thread should be affined as
  well

Status in OpenStack Compute (Nova):
  New

Bug description:
  I'm running nova trunk, commit 752954a.

  I configured a flavor with two vcpus and extra specs
  "hw:cpu_policy=dedicated" in order to enable vcpu pinning.

  I booted up an instance with this flavor, and "virsh dumpxml" shows
  that the two vcpus were affined suitably to host cpus, but the
  emulator thread was left to float across the available host cores on
  that numa node.

    <cputune>
      <shares>2048</shares>
          <vcpupin vcpu='0' policy='other' priority='0' cpuset='4'/>
          <vcpupin vcpu='1' policy='other' priority='0' cpuset='5'/>
      <emulatorpin cpuset='3-11'/>
    </cputune>


  Looking at the kvm process shortly after creation, we see quite a few
  emulator threads running with the emulatorpin affinity:

  compute-2:~$ taskset -apc 136143
  pid 136143's current affinity list: 3-11
  pid 136144's current affinity list: 0,3-24,27-47
  pid 136146's current affinity list: 4
  pid 136147's current affinity list: 5
  pid 136149's current affinity list: 0
  pid 136433's current affinity list: 3-11
  pid 136434's current affinity list: 3-11
  pid 136435's current affinity list: 3-11
  pid 136436's current affinity list: 3-11
  pid 136437's current affinity list: 3-11
  pid 136438's current affinity list: 3-11
  pid 136439's current affinity list: 3-11
  pid 136440's current affinity list: 3-11
  pid 136441's current affinity list: 3-11
  pid 136442's current affinity list: 3-11
  pid 136443's current affinity list: 3-11
  pid 136444's current affinity list: 3-11
  pid 136445's current affinity list: 3-11
  pid 136446's current affinity list: 3-11
  pid 136447's current affinity list: 3-11
  pid 136448's current affinity list: 3-11
  pid 136449's current affinity list: 3-11
  pid 136450's current affinity list: 3-11
  pid 136451's current affinity list: 3-11
  pid 136452's current affinity list: 3-11
  pid 136453's current affinity list: 3-11
  pid 136454's current affinity list: 3-11

  
  Since the purpose of "hw:cpu_policy=dedicated" is to provide a dedicated host CPU for each guest CPU, the libvirt emulatorpin cpuset for a given guest should be set to one (or possibly more) of the CPUs specified for that guest.  Otherwise, any work done by the emulator threads could rob CPU time from another guest instance.

  Personally I'd like to see the emulator thread affined the same as
  guest vCPU 0 (we use guest vCPU0 as a maintenance processor while
  doing the "real work" on the other vCPUs), but an argument could be
  made that it should be affined to the logical OR of all the guest vCPU
  cpusets.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1417671/+subscriptions