yahoo-eng-team team mailing list archive

[Launchpad Bug Tracker <1614054@xxxxxxxxxxxxxxxxxx>]

This bug was fixed in the package nova - 2:13.1.4-0ubuntu2

---------------
nova (2:13.1.4-0ubuntu2) xenial; urgency=medium

  * d/p/libvirt-fix-incorrect-host-cpus-giving-to-emulator-t.patch:
    Backport fix for cpu pinning libvirt config incorrect emulator
    pin cpuset (LP: #1614054).

 -- Edward Hope-Morley <edward.hope-morley@xxxxxxxxxxxxx>  Wed, 26 Jul
2017 06:46:18 +0100

** Changed in: nova (Ubuntu Xenial)
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1614054

Title:
  [SRU] Incorrect host cpu is given to emulator threads when
  cpu_realtime_mask flag is set

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive mitaka series:
  Fix Committed
Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) newton series:
  Fix Committed
Status in nova package in Ubuntu:
  Fix Released
Status in nova source package in Xenial:
  Fix Released

Bug description:
  [Impact]

  This bug affects users of Openstack Nova who want to create instances
  that will leverage the realtime functionality that libvirt/qemu offers
  by, amongst other things, pinning guest vcpus and qemu emulator
  threads to specific pcpus. Nova provides the means for the user to
  control, via the flavor hw:cpu_realtime_mask or image property
  hw_cpu_realtime_mask, which physical cpus these resources will pinned
  to. This mask allows you to mask the set of N pins that Nova selects
  such that 1 or more of your vcpus can be declared "real-time" by
  ensuring that they do not have emulator threads also pinned to them.
  The remaining "non-realtime" vcpus will have vcpu and emulator threads
  colocated. The bug fixes the case where e.g. you have a guest that has
  2 vcpus (logically 0 and 1) and Nova selects pcpus 14 and 22 and you
  have mask ^0 to indicate that you want all but the first vcpu to be
  realtime. This should result in the following being present in your
  libvirt xml for the guest:

    <cputune>
      <vcpupin vcpu='0' cpuset='14'/>
      <vcpupin vcpu='1' cpuset='22'/>
      <emulatorpin cpuset='14'/>
      <vcpusched vcpus='1' scheduler='fifo' priority='1'/>
    </cputune>

  But (current only Mitaka since it does not have this patch) you will
  get this:

    <cputune>
      <vcpupin vcpu='0' cpuset='14'/>
      <vcpupin vcpu='1' cpuset='22'/>
      <emulatorpin cpuset='0'/>
      <vcpusched vcpus='1' scheduler='fifo' priority='1'/>
    </cputune>

  i.e. Nova will always set the emulator pin to be id of the vcpu
  instead of the corresponding pcpu that it is pinned to.

  In terms of actual impact this could result in vcpus that are supposed
  to be isolated not being so and therefore not behaving as expected.

  [Test Case]

   * deploy openstack mitaka and configure nova.conf with vcpu-pin-
  set=0,1,2,3

     https://pastebin.ubuntu.com/25133260/

   * configure compute host kernel opts with "isolcpus=0,1,2,3" + reboot

   * create flavor with:

     openstack flavor create --public --ram 2048 --disk 10 --vcpus 2 --swap 0 test_flavor
     openstack flavor set --property hw:cpu_realtime_mask='^0' test_flavor
     openstack flavor set --property hw:cpu_policy=dedicated test_flavor
     openstack flavor set --property hw:cpu_thread_policy=prefer test_flavor
     openstack flavor set --property hw:cpu_realtime=yes test_flavor

   * boot instance with ^^ flavor

   * check that libvirt xml for vm has correct emulator pin cpuset #

  [Regression Potential]

  Since the patch being backported only touches the specific aread of
  code that was causing the original problem  and that code only serves
  to select cpusets based on flavor filters, i can't think of any
  regressions that it would introduce. However, one potential side
  effect/change to be aware of is that once nova-compute is upgraded to
  this newer version, any new instances created will have the
  correct/expected cpuset assignments whereas instances created prior to
  upgrade will remain unchanged i.e. they will all likely still have
  their emulation threads pinned to the wrong pcpu. In terms of side
  effects this will mean less load on the pcpu that was previously
  incorrectly chosen for existing guests but it will mean that older
  instances will need to be recreated in order to benefit from the fix.

  ====

  Description of problem:
  When using the cpu_realtime and cpu_realtim_mask flag to create new instance, the 'cpuset' of 'emulatorpin' option is using the id of vcpu which is incorrect. The id of host cpu should be used here.

  e.g.
    <cputune>
      <vcpupin vcpu='0' cpuset='2'/>
      <vcpupin vcpu='1' cpuset='3'/>
      <emulatorpin cpuset='0'/>          ### the cpuset should be '2' here, when cpu_realtime_mask=^0.
      <vcpusched vcpus='1' scheduler='fifo' priority='1'/>
    </cputune>

  How reproducible:
  Boot new instance with cpu_realtime_mask flavor.

  Steps to Reproduce:
  1. Create RT flavor
  nova flavor-create m1.small.performance 6 2048 20 2
  nova flavor-key m1.small.performance set hw:cpu_realtime=yes
  nova flavor-key m1.small.performance set hw:cpu_realtime_mask=^0
  nova flavor-key m1.small.performance set hw:cpu_policy=dedicated
  2. Boot a instance with this flavor
  3. Check the xml of the new instance

  Actual results:
    <cputune>
      <vcpupin vcpu='0' cpuset='2'/>
      <vcpupin vcpu='1' cpuset='3'/>
      <emulatorpin cpuset='0'/>
      <vcpusched vcpus='1' scheduler='fifo' priority='1'/>
    </cputune>

  Expected results:
    <cputune>
      <vcpupin vcpu='0' cpuset='2'/>
      <vcpupin vcpu='1' cpuset='3'/>
      <emulatorpin cpuset='2'/>
      <vcpusched vcpus='1' scheduler='fifo' priority='1'/>
    </cputune>

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1614054/+subscriptions