yahoo-eng-team team mailing list archive

[sean mooney <1930706@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

today when ever you use a realtime guest you are required to enable cpu
pinning and other feature such as spcifing a real time core mask via
hw:cpu_realtime_mask or hw_cpu_realtime_mask.

in the victoria release this requriement was relaxed somewhat with the
intoduction of mixed cpu policy guest that are assinged pinned and
floating cores.

https://github.com/openstack/nova/commit/9fc63c764429c10f9041e6b53659e0cbd595bf6b


It is now possible to allocate all cores in an instance to realtime and
omit the ``hw:cpu_realtime_mask`` extra spec. This requires specifying the
``hw:emulator_threads_policy`` extra spec.

https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/releasenotes/notes/bug-1884231-16acf297d88b122e.yaml

however while that works well it also possible to hw:cpu_realtime_mask
but not specify hw:emulator_threads_policy which leads to sub optimal
xml generation for the libvirt driver.

this is reported downstream as
https://bugzilla.redhat.com/show_bug.cgi?id=1700390 for older releas
that predata the changes referenced above.

though in revaluation of this a possible improvment can be made as
detailed in https://bugzilla.redhat.com/show_bug.cgi?id=1700390#c11


today if we have a 2 core vm where guest cpu 0 is non realtime and guest cpu 1 is realtime we 
.e.g. hw:cpu_policy=dedicated hw:cpu_realtime=True hw:cpu_realtime_mask=^0
would generate the xml as follows
  <vcpupin vcpu="0" cpuset="0"/>
  <vcpupin vcpu="1" cpuset="1"/>
  <vcpusched vcpus='1' scheduler='fifo' priority='1'/>
  <emulatorpin cpuset="0,1"/>

this is because the default behavior when no emulator_threads_policy is specifed is for the
emulator thread to float over all the vm cores.

but a slight modifcation to the xml could be made to have a more optimal default in this case
useing the cpu_realtime_mask we can instead restrict the emulator thread to float over the non realtime cores with realtime priortiy.

  <vcpupin vcpu="0" cpuset="0"/>
  <vcpupin vcpu="1" cpuset="1"/>
  <vcpusched vcpus='1' scheduler='fifo' priority='1'/>
  <emulatorpin cpuset="0"/>
  <emulatorsched scheduler='fifo' priority='1'>

this will ensure that if qemu need to process a request for a device attach for example
that the emulator thread has higher priorty then the guest vcpus that deal with guest house keeping task but will not interupt the realtime cores.

this would give many of the benifits of emulator_threads_policy=share or
emulator_threads_policy=isolate without increase resource usage or
requireing any config,flavor or image changes. this should also be a
backporable solution to this problem.

this is espically important given realtime host often are deplopy with
the kernel isolcpus paramater which mean that the kernel will not load
balance the emulator thread acrros the range and will instead leave it
onthe core it intially spwaned on. today you coudl get lucky and it
could be spawn on core 0 in which case the new behvior would be the same
or it coudl get spwaned on core 1. wehn the emulatro thread is spawned
on core 1 sicne it has less priority then the vcpu thread it will only
run if the guest vcpu idels resulting in the iablity for qemu to process
device attach and other qemu monitor commands form libvirt or the user.

** Affects: nova
     Importance: Wishlist
         Status: Triaged


** Tags: libvirt numa

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1930706

Title:
  nova allows suboptimal emulator tread pinning for realtime guests

Status in OpenStack Compute (nova):
  Triaged

Bug description:
  today when ever you use a realtime guest you are required to enable
  cpu pinning and other feature such as spcifing a real time core mask
  via hw:cpu_realtime_mask or hw_cpu_realtime_mask.

  in the victoria release this requriement was relaxed somewhat with the
  intoduction of mixed cpu policy guest that are assinged pinned and
  floating cores.

  https://github.com/openstack/nova/commit/9fc63c764429c10f9041e6b53659e0cbd595bf6b

  
  It is now possible to allocate all cores in an instance to realtime and
  omit the ``hw:cpu_realtime_mask`` extra spec. This requires specifying the
  ``hw:emulator_threads_policy`` extra spec.

  https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/releasenotes/notes/bug-1884231-16acf297d88b122e.yaml

  however while that works well it also possible to hw:cpu_realtime_mask
  but not specify hw:emulator_threads_policy which leads to sub optimal
  xml generation for the libvirt driver.

  this is reported downstream as
  https://bugzilla.redhat.com/show_bug.cgi?id=1700390 for older releas
  that predata the changes referenced above.

  though in revaluation of this a possible improvment can be made as
  detailed in https://bugzilla.redhat.com/show_bug.cgi?id=1700390#c11

  
  today if we have a 2 core vm where guest cpu 0 is non realtime and guest cpu 1 is realtime we 
  .e.g. hw:cpu_policy=dedicated hw:cpu_realtime=True hw:cpu_realtime_mask=^0
  would generate the xml as follows
    <vcpupin vcpu="0" cpuset="0"/>
    <vcpupin vcpu="1" cpuset="1"/>
    <vcpusched vcpus='1' scheduler='fifo' priority='1'/>
    <emulatorpin cpuset="0,1"/>

  this is because the default behavior when no emulator_threads_policy is specifed is for the
  emulator thread to float over all the vm cores.

  but a slight modifcation to the xml could be made to have a more optimal default in this case
  useing the cpu_realtime_mask we can instead restrict the emulator thread to float over the non realtime cores with realtime priortiy.

    <vcpupin vcpu="0" cpuset="0"/>
    <vcpupin vcpu="1" cpuset="1"/>
    <vcpusched vcpus='1' scheduler='fifo' priority='1'/>
    <emulatorpin cpuset="0"/>
    <emulatorsched scheduler='fifo' priority='1'>

  this will ensure that if qemu need to process a request for a device attach for example
  that the emulator thread has higher priorty then the guest vcpus that deal with guest house keeping task but will not interupt the realtime cores.

  this would give many of the benifits of emulator_threads_policy=share
  or emulator_threads_policy=isolate without increase resource usage or
  requireing any config,flavor or image changes. this should also be a
  backporable solution to this problem.

  this is espically important given realtime host often are deplopy with
  the kernel isolcpus paramater which mean that the kernel will not load
  balance the emulator thread acrros the range and will instead leave it
  onthe core it intially spwaned on. today you coudl get lucky and it
  could be spawn on core 0 in which case the new behvior would be the
  same or it coudl get spwaned on core 1. wehn the emulatro thread is
  spawned on core 1 sicne it has less priority then the vcpu thread it
  will only run if the guest vcpu idels resulting in the iablity for
  qemu to process device attach and other qemu monitor commands form
  libvirt or the user.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1930706/+subscriptions