yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1924257] [NEW] nova get_capabilities should not use host capabilities

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: HYSong <1924257@xxxxxxxxxxxxxxxxxx>
Date: Thu, 15 Apr 2021 09:51:34 -0000
Reply-to: Bug 1924257 <1924257@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:


Nova-compute will configure Broadwell-IBRS's features if using host capabilities by the function of get_capabilities, but the VM's real features is depend on the model of Broadwell-IBRS when cpu_mode is configured to host-model in nova.conf.

I think it is unreasonable because nova will use host capabilities to
compare CPU when live_migration, but the VM isn't it. It is likely lead
to live migrate ERROR because of the incompatible features.

-----------------------------------------------------------------------------------------------------------
root@cmp004:~# qemu-system-x86_64 --version
QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1.4~u16.04+mcp2)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers

root@cmp004:~# libvirtd -V
libvirtd (libvirt) 4.0.0

openstack version: queens

root@cmp004:~# virsh domcapabilities
<domainCapabilities>
  <cpu>
    <mode name='host-model' supported='yes'>
      <model fallback='forbid'>Skylake-Client-IBRS</model>
      <vendor>Intel</vendor>
      <feature policy='require' name='ss'/>
      <feature policy='require' name='vmx'/>
      <feature policy='require' name='hypervisor'/>
      <feature policy='require' name='tsc_adjust'/>
      <feature policy='require' name='ssbd'/>
      <feature policy='require' name='pdpe1gb'/>
      <feature policy='require' name='invtsc'/>
      <feature policy='disable' name='mpx'/>
      <feature policy='disable' name='xsavec'/>
      <feature policy='disable' name='xgetbv1'/>
    </mode>
  </cpu>
</domainCapabilities>

root@cmp004:~# virsh capabilities
<capabilities>
  <host>
    <cpu>
      <arch>x86_64</arch>
      <model>Broadwell-IBRS</model>
      <vendor>Intel</vendor>
      <microcode version='184549422'/>
      <topology sockets='1' cores='8' threads='2'/>
      <feature name='vme'/>
      <feature name='ds'/>
      <feature name='acpi'/>
      <feature name='ss'/>
      <feature name='ht'/>
      <feature name='tm'/>
      <feature name='pbe'/>
      <feature name='dtes64'/>
      <feature name='monitor'/>
      <feature name='ds_cpl'/>
      <feature name='vmx'/>
      <feature name='smx'/>
      <feature name='est'/>
      <feature name='tm2'/>
      <feature name='xtpr'/>
      <feature name='pdcm'/>
      <feature name='dca'/>
      <feature name='osxsave'/>
      <feature name='f16c'/>
      <feature name='rdrand'/>
      <feature name='arat'/>
      <feature name='tsc_adjust'/>
      <feature name='cmt'/>
      <feature name='ssbd'/>
      <feature name='xsaveopt'/>
      <feature name='mbm_total'/>
      <feature name='mbm_local'/>
      <feature name='pdpe1gb'/>
      <feature name='abm'/>
      <feature name='invtsc'/>
      <pages unit='KiB' size='4'/>
      <pages unit='KiB' size='1048576'/>
    </cpu>
  </host>
<capabilities>

root@ctl01:~# nova hypervisor-show c8f34226-c2e9-4c09-bdbe-aaaff1a1d370
+---------------------------+------------------------------------------+
| Property                  | Value                                    |
+---------------------------+------------------------------------------+
| cpu_info_arch             | x86_64                                   |
| cpu_info_features         | ["pge", "avx", "xsaveopt", "clflush",    |
|                           | "sep", "rtm", "tsc_adjust", "tsc-        |
|                           | deadline", "dtes64", "invpcid", "tsc",   |
|                           | "fsgsbase", "xsave", "smap", "vmx",      |
|                           | "erms", "xtpr", "cmov", "hle", "smep",   |
|                           | "ssse3", "est", "pat", "monitor", "smx", |
|                           | "pbe", "lm", "msr", "adx",               |
|                           | "3dnowprefetch", "nx", "fxsr",           |
|                           | "syscall", "tm", "sse4.1", "pae",        |
|                           | "sse4.2", "pclmuldq", "cx16", "pcid",    |
|                           | "fma", "vme", "popcnt", "mmx",           |
|                           | "osxsave", "cx8", "mce", "de", "rdtscp", |
|                           | "ht", "dca", "lahf_lm", "abm", "rdseed", |
|                           | "pdcm", "mca", "pdpe1gb", "apic", "sse", |
|                           | "f16c", "pse", "ds", "invtsc", "pni",    |
|                           | "tm2", "avx2", "aes", "sse2", "ss",      |
|                           | "ds_cpl", "arat", "bmi1", "bmi2",        |
|                           | "acpi", "spec-ctrl", "fpu", "ssbd",      |
|                           | "pse36", "mtrr", "movbe", "rdrand",      |
|                           | "x2apic"]                                |
| cpu_info_model            | Broadwell-IBRS                           |
| service_host              | cmp004                                   |
| service_id                | 5e04fa07-db8a-4e84-a895-411c704b9d64     |
+---------------------------+------------------------------------------+

root@cmp004:~# ps -ef |grep instance-000124b5
root     16456 10674  0 17:24 pts/12   00:00:00 grep --color=auto instance-000124b5
libvirt+ 18623     1  7 16:15 ?        00:04:59 qemu-system-x86_64 -enable-kvm -cpu Skylake-Client-IBRS,ss=on,vmx=on,hypervisor=on,tsc_adjust=on,ssbd=on,pdpe1gb=on,mpx=off,xsavec=off,xgetbv1=off ...

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1924257

Title:
  nova get_capabilities should not use host capabilities

Status in OpenStack Compute (nova):
  New

Bug description:
  
  Nova-compute will configure Broadwell-IBRS's features if using host capabilities by the function of get_capabilities, but the VM's real features is depend on the model of Broadwell-IBRS when cpu_mode is configured to host-model in nova.conf.

  I think it is unreasonable because nova will use host capabilities to
  compare CPU when live_migration, but the VM isn't it. It is likely
  lead to live migrate ERROR because of the incompatible features.

  -----------------------------------------------------------------------------------------------------------
  root@cmp004:~# qemu-system-x86_64 --version
  QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1.4~u16.04+mcp2)
  Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers

  root@cmp004:~# libvirtd -V
  libvirtd (libvirt) 4.0.0

  openstack version: queens

  root@cmp004:~# virsh domcapabilities
  <domainCapabilities>
    <cpu>
      <mode name='host-model' supported='yes'>
        <model fallback='forbid'>Skylake-Client-IBRS</model>
        <vendor>Intel</vendor>
        <feature policy='require' name='ss'/>
        <feature policy='require' name='vmx'/>
        <feature policy='require' name='hypervisor'/>
        <feature policy='require' name='tsc_adjust'/>
        <feature policy='require' name='ssbd'/>
        <feature policy='require' name='pdpe1gb'/>
        <feature policy='require' name='invtsc'/>
        <feature policy='disable' name='mpx'/>
        <feature policy='disable' name='xsavec'/>
        <feature policy='disable' name='xgetbv1'/>
      </mode>
    </cpu>
  </domainCapabilities>

  root@cmp004:~# virsh capabilities
  <capabilities>
    <host>
      <cpu>
        <arch>x86_64</arch>
        <model>Broadwell-IBRS</model>
        <vendor>Intel</vendor>
        <microcode version='184549422'/>
        <topology sockets='1' cores='8' threads='2'/>
        <feature name='vme'/>
        <feature name='ds'/>
        <feature name='acpi'/>
        <feature name='ss'/>
        <feature name='ht'/>
        <feature name='tm'/>
        <feature name='pbe'/>
        <feature name='dtes64'/>
        <feature name='monitor'/>
        <feature name='ds_cpl'/>
        <feature name='vmx'/>
        <feature name='smx'/>
        <feature name='est'/>
        <feature name='tm2'/>
        <feature name='xtpr'/>
        <feature name='pdcm'/>
        <feature name='dca'/>
        <feature name='osxsave'/>
        <feature name='f16c'/>
        <feature name='rdrand'/>
        <feature name='arat'/>
        <feature name='tsc_adjust'/>
        <feature name='cmt'/>
        <feature name='ssbd'/>
        <feature name='xsaveopt'/>
        <feature name='mbm_total'/>
        <feature name='mbm_local'/>
        <feature name='pdpe1gb'/>
        <feature name='abm'/>
        <feature name='invtsc'/>
        <pages unit='KiB' size='4'/>
        <pages unit='KiB' size='1048576'/>
      </cpu>
    </host>
  <capabilities>

  root@ctl01:~# nova hypervisor-show c8f34226-c2e9-4c09-bdbe-aaaff1a1d370
  +---------------------------+------------------------------------------+
  | Property                  | Value                                    |
  +---------------------------+------------------------------------------+
  | cpu_info_arch             | x86_64                                   |
  | cpu_info_features         | ["pge", "avx", "xsaveopt", "clflush",    |
  |                           | "sep", "rtm", "tsc_adjust", "tsc-        |
  |                           | deadline", "dtes64", "invpcid", "tsc",   |
  |                           | "fsgsbase", "xsave", "smap", "vmx",      |
  |                           | "erms", "xtpr", "cmov", "hle", "smep",   |
  |                           | "ssse3", "est", "pat", "monitor", "smx", |
  |                           | "pbe", "lm", "msr", "adx",               |
  |                           | "3dnowprefetch", "nx", "fxsr",           |
  |                           | "syscall", "tm", "sse4.1", "pae",        |
  |                           | "sse4.2", "pclmuldq", "cx16", "pcid",    |
  |                           | "fma", "vme", "popcnt", "mmx",           |
  |                           | "osxsave", "cx8", "mce", "de", "rdtscp", |
  |                           | "ht", "dca", "lahf_lm", "abm", "rdseed", |
  |                           | "pdcm", "mca", "pdpe1gb", "apic", "sse", |
  |                           | "f16c", "pse", "ds", "invtsc", "pni",    |
  |                           | "tm2", "avx2", "aes", "sse2", "ss",      |
  |                           | "ds_cpl", "arat", "bmi1", "bmi2",        |
  |                           | "acpi", "spec-ctrl", "fpu", "ssbd",      |
  |                           | "pse36", "mtrr", "movbe", "rdrand",      |
  |                           | "x2apic"]                                |
  | cpu_info_model            | Broadwell-IBRS                           |
  | service_host              | cmp004                                   |
  | service_id                | 5e04fa07-db8a-4e84-a895-411c704b9d64     |
  +---------------------------+------------------------------------------+

  root@cmp004:~# ps -ef |grep instance-000124b5
  root     16456 10674  0 17:24 pts/12   00:00:00 grep --color=auto instance-000124b5
  libvirt+ 18623     1  7 16:15 ?        00:04:59 qemu-system-x86_64 -enable-kvm -cpu Skylake-Client-IBRS,ss=on,vmx=on,hypervisor=on,tsc_adjust=on,ssbd=on,pdpe1gb=on,mpx=off,xsavec=off,xgetbv1=off ...

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1924257/+subscriptions