yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1829696] [NEW] qemu-kvm takes 100% CPU when running redhat/centos 7.6 guest VM OS

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: qianxi <qianxi416@xxxxxx>
Date: Mon, 20 May 2019 03:55:05 -0000
Reply-to: Bug 1829696 <1829696@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========
When running redhat or centos 7.6 guest os on vm,
the cpu usage is very low on vm(100% idle), but on host,
qemu-kvm reports 100% cpu busy usage.

After searching some related bugs report,
I suspect that it is due to the clock settings in vm's domain xml.
My Openstack cluster uses the default clock settings as follow:
    <clock offset='utc'>
      <timer name='rtc' tickpolicy='catchup'/>
      <timer name='pit' tickpolicy='delay'/>
      <timer name='hpet' present='no'/>
    </clock>
And in this report, https://bugs.launchpad.net/qemu/+bug/1174654
it claims that <timer name='rtc' track='guest'/> can solve the 100% cpu usage problem when using Windows Image Guest OS,
but I makes some tests, the solusion dose not work for me.


Steps to reproduce
==================
* create a vm using centos or redhat 7.6 image
* using sar tool inside vm and host to check the cpu usage, and compare them


Expected result
===============
host's cpu usage report should be same with vm's cpu usage


Actual result
=============
vm's cpu usage is 100% idle, host's cpu usage is 100% busy


Environment
===========
1. Exact version of OpenStack you are running.
# rpm -qa | grep nova
openstack-nova-compute-13.1.2-1.el7.noarch
python2-novaclient-3.3.2-1.el7.noarch
python-nova-13.1.2-1.el7.noarch
openstack-nova-common-13.1.2-1.el7.noarch

2. Which hypervisor did you use?
   (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
   What's the version of that?
# libvirtd -V
libvirtd (libvirt) 3.9.0

# /usr/libexec/qemu-kvm --version
QEMU emulator version 2.6.0 (qemu-kvm-ev-2.6.0-28.el7_3.6.1), Copyright (c) 2003-2008 Fabrice Bellard


Logs & Configs
==============
The VM xml:
<domain type='kvm' id='29'>
  <name>instance-00005022</name>
  <uuid>7f5a66a5-****-****-****-75dec****bbb</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";>
      <nova:package version="13.1.2-1.el7"/>
      <nova:name>*******</nova:name>
      <nova:creationTime>2019-05-20 03:08:46</nova:creationTime>
      <nova:flavor name="2d2dab36-****-****-****-246e9****110">
        <nova:memory>2048</nova:memory>
        <nova:disk>12</nova:disk>
        <nova:swap>2048</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>1</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="********************">****</nova:user>
        <nova:project uuid="********************">****</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="4496a420-****-****-****-b50f****ada3"/>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <cputune>
    <shares>1024</shares>
    <vcpupin vcpu='0' cpuset='27'/>
    <emulatorpin cpuset='27'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='1'/>
    <memnode cellid='0' mode='strict' nodeset='1'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>Fedora Project</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>13.1.2-1.el7</entry>
      <entry name='serial'>64ab0e89-****-****-****-05312ef66983</entry>
      <entry name='uuid'>7f5a66a5-****-****-****-75decaf82bbb</entry>
      <entry name='family'>Virtual Machine</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.3.0'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>IvyBridge</model>
    <topology sockets='1' cores='1' threads='1'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='arat'/>
    <feature policy='require' name='xsaveopt'/>
    <numa>
      <cell id='0' cpus='0' memory='2097152' unit='KiB'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/data/instances/7f5a66a5-****-****-****-75decaf82bbb/disk'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/data/instances/7f5a66a5-****-****-****-75decaf82bbb/disk.swap'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/data/instances/7f5a66a5-****-****-****-75decaf82bbb/disk.config'/>
      <backingStore/>
      <target dev='hdd' bus='ide'/>
      <readonly/>
      <alias name='ide0-1-1'/>
      <address type='drive' controller='0' bus='1' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='piix3-uhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='fa:16:3e:a6:ea:4f'/>
      <source bridge='brq52c66dc3-64'/>
      <bandwidth>
        <inbound average='102400'/>
        <outbound average='102400'/>
      </bandwidth>
      <target dev='tapa29e94e5-42'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='file'>
      <source path='/data/instances/7f5a66a5-****-****-****-75decaf82bbb/console.log'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <serial type='pty'>
      <source path='/dev/pts/10'/>
      <target type='isa-serial' port='1'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial1'/>
    </serial>
    <console type='file'>
      <source path='/data/instances/7f5a66a5-****-****-****-75decaf82bbb/console.log'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='vnc' port='5910' autoport='yes' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <stats period='10'/>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+107:+107</label>
    <imagelabel>+107:+107</imagelabel>
  </seclabel>
</domain>

CPU Usage Report inside VM:
# sar -u -P 0 1 5
Linux 3.10.0-957.el7.x86_64 (******) 	05/20/2019 	_x86_64_	(1 CPU)

11:34:40 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
11:34:41 AM       0      0.00      0.00      0.00      0.00      0.00    100.00
11:34:42 AM       0      0.00      0.00      0.00      0.00      0.00    100.00
11:34:43 AM       0      0.00      0.00      0.00      0.00      0.00    100.00
11:34:44 AM       0      0.00      0.00      0.00      0.00      0.00    100.00
11:34:45 AM       0      0.00      0.00      0.00      0.00      0.00    100.00
Average:          0      0.00      0.00      0.00      0.00      0.00    100.00

CPU Usage Report ON HOST(the vm's cpu is pinned on host's no.27 physic cpu):
# sar -u -P 27 1 5
Linux 3.10.0-862.el7.x86_64 (******) 	05/20/2019 	_x86_64_	(48 CPU)

11:34:40 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
11:34:41 AM      27    100.00      0.00      0.00      0.00      0.00      0.00
11:34:42 AM      27    100.00      0.00      0.00      0.00      0.00      0.00
11:34:43 AM      27    100.00      0.00      0.00      0.00      0.00      0.00
11:34:44 AM      27    100.00      0.00      0.00      0.00      0.00      0.00
11:34:45 AM      27    100.00      0.00      0.00      0.00      0.00      0.00
Average:         27    100.00      0.00      0.00      0.00      0.00      0.00

clocksource inside VM:
# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
kvm_clock

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: centos clock qemu-kvm redhat

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1829696

Title:
  qemu-kvm takes 100% CPU when running redhat/centos 7.6 guest VM OS

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  When running redhat or centos 7.6 guest os on vm,
  the cpu usage is very low on vm(100% idle), but on host,
  qemu-kvm reports 100% cpu busy usage.

  After searching some related bugs report,
  I suspect that it is due to the clock settings in vm's domain xml.
  My Openstack cluster uses the default clock settings as follow:
      <clock offset='utc'>
        <timer name='rtc' tickpolicy='catchup'/>
        <timer name='pit' tickpolicy='delay'/>
        <timer name='hpet' present='no'/>
      </clock>
  And in this report, https://bugs.launchpad.net/qemu/+bug/1174654
  it claims that <timer name='rtc' track='guest'/> can solve the 100% cpu usage problem when using Windows Image Guest OS,
  but I makes some tests, the solusion dose not work for me.

  
  Steps to reproduce
  ==================
  * create a vm using centos or redhat 7.6 image
  * using sar tool inside vm and host to check the cpu usage, and compare them

  
  Expected result
  ===============
  host's cpu usage report should be same with vm's cpu usage

  
  Actual result
  =============
  vm's cpu usage is 100% idle, host's cpu usage is 100% busy

  
  Environment
  ===========
  1. Exact version of OpenStack you are running.
  # rpm -qa | grep nova
  openstack-nova-compute-13.1.2-1.el7.noarch
  python2-novaclient-3.3.2-1.el7.noarch
  python-nova-13.1.2-1.el7.noarch
  openstack-nova-common-13.1.2-1.el7.noarch

  2. Which hypervisor did you use?
     (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
     What's the version of that?
  # libvirtd -V
  libvirtd (libvirt) 3.9.0

  # /usr/libexec/qemu-kvm --version
  QEMU emulator version 2.6.0 (qemu-kvm-ev-2.6.0-28.el7_3.6.1), Copyright (c) 2003-2008 Fabrice Bellard

  
  Logs & Configs
  ==============
  The VM xml:
  <domain type='kvm' id='29'>
    <name>instance-00005022</name>
    <uuid>7f5a66a5-****-****-****-75dec****bbb</uuid>
    <metadata>
      <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";>
        <nova:package version="13.1.2-1.el7"/>
        <nova:name>*******</nova:name>
        <nova:creationTime>2019-05-20 03:08:46</nova:creationTime>
        <nova:flavor name="2d2dab36-****-****-****-246e9****110">
          <nova:memory>2048</nova:memory>
          <nova:disk>12</nova:disk>
          <nova:swap>2048</nova:swap>
          <nova:ephemeral>0</nova:ephemeral>
          <nova:vcpus>1</nova:vcpus>
        </nova:flavor>
        <nova:owner>
          <nova:user uuid="********************">****</nova:user>
          <nova:project uuid="********************">****</nova:project>
        </nova:owner>
        <nova:root type="image" uuid="4496a420-****-****-****-b50f****ada3"/>
      </nova:instance>
    </metadata>
    <memory unit='KiB'>2097152</memory>
    <currentMemory unit='KiB'>2097152</currentMemory>
    <vcpu placement='static'>1</vcpu>
    <cputune>
      <shares>1024</shares>
      <vcpupin vcpu='0' cpuset='27'/>
      <emulatorpin cpuset='27'/>
    </cputune>
    <numatune>
      <memory mode='strict' nodeset='1'/>
      <memnode cellid='0' mode='strict' nodeset='1'/>
    </numatune>
    <resource>
      <partition>/machine</partition>
    </resource>
    <sysinfo type='smbios'>
      <system>
        <entry name='manufacturer'>Fedora Project</entry>
        <entry name='product'>OpenStack Nova</entry>
        <entry name='version'>13.1.2-1.el7</entry>
        <entry name='serial'>64ab0e89-****-****-****-05312ef66983</entry>
        <entry name='uuid'>7f5a66a5-****-****-****-75decaf82bbb</entry>
        <entry name='family'>Virtual Machine</entry>
      </system>
    </sysinfo>
    <os>
      <type arch='x86_64' machine='pc-i440fx-rhel7.3.0'>hvm</type>
      <boot dev='hd'/>
      <smbios mode='sysinfo'/>
    </os>
    <features>
      <acpi/>
      <apic/>
    </features>
    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>IvyBridge</model>
      <topology sockets='1' cores='1' threads='1'/>
      <feature policy='require' name='hypervisor'/>
      <feature policy='require' name='arat'/>
      <feature policy='require' name='xsaveopt'/>
      <numa>
        <cell id='0' cpus='0' memory='2097152' unit='KiB'/>
      </numa>
    </cpu>
    <clock offset='utc'>
      <timer name='pit' tickpolicy='delay'/>
      <timer name='rtc' tickpolicy='catchup'/>
      <timer name='hpet' present='no'/>
    </clock>
    <on_poweroff>destroy</on_poweroff>
    <on_reboot>restart</on_reboot>
    <on_crash>destroy</on_crash>
    <devices>
      <emulator>/usr/libexec/qemu-kvm</emulator>
      <disk type='file' device='disk'>
        <driver name='qemu' type='raw' cache='none'/>
        <source file='/data/instances/7f5a66a5-****-****-****-75decaf82bbb/disk'/>
        <backingStore/>
        <target dev='vda' bus='virtio'/>
        <alias name='virtio-disk0'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
      </disk>
      <disk type='file' device='disk'>
        <driver name='qemu' type='raw' cache='none'/>
        <source file='/data/instances/7f5a66a5-****-****-****-75decaf82bbb/disk.swap'/>
        <backingStore/>
        <target dev='vdb' bus='virtio'/>
        <alias name='virtio-disk1'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
      </disk>
      <disk type='file' device='cdrom'>
        <driver name='qemu' type='raw' cache='none'/>
        <source file='/data/instances/7f5a66a5-****-****-****-75decaf82bbb/disk.config'/>
        <backingStore/>
        <target dev='hdd' bus='ide'/>
        <readonly/>
        <alias name='ide0-1-1'/>
        <address type='drive' controller='0' bus='1' target='0' unit='1'/>
      </disk>
      <controller type='usb' index='0' model='piix3-uhci'>
        <alias name='usb'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
      </controller>
      <controller type='pci' index='0' model='pci-root'>
        <alias name='pci.0'/>
      </controller>
      <controller type='ide' index='0'>
        <alias name='ide'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
      </controller>
      <interface type='bridge'>
        <mac address='fa:16:3e:a6:ea:4f'/>
        <source bridge='brq52c66dc3-64'/>
        <bandwidth>
          <inbound average='102400'/>
          <outbound average='102400'/>
        </bandwidth>
        <target dev='tapa29e94e5-42'/>
        <model type='virtio'/>
        <alias name='net0'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
      </interface>
      <serial type='file'>
        <source path='/data/instances/7f5a66a5-****-****-****-75decaf82bbb/console.log'/>
        <target type='isa-serial' port='0'>
          <model name='isa-serial'/>
        </target>
        <alias name='serial0'/>
      </serial>
      <serial type='pty'>
        <source path='/dev/pts/10'/>
        <target type='isa-serial' port='1'>
          <model name='isa-serial'/>
        </target>
        <alias name='serial1'/>
      </serial>
      <console type='file'>
        <source path='/data/instances/7f5a66a5-****-****-****-75decaf82bbb/console.log'/>
        <target type='serial' port='0'/>
        <alias name='serial0'/>
      </console>
      <input type='tablet' bus='usb'>
        <alias name='input0'/>
        <address type='usb' bus='0' port='1'/>
      </input>
      <input type='mouse' bus='ps2'>
        <alias name='input1'/>
      </input>
      <input type='keyboard' bus='ps2'>
        <alias name='input2'/>
      </input>
      <graphics type='vnc' port='5910' autoport='yes' listen='0.0.0.0' keymap='en-us'>
        <listen type='address' address='0.0.0.0'/>
      </graphics>
      <video>
        <model type='cirrus' vram='16384' heads='1' primary='yes'/>
        <alias name='video0'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
      </video>
      <memballoon model='virtio'>
        <stats period='10'/>
        <alias name='balloon0'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
      </memballoon>
    </devices>
    <seclabel type='dynamic' model='dac' relabel='yes'>
      <label>+107:+107</label>
      <imagelabel>+107:+107</imagelabel>
    </seclabel>
  </domain>

  CPU Usage Report inside VM:
  # sar -u -P 0 1 5
  Linux 3.10.0-957.el7.x86_64 (******) 	05/20/2019 	_x86_64_	(1 CPU)

  11:34:40 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
  11:34:41 AM       0      0.00      0.00      0.00      0.00      0.00    100.00
  11:34:42 AM       0      0.00      0.00      0.00      0.00      0.00    100.00
  11:34:43 AM       0      0.00      0.00      0.00      0.00      0.00    100.00
  11:34:44 AM       0      0.00      0.00      0.00      0.00      0.00    100.00
  11:34:45 AM       0      0.00      0.00      0.00      0.00      0.00    100.00
  Average:          0      0.00      0.00      0.00      0.00      0.00    100.00

  CPU Usage Report ON HOST(the vm's cpu is pinned on host's no.27 physic cpu):
  # sar -u -P 27 1 5
  Linux 3.10.0-862.el7.x86_64 (******) 	05/20/2019 	_x86_64_	(48 CPU)

  11:34:40 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
  11:34:41 AM      27    100.00      0.00      0.00      0.00      0.00      0.00
  11:34:42 AM      27    100.00      0.00      0.00      0.00      0.00      0.00
  11:34:43 AM      27    100.00      0.00      0.00      0.00      0.00      0.00
  11:34:44 AM      27    100.00      0.00      0.00      0.00      0.00      0.00
  11:34:45 AM      27    100.00      0.00      0.00      0.00      0.00      0.00
  Average:         27    100.00      0.00      0.00      0.00      0.00      0.00

  clocksource inside VM:
  # cat /sys/devices/system/clocksource/clocksource0/current_clocksource
  kvm_clock

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1829696/+subscriptions
Follow ups

[Bug 1829696] Re: qemu-kvm takes 100% CPU when running redhat/centos 7.6 guest VM OS
From: Kashyap Chamarthy, 2019-06-20