yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #78980
[Bug 1829696] Re: qemu-kvm takes 100% CPU when running redhat/centos 7.6 guest VM OS
** Project changed: nova => qemu
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1829696
Title:
qemu-kvm takes 100% CPU when running redhat/centos 7.6 guest VM OS
Status in QEMU:
New
Bug description:
Description
===========
When running redhat or centos 7.6 guest os on vm,
the cpu usage is very low on vm(100% idle), but on host,
qemu-kvm reports 100% cpu busy usage.
After searching some related bugs report,
I suspect that it is due to the clock settings in vm's domain xml.
My Openstack cluster uses the default clock settings as follow:
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
And in this report, https://bugs.launchpad.net/qemu/+bug/1174654
it claims that <timer name='rtc' track='guest'/> can solve the 100% cpu usage problem when using Windows Image Guest OS,
but I makes some tests, the solusion dose not work for me.
Steps to reproduce
==================
* create a vm using centos or redhat 7.6 image
* using sar tool inside vm and host to check the cpu usage, and compare them
Expected result
===============
host's cpu usage report should be same with vm's cpu usage
Actual result
=============
vm's cpu usage is 100% idle, host's cpu usage is 100% busy
Environment
===========
1. Exact version of OpenStack you are running.
# rpm -qa | grep nova
openstack-nova-compute-13.1.2-1.el7.noarch
python2-novaclient-3.3.2-1.el7.noarch
python-nova-13.1.2-1.el7.noarch
openstack-nova-common-13.1.2-1.el7.noarch
2. Which hypervisor did you use?
(For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
What's the version of that?
# libvirtd -V
libvirtd (libvirt) 3.9.0
# /usr/libexec/qemu-kvm --version
QEMU emulator version 2.6.0 (qemu-kvm-ev-2.6.0-28.el7_3.6.1), Copyright (c) 2003-2008 Fabrice Bellard
Logs & Configs
==============
The VM xml:
<domain type='kvm' id='29'>
<name>instance-00005022</name>
<uuid>7f5a66a5-****-****-****-75dec****bbb</uuid>
<metadata>
<nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
<nova:package version="13.1.2-1.el7"/>
<nova:name>*******</nova:name>
<nova:creationTime>2019-05-20 03:08:46</nova:creationTime>
<nova:flavor name="2d2dab36-****-****-****-246e9****110">
<nova:memory>2048</nova:memory>
<nova:disk>12</nova:disk>
<nova:swap>2048</nova:swap>
<nova:ephemeral>0</nova:ephemeral>
<nova:vcpus>1</nova:vcpus>
</nova:flavor>
<nova:owner>
<nova:user uuid="********************">****</nova:user>
<nova:project uuid="********************">****</nova:project>
</nova:owner>
<nova:root type="image" uuid="4496a420-****-****-****-b50f****ada3"/>
</nova:instance>
</metadata>
<memory unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>2097152</currentMemory>
<vcpu placement='static'>1</vcpu>
<cputune>
<shares>1024</shares>
<vcpupin vcpu='0' cpuset='27'/>
<emulatorpin cpuset='27'/>
</cputune>
<numatune>
<memory mode='strict' nodeset='1'/>
<memnode cellid='0' mode='strict' nodeset='1'/>
</numatune>
<resource>
<partition>/machine</partition>
</resource>
<sysinfo type='smbios'>
<system>
<entry name='manufacturer'>Fedora Project</entry>
<entry name='product'>OpenStack Nova</entry>
<entry name='version'>13.1.2-1.el7</entry>
<entry name='serial'>64ab0e89-****-****-****-05312ef66983</entry>
<entry name='uuid'>7f5a66a5-****-****-****-75decaf82bbb</entry>
<entry name='family'>Virtual Machine</entry>
</system>
</sysinfo>
<os>
<type arch='x86_64' machine='pc-i440fx-rhel7.3.0'>hvm</type>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='custom' match='exact' check='full'>
<model fallback='forbid'>IvyBridge</model>
<topology sockets='1' cores='1' threads='1'/>
<feature policy='require' name='hypervisor'/>
<feature policy='require' name='arat'/>
<feature policy='require' name='xsaveopt'/>
<numa>
<cell id='0' cpus='0' memory='2097152' unit='KiB'/>
</numa>
</cpu>
<clock offset='utc'>
<timer name='pit' tickpolicy='delay'/>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source file='/data/instances/7f5a66a5-****-****-****-75decaf82bbb/disk'/>
<backingStore/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source file='/data/instances/7f5a66a5-****-****-****-75decaf82bbb/disk.swap'/>
<backingStore/>
<target dev='vdb' bus='virtio'/>
<alias name='virtio-disk1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw' cache='none'/>
<source file='/data/instances/7f5a66a5-****-****-****-75decaf82bbb/disk.config'/>
<backingStore/>
<target dev='hdd' bus='ide'/>
<readonly/>
<alias name='ide0-1-1'/>
<address type='drive' controller='0' bus='1' target='0' unit='1'/>
</disk>
<controller type='usb' index='0' model='piix3-uhci'>
<alias name='usb'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci.0'/>
</controller>
<controller type='ide' index='0'>
<alias name='ide'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<interface type='bridge'>
<mac address='fa:16:3e:a6:ea:4f'/>
<source bridge='brq52c66dc3-64'/>
<bandwidth>
<inbound average='102400'/>
<outbound average='102400'/>
</bandwidth>
<target dev='tapa29e94e5-42'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<serial type='file'>
<source path='/data/instances/7f5a66a5-****-****-****-75decaf82bbb/console.log'/>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
<alias name='serial0'/>
</serial>
<serial type='pty'>
<source path='/dev/pts/10'/>
<target type='isa-serial' port='1'>
<model name='isa-serial'/>
</target>
<alias name='serial1'/>
</serial>
<console type='file'>
<source path='/data/instances/7f5a66a5-****-****-****-75decaf82bbb/console.log'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<input type='tablet' bus='usb'>
<alias name='input0'/>
<address type='usb' bus='0' port='1'/>
</input>
<input type='mouse' bus='ps2'>
<alias name='input1'/>
</input>
<input type='keyboard' bus='ps2'>
<alias name='input2'/>
</input>
<graphics type='vnc' port='5910' autoport='yes' listen='0.0.0.0' keymap='en-us'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<video>
<model type='cirrus' vram='16384' heads='1' primary='yes'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<memballoon model='virtio'>
<stats period='10'/>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</memballoon>
</devices>
<seclabel type='dynamic' model='dac' relabel='yes'>
<label>+107:+107</label>
<imagelabel>+107:+107</imagelabel>
</seclabel>
</domain>
CPU Usage Report inside VM:
# sar -u -P 0 1 5
Linux 3.10.0-957.el7.x86_64 (******) 05/20/2019 _x86_64_ (1 CPU)
11:34:40 AM CPU %user %nice %system %iowait %steal %idle
11:34:41 AM 0 0.00 0.00 0.00 0.00 0.00 100.00
11:34:42 AM 0 0.00 0.00 0.00 0.00 0.00 100.00
11:34:43 AM 0 0.00 0.00 0.00 0.00 0.00 100.00
11:34:44 AM 0 0.00 0.00 0.00 0.00 0.00 100.00
11:34:45 AM 0 0.00 0.00 0.00 0.00 0.00 100.00
Average: 0 0.00 0.00 0.00 0.00 0.00 100.00
CPU Usage Report ON HOST(the vm's cpu is pinned on host's no.27 physic cpu):
# sar -u -P 27 1 5
Linux 3.10.0-862.el7.x86_64 (******) 05/20/2019 _x86_64_ (48 CPU)
11:34:40 AM CPU %user %nice %system %iowait %steal %idle
11:34:41 AM 27 100.00 0.00 0.00 0.00 0.00 0.00
11:34:42 AM 27 100.00 0.00 0.00 0.00 0.00 0.00
11:34:43 AM 27 100.00 0.00 0.00 0.00 0.00 0.00
11:34:44 AM 27 100.00 0.00 0.00 0.00 0.00 0.00
11:34:45 AM 27 100.00 0.00 0.00 0.00 0.00 0.00
Average: 27 100.00 0.00 0.00 0.00 0.00 0.00
clocksource inside VM:
# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
kvm_clock
To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1829696/+subscriptions
References