yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1761798] Re: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8"

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Balazs Gibizer <1761798@xxxxxxxxxxxxxxxxxx>
Date: Mon, 23 Nov 2020 15:17:15 -0000
Reply-to: Bug 1761798 <1761798@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Source node 
https://zuul.opendev.org/t/openstack/build/d50877ae15db4022b82f4bb1d1d52cea/log/logs/subnode-2/libvirt/qemu/instance-0000001a.txt

2020-11-20 14:25:24.887+0000: starting up libvirt version: 6.0.0,
package: 0ubuntu8.4~cloud0 (Openstack Ubuntu Testing Bot <openstack-
testing-bot@xxxxxxxxxx> Tue, 15 Sep 2020 20:36:28 +0000), qemu version:
4.2.1Debian 1:4.2-3ubuntu6.7~cloud0, kernel: 4.15.0-124-generic,
hostname: ubuntu-bionic-ovh-bhs1-0021872195

LC_ALL=C \

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \

HOME=/var/lib/libvirt/qemu/domain-19-instance-0000001a \

XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-19-instance-
0000001a/.local/share \

XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-19-instance-0000001a/.cache
\

XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-19-instance-
0000001a/.config \

QEMU_AUDIO_DRV=none \

/usr/bin/qemu-system-x86_64 \

-name guest=instance-0000001a,debug-threads=on \

-S \

-object
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-19
-instance-0000001a/master-key.aes \

-machine pc-i440fx-4.2,accel=tcg,usb=off,dump-guest-core=off \

-cpu qemu64,hypervisor=on,lahf-lm=on \

-m 128 \

-overcommit mem-lock=off \

-smp 1,sockets=1,cores=1,threads=1 \

-uuid 2c468d92-4b19-426a-8c25-16b4624c21a4 \

-smbios 'type=1,manufacturer=OpenStack Foundation,product=OpenStack
Nova,version=22.1.0,serial=2c468d92-4b19-426a-
8c25-16b4624c21a4,uuid=2c468d92-4b19-426a-
8c25-16b4624c21a4,family=Virtual Machine' \

-no-user-config \

-nodefaults \

-chardev socket,id=charmonitor,fd=35,server,nowait \

-mon chardev=charmonitor,id=monitor,mode=control \

-rtc base=utc \

-no-shutdown \

-boot strict=on \

-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \

-blockdev
'{"driver":"file","filename":"/opt/stack/data/nova/instances/_base/61bd5e531ab4c82456aa5300ede7266b3610be79
","node-name":"libvirt-2-storage","cache":{"direct":true,"no-
flush":false},"auto-read-only":true,"discard":"unmap"}' \

-blockdev '{"node-name":"libvirt-2-format","read-
only":true,"cache":{"direct":true,"no-
flush":false},"driver":"raw","file":"libvirt-2-storage"}' \

-blockdev
'{"driver":"file","filename":"/opt/stack/data/nova/instances/2c468d92-4b19
-426a-8c25-16b4624c21a4/disk","node-
name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false
},"auto-read-only":true,"discard":"unmap"}' \

-blockdev '{"node-name":"libvirt-1-format","read-
only":false,"cache":{"direct":true,"no-
flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":"libvirt-2-format"}'
\

-device virtio-blk-
pci,scsi=off,bus=pci.0,addr=0x4,drive=libvirt-1-format,id=virtio-
disk0,bootindex=1,write-cache=on \

-netdev tap,fd=37,id=hostnet0 \

-device virtio-net-
pci,host_mtu=1400,netdev=hostnet0,id=net0,mac=fa:16:3e:43:11:f4,bus=pci.0,addr=0x3
\

-add-fd set=2,fd=39 \

-chardev pty,id=charserial0,logfile=/dev/fdset/2,logappend=on \

-device isa-serial,chardev=charserial0,id=serial0 \

-vnc 127.0.0.1:0 \

-device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \

-incoming defer \

-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 \

-object rng-random,id=objrng0,filename=/dev/urandom \

-device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 \

-sandbox
on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny
\

-msg timestamp=on

char device redirected to /dev/pts/2 (label charserial0)

virtio: bogus descriptor or out of resources

2020-11-20 14:25:39.911+0000: initiating migration

2020-11-20T14:26:21.409517Z qemu-system-x86_64: terminating on signal 15
from pid 17395 (/usr/sbin/libvirtd)

2020-11-20 14:26:21.610+0000: shutting down, reason=destroyed


Target node
https://zuul.opendev.org/t/openstack/build/d50877ae15db4022b82f4bb1d1d52cea/log/logs/libvir
t/qemu/instance-0000001a.txt

2020-11-20 14:25:11.589+0000: starting up libvirt version: 6.0.0,
package: 0ubuntu8.4~cloud0 (Openstack Ubuntu Testing Bot <openstack-
testing-bot@xxxxxxxxxx> Tue, 15 Sep 2020 20:36:28 +0000), qemu version:
4.2.1Debian 1:4.2-3ubuntu6.7~cloud0, kernel: 4.15.0-124-generic,
hostname: ubuntu-bionic-ovh-bhs1-0021872194

LC_ALL=C \

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \

HOME=/var/lib/libvirt/qemu/domain-10-instance-0000001a \

XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-10-instance-
0000001a/.local/share \

XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-10-instance-0000001a/.cache
\

XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-10-instance-
0000001a/.config \

QEMU_AUDIO_DRV=none \

/usr/bin/qemu-system-x86_64 \

-name guest=instance-0000001a,debug-threads=on \

-S \

-object
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-10
-instance-0000001a/master-key.aes \

-machine pc-i440fx-4.2,accel=tcg,usb=off,dump-guest-core=off \

-cpu qemu64 \

-m 128 \

-overcommit mem-lock=off \

-smp 1,sockets=1,cores=1,threads=1 \

-uuid 2c468d92-4b19-426a-8c25-16b4624c21a4 \

-smbios 'type=1,manufacturer=OpenStack Foundation,product=OpenStack
Nova,version=22.1.0,serial=2c468d92-4b19-426a-
8c25-16b4624c21a4,uuid=2c468d92-4b19-426a-
8c25-16b4624c21a4,family=Virtual Machine' \

-no-user-config \

-nodefaults \

-chardev socket,id=charmonitor,fd=32,server,nowait \

-mon chardev=charmonitor,id=monitor,mode=control \

-rtc base=utc \

-no-shutdown \

-boot strict=on \

-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \

-blockdev
'{"driver":"file","filename":"/opt/stack/data/nova/instances/_base/61bd5e531ab4c82456aa5300ede7266b3610be79
","node-name":"libvirt-2-storage","cache":{"direct":true,"no-
flush":false},"auto-read-only":true,"discard":"unmap"}' \

-blockdev '{"node-name":"libvirt-2-format","read-
only":true,"cache":{"direct":true,"no-
flush":false},"driver":"raw","file":"libvirt-2-storage"}' \

-blockdev
'{"driver":"file","filename":"/opt/stack/data/nova/instances/2c468d92-4b19
-426a-8c25-16b4624c21a4/disk","node-
name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false
},"auto-read-only":true,"discard":"unmap"}' \

-blockdev '{"node-name":"libvirt-1-format","read-
only":false,"cache":{"direct":true,"no-
flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":"libvirt-2-format"}'
\

-device virtio-blk-
pci,scsi=off,bus=pci.0,addr=0x4,drive=libvirt-1-format,id=virtio-
disk0,bootindex=1,write-cache=on \

-netdev tap,fd=34,id=hostnet0 \

-device virtio-net-
pci,host_mtu=1400,netdev=hostnet0,id=net0,mac=fa:16:3e:43:11:f4,bus=pci.0,addr=0x3
\

-add-fd set=2,fd=36 \

-chardev pty,id=charserial0,logfile=/dev/fdset/2,logappend=on \

-device isa-serial,chardev=charserial0,id=serial0 \

-vnc 0.0.0.0:0 \

-device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \

-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 \

-object rng-random,id=objrng0,filename=/dev/urandom \

-device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 \

-sandbox
on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny
\

-msg timestamp=on

char device redirected to /dev/pts/0 (label charserial0)

2020-11-20 14:25:25.637+0000: initiating migration

2020-11-20 14:25:26.776+0000: shutting down, reason=migrated

2020-11-20T14:25:26.777394Z qemu-system-x86_64: terminating on signal 15
from pid 31113 (/usr/sbin/libvirtd)

2020-11-20 14:25:38.909+0000: starting up libvirt version: 6.0.0,
package: 0ubuntu8.4~cloud0 (Openstack Ubuntu Testing Bot <openstack-
testing-bot@xxxxxxxxxx> Tue, 15 Sep 2020 20:36:28 +0000), qemu version:
4.2.1Debian 1:4.2-3ubuntu6.7~cloud0, kernel: 4.15.0-124-generic,
hostname: ubuntu-bionic-ovh-bhs1-0021872194

LC_ALL=C \

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \

HOME=/var/lib/libvirt/qemu/domain-13-instance-0000001a \

XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-13-instance-
0000001a/.local/share \

XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-13-instance-0000001a/.cache
\

XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-13-instance-
0000001a/.config \

QEMU_AUDIO_DRV=none \

/usr/bin/qemu-system-x86_64 \

-name guest=instance-0000001a,debug-threads=on \

-S \

-object
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-13
-instance-0000001a/master-key.aes \

-machine pc-i440fx-4.2,accel=tcg,usb=off,dump-guest-core=off \

-cpu qemu64,hypervisor=on,lahf-lm=on \

-m 128 \

-overcommit mem-lock=off \

-smp 1,sockets=1,cores=1,threads=1 \

-uuid 2c468d92-4b19-426a-8c25-16b4624c21a4 \

-smbios 'type=1,manufacturer=OpenStack Foundation,product=OpenStack
Nova,version=22.1.0,serial=2c468d92-4b19-426a-
8c25-16b4624c21a4,uuid=2c468d92-4b19-426a-
8c25-16b4624c21a4,family=Virtual Machine' \

-no-user-config \

-nodefaults \

-chardev socket,id=charmonitor,fd=34,server,nowait \

-mon chardev=charmonitor,id=monitor,mode=control \

-rtc base=utc \

-no-shutdown \

-boot strict=on \

-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \

-blockdev
'{"driver":"file","filename":"/opt/stack/data/nova/instances/_base/61bd5e531ab4c82456aa5300ede7266b3610be79
","node-name":"libvirt-2-storage","cache":{"direct":true,"no-
flush":false},"auto-read-only":true,"discard":"unmap"}' \

-blockdev '{"node-name":"libvirt-2-format","read-
only":true,"cache":{"direct":true,"no-
flush":false},"driver":"raw","file":"libvirt-2-storage"}' \

-blockdev
'{"driver":"file","filename":"/opt/stack/data/nova/instances/2c468d92-4b19
-426a-8c25-16b4624c21a4/disk","node-
name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false
},"auto-read-only":true,"discard":"unmap"}' \

-blockdev '{"node-name":"libvirt-1-format","read-
only":false,"cache":{"direct":true,"no-
flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":"libvirt-2-format"}'
\

-device virtio-blk-
pci,scsi=off,bus=pci.0,addr=0x4,drive=libvirt-1-format,id=virtio-
disk0,bootindex=1,write-cache=on \

-netdev tap,fd=36,id=hostnet0 \

-device virtio-net-
pci,host_mtu=1400,netdev=hostnet0,id=net0,mac=fa:16:3e:43:11:f4,bus=pci.0,addr=0x3
\

-add-fd set=2,fd=38 \

-chardev pty,id=charserial0,logfile=/dev/fdset/2,logappend=on \

-device isa-serial,chardev=charserial0,id=serial0 \

-vnc 0.0.0.0:0 \

-device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \

-incoming defer \

-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 \

-object rng-random,id=objrng0,filename=/dev/urandom \

-device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 \

-sandbox
on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny
\

-msg timestamp=on

char device redirected to /dev/pts/1 (label charserial0)

2020-11-20T14:25:40.720757Z qemu-system-x86_64: VQ 0 size 0x80 Guest
index 0xb8 inconsistent with Host index 0xe0: delta 0xffd8

2020-11-20T14:25:40.720785Z qemu-system-x86_64: Failed to load virtio-
blk:virtio

2020-11-20T14:25:40.720790Z qemu-system-x86_64: error while loading
state for instance 0x0 of device '0000:00:04.0/virtio-blk'

2020-11-20T14:25:40.720824Z qemu-system-x86_64: load of migration
failed: Operation not permitted

2020-11-20 14:25:40.778+0000: shutting down, reason=failed



** Also affects: qemu
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1761798

Title:
  live migration intermittently fails in CI with "VQ 0 size 0x80 Guest
  index 0x12c inconsistent with Host index 0x134: delta 0xfff8"

Status in OpenStack Compute (nova):
  Confirmed
Status in QEMU:
  New

Bug description:
  Seen here:

  http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm-
  multinode-live-
  migration/8de6e74/logs/subnode-2/libvirt/qemu/instance-00000002.txt.gz

  2018-04-05T21:48:38.205752Z qemu-system-x86_64: -chardev pty,id=charserial0,logfile=/dev/fdset/1,logappend=on: char device redirected to /dev/pts/0 (label charserial0)
  warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
  2018-04-05T21:48:43.153268Z qemu-system-x86_64: VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8
  2018-04-05T21:48:43.153288Z qemu-system-x86_64: Failed to load virtio-blk:virtio
  2018-04-05T21:48:43.153292Z qemu-system-x86_64: error while loading state for instance 0x0 of device '0000:00:04.0/virtio-blk'
  2018-04-05T21:48:43.153347Z qemu-system-x86_64: load of migration failed: Operation not permitted
  2018-04-05 21:48:43.198+0000: shutting down, reason=crashed

  And in the n-cpu logs on the other host:

  http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm-
  multinode-live-
  migration/8de6e74/logs/screen-n-cpu.txt.gz#_Apr_05_21_48_43_257541

  There is a related Red Hat bug:

  https://bugzilla.redhat.com/show_bug.cgi?id=1450524

  The CI job failures are at present using the Pike UCA:

  ii  libvirt-bin                         3.6.0-1ubuntu6.2~cloud0

  ii  qemu-system-x86                     1:2.10+dfsg-0ubuntu3.5~cloud0

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1761798/+subscriptions
References

[Bug 1761798] [NEW] live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8"
From: Matt Riedemann, 2018-04-06