yahoo-eng-team team mailing list archive

Thread
Date

[Bug 1807352] [NEW] Virtual devices in Windows VMs occasionally loose connection to host

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Pascal <1807352@xxxxxxxxxxxxxxxxxx>
Date: Fri, 07 Dec 2018 08:57:20 -0000
Reply-to: Bug 1807352 <1807352@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx

Public bug reported:

Description
===========
Multiple Windows virtual machine running on our OpenStack cloud seems to occasionally loose the connection from their disk & network virtual device (virtio) to the underlying host. The Windows VM is then no longer able to write down to the disk (see below for logs). Additionally the network connection is shown as "No internet access" (yellow alert symbol). Rebooting the corresponding VM fixes the problems.

Steps to reproduce
==================
We don't know how to reproduce the situation.

Expected result
===============
Windows VMs should run without any virtual device connectivity issues.

Actual result
=============
Virtual devices in Windows VMs occasionally loose connection to host, as described above.

Environment
===========
Host:
* CentOS Linux release 7.5.1804 (Core) / Kernel: 3.10.0-862
* Nova: openstack-nova-compute-15.1.3-1
* Hypervisor: Libvirt + KVM (Libvirt 3.9.0 / QEMU 2.10.0)
* Storage: Ceph 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe)
* Networking: Neutron with OpenVSwitch

VM:
* Windows Server 2016 Standard
* Red Hat VirtIO SCSI Disk Device (Driver version: 10.0.14393.1613)
* Red Hat VirtIO Ethernet Adapter (Driver version: 100.76.104.14900)
* Network: Static IP configuration

Logs & Configs
==============
Compute and storage servers:
No relevant log entries found.

Windows VM:
Level Date and Time Source Event ID Task Category
Warning 05.12.2018 07:54:44 vhdmp 129 None "The description for Event ID 129 from source vhdmp cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had
to be saved with the event.

The following information was included with the event:

\Device\RaidPort2
"
Warning 05.12.2018 07:54:37 Disk 153 None The IO operation at logical block address 0xa968 for Disk 9 (PDO name: \Device\00000f0d) was retried.
Warning 05.12.2018 07:54:33 NTFS 50 None {Delayed Write Failed} Windows was unable to save all the data for the file . The data has been lost. This error may be caused by a failure of your computer hardware or network connection. Please try to save this file elsewhere.
Information 05.12.2018 07:54:33 Application Popup 26 None Application popup: Windows - Delayed Write Failed : Exception Processing Message 0xc0000222 Parameters 0x7ffdf3051d28 0x7ffdf3051d28 0x7ffdf3051d28 0x7ffdf3051d28
Information 05.12.2018 07:54:33 Application Popup 26 None Application popup: Windows - Delayed Write Failed : Exception Processing Message 0xc0000222 Parameters 0x7ffdf3051d28 0x7ffdf3051d28 0x7ffdf3051d28 0x7ffdf3051d28
Warning 05.12.2018 07:54:33 NTFS 50 None {Delayed Write Failed} Windows was unable to save all the data for the file \$Extend\$UsnJrnl:$J:$DATA. The data has been lost. This error may be caused by a failure of your computer hardware or network connection. Please try to save this file elsewhere.
Warning 05.12.2018 07:54:33 NTFS 50 None {Delayed Write Failed} Windows was unable to save all the data for the file \AppData\Local\Microsoft\Windows\WebCache\V01.loglog. The data has been lost. This error may be caused by a failure of your computer hardware or network connection. Please try to save this file elsewhere.
Warning 05.12.2018 07:54:33 NTFS 50 None {Delayed Write Failed} Windows was unable to save all the data for the file \AppData\Local\Microsoft\Windows\WebCache\WebCacheV01.jfm. The data has been lost. This error may be caused by a failure of your computer hardware or network connection. Please try to save this file elsewhere.
Information 05.12.2018 07:54:33 Application Popup 26 None Application popup: Windows - Delayed Write Failed : Exception Processing Message 0xc0000222 Parameters 0x7ffdf3051d28 0x7ffdf3051d28 0x7ffdf3051d28 0x7ffdf3051d28
Warning 05.12.2018 07:54:32 Microsoft-Windows-Ntfs 140 None "The system failed to flush data to the transaction log. Corruption may occur in VolumeId: \\?\Volume{de53bca7-0000-0000-0000-100000000000}, DeviceName: \Device\HarddiskVolume100.
(A device which does not exist was specified.)"
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Microsoft-Windows-Ntfs 140 None "The system failed to flush data to the transaction log. Corruption may occur in VolumeId: \\?\Volume{de53bca7-0000-0000-0000-100000000000}, DeviceName: \Device\HarddiskVolume100.
(A device which does not exist was specified.)"
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Microsoft-Windows-Ntfs 140 None "The system failed to flush data to the transaction log. Corruption may occur in VolumeId: \\?\Volume{de53bca7-0000-0000-0000-100000000000}, DeviceName: \Device\HarddiskVolume100.
(A device which does not exist was specified.)"
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Microsoft-Windows-Ntfs 140 None "The system failed to flush data to the transaction log. Corruption may occur in VolumeId: \\?\Volume{de53bca7-0000-0000-0000-100000000000}, DeviceName: \Device\HarddiskVolume100.
(A device which does not exist was specified.)"
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Microsoft-Windows-Ntfs 140 None "The system failed to flush data to the transaction log. Corruption may occur in VolumeId: \\?\Volume{de53bca7-0000-0000-0000-100000000000}, DeviceName: \Device\HarddiskVolume100.
(A device which does not exist was specified.)"
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Microsoft-Windows-Ntfs 140 None "The system failed to flush data to the transaction log. Corruption may occur in VolumeId: \\?\Volume{de53bca7-0000-0000-0000-100000000000}, DeviceName: \Device\HarddiskVolume100.
(A device which does not exist was specified.)"
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Microsoft-Windows-Ntfs 140 None "The system failed to flush data to the transaction log. Corruption may occur in VolumeId: \\?\Volume{de53bca7-0000-0000-0000-100000000000}, DeviceName: \Device\HarddiskVolume100.
(A device which does not exist was specified.)"
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Microsoft-Windows-Ntfs 140 None "The system failed to flush data to the transaction log. Corruption may occur in VolumeId: \\?\Volume{de53bca7-0000-0000-0000-100000000000}, DeviceName: \Device\HarddiskVolume100.
(A device which does not exist was specified.)"
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Microsoft-Windows-Ntfs 140 None "The system failed to flush data to the transaction log. Corruption may occur in VolumeId: \\?\Volume{de53bca7-0000-0000-0000-100000000000}, DeviceName: \Device\HarddiskVolume100.
(A device which does not exist was specified.)"
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Microsoft-Windows-Ntfs 140 None "The system failed to flush data to the transaction log. Corruption may occur in VolumeId: \\?\Volume{de53bca7-0000-0000-0000-100000000000}, DeviceName: \Device\HarddiskVolume100.
(A device which does not exist was specified.)"
Error 05.12.2018 07:54:32 NTFS 137 (2) The default transaction resource manager on volume \\?\Volume{de53bca7-0000-0000-0000-100000000000} encountered a non-retryable error and could not start. The data contains the error code.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Microsoft-Windows-Ntfs 140 None "The system failed to flush data to the transaction log. Corruption may occur in VolumeId: \\?\Volume{de53bca7-0000-0000-0000-100000000000}, DeviceName: \Device\HarddiskVolume100.
(A device which does not exist was specified.)"
Warning 05.12.2018 07:54:32 Microsoft-Windows-Ntfs 140 None "The system failed to flush data to the transaction log. Corruption may occur in VolumeId: \\?\Volume{de53bca7-0000-0000-0000-100000000000}, DeviceName: \Device\HarddiskVolume100.
(A device which does not exist was specified.)"
Warning 05.12.2018 07:54:32 Microsoft-Windows-Ntfs 140 None "The system failed to flush data to the transaction log. Corruption may occur in VolumeId: \\?\Volume{de53bca7-0000-0000-0000-100000000000}, DeviceName: \Device\HarddiskVolume100.
(A device which does not exist was specified.)"
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 153 None The IO operation at logical block address 0x2dc40 for Disk 6 (PDO name: \Device\00000ea0) was retried.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 51 None An error was detected on device \Device\Harddisk6\DR98 during a paging operation.
Warning 05.12.2018 07:54:32 Disk 153 None The IO operation at logical block address 0xace0 for Disk 6 (PDO name: \Device\00000ea0) was retried.
Warning 05.12.2018 07:54:32 Disk 153 None The IO operation at logical block address 0x32ac0 for Disk 6 (PDO name: \Device\00000ea0) was retried.

** Affects: nova
Importance: Undecided
Status: New

** Tags: devices virtio windows

--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1807352

Title:
Virtual devices in Windows VMs occasionally loose connection to host

Status in OpenStack Compute (nova):
New

Bug description:
Description
===========
Multiple Windows virtual machine running on our OpenStack cloud seems to occasionally loose the connection from their disk & network virtual device (virtio) to the underlying host. The Windows VM is then no longer able to write down to the disk (see below for logs). Additionally the network connection is shown as "No internet access" (yellow alert symbol). Rebooting the corresponding VM fixes the problems.

Steps to reproduce
==================
We don't know how to reproduce the situation.

Expected result
===============
Windows VMs should run without any virtual device connectivity issues.

Actual result
=============
Virtual devices in Windows VMs occasionally loose connection to host, as described above.

Logs & Configs
==============
Compute and storage servers:
No relevant log entries found.

If the event originated on another computer, the display information
had to be saved with the event.

The following information was included with the event:

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1807352/+subscriptions

Follow ups

[Bug 1807352] Re: Virtual devices in Windows VMs occasionally loose connection to host
From: Launchpad Bug Tracker, 2020-07-05