yahoo-eng-team team mailing list archive

[Dan Smith <1914664@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Seeing this failure in the gate:

https://zuul.opendev.org/t/openstack/build/7c71502b04fe47039b87f76fbe04fe56/log/controller/logs/screen-n-cpu.txt#33096


Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [req-77f51485-cbc2-4f2a-8a6d-4a8ed910e585 req-a221d4f9-401e-420a-911e-8d32536a1d23 service nova] [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] detaching network adapter failed.: libvirt.libvirtError: internal error: End of file from qemu monitor

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431] Traceback (most recent call last):

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]   File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 2210, in
detach_interface

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]     wait_for_detach =
guest.detach_device_with_retry(

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]   File
"/opt/stack/nova/nova/virt/libvirt/guest.py", line 423, in
detach_device_with_retry

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]     _try_detach_device(conf, persistent,
live)

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]   File
"/opt/stack/nova/nova/virt/libvirt/guest.py", line 412, in
_try_detach_device

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]     ctx.reraise = True

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]   File "/usr/local/lib/python3.8/dist-
packages/oslo_utils/excutils.py", line 220, in __exit__

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]     self.force_reraise()

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]   File "/usr/local/lib/python3.8/dist-
packages/oslo_utils/excutils.py", line 196, in force_reraise

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]     six.reraise(self.type_, self.value,
self.tb)

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]   File "/usr/local/lib/python3.8/dist-
packages/six.py", line 703, in reraise

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]     raise value

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]   File
"/opt/stack/nova/nova/virt/libvirt/guest.py", line 398, in
_try_detach_device

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]     self.detach_device(conf,
persistent=persistent, live=live)

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]   File
"/opt/stack/nova/nova/virt/libvirt/guest.py", line 473, in detach_device

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]
self._domain.detachDeviceFlags(device_xml, flags=flags)

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]   File "/usr/local/lib/python3.8/dist-
packages/eventlet/tpool.py", line 190, in doit

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]     result = proxy_call(self._autowrap, f,
*args, **kwargs)

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]   File "/usr/local/lib/python3.8/dist-
packages/eventlet/tpool.py", line 148, in proxy_call

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]     rv = execute(f, *args, **kwargs)

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]   File "/usr/local/lib/python3.8/dist-
packages/eventlet/tpool.py", line 129, in execute

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]     six.reraise(c, e, tb)

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]   File "/usr/local/lib/python3.8/dist-
packages/six.py", line 703, in reraise

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]     raise value

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]   File "/usr/local/lib/python3.8/dist-
packages/eventlet/tpool.py", line 83, in tworker

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]     rv = meth(*args, **kwargs)

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]   File "/usr/local/lib/python3.8/dist-
packages/libvirt.py", line 1515, in detachDeviceFlags

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]     raise
libvirtError('virDomainDetachDeviceFlags() failed')

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431] libvirt.libvirtError: internal error: End
of file from qemu monitor

Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-
compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-
8cf4-46c7-9498-2c5dbc452431]

This was in a cinder-glance-multistore test, but I'm not sure it has
anything to do with those. That test backs glance with cinder, but
otherwise I wouldn't think nova would notice unless it was a failure to
snapshot, use a volume, etc.

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1914664

Title:
  QEMU monitor read failure in ServerStableDeviceRescueTest

Status in OpenStack Compute (nova):
  New

Bug description:
  Seeing this failure in the gate:

  https://zuul.opendev.org/t/openstack/build/7c71502b04fe47039b87f76fbe04fe56/log/controller/logs/screen-n-cpu.txt#33096

  
  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [req-77f51485-cbc2-4f2a-8a6d-4a8ed910e585 req-a221d4f9-401e-420a-911e-8d32536a1d23 service nova] [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] detaching network adapter failed.: libvirt.libvirtError: internal error: End of file from qemu monitor

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431] Traceback (most recent call
  last):

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]   File
  "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2210, in
  detach_interface

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]     wait_for_detach =
  guest.detach_device_with_retry(

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]   File
  "/opt/stack/nova/nova/virt/libvirt/guest.py", line 423, in
  detach_device_with_retry

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]     _try_detach_device(conf,
  persistent, live)

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]   File
  "/opt/stack/nova/nova/virt/libvirt/guest.py", line 412, in
  _try_detach_device

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]     ctx.reraise = True

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]   File "/usr/local/lib/python3.8
  /dist-packages/oslo_utils/excutils.py", line 220, in __exit__

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]     self.force_reraise()

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]   File "/usr/local/lib/python3.8
  /dist-packages/oslo_utils/excutils.py", line 196, in force_reraise

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]     six.reraise(self.type_,
  self.value, self.tb)

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]   File "/usr/local/lib/python3.8
  /dist-packages/six.py", line 703, in reraise

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]     raise value

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]   File
  "/opt/stack/nova/nova/virt/libvirt/guest.py", line 398, in
  _try_detach_device

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]     self.detach_device(conf,
  persistent=persistent, live=live)

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]   File
  "/opt/stack/nova/nova/virt/libvirt/guest.py", line 473, in
  detach_device

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]
  self._domain.detachDeviceFlags(device_xml, flags=flags)

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]   File "/usr/local/lib/python3.8
  /dist-packages/eventlet/tpool.py", line 190, in doit

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]     result =
  proxy_call(self._autowrap, f, *args, **kwargs)

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]   File "/usr/local/lib/python3.8
  /dist-packages/eventlet/tpool.py", line 148, in proxy_call

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]     rv = execute(f, *args,
  **kwargs)

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]   File "/usr/local/lib/python3.8
  /dist-packages/eventlet/tpool.py", line 129, in execute

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]     six.reraise(c, e, tb)

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]   File "/usr/local/lib/python3.8
  /dist-packages/six.py", line 703, in reraise

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]     raise value

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]   File "/usr/local/lib/python3.8
  /dist-packages/eventlet/tpool.py", line 83, in tworker

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]     rv = meth(*args, **kwargs)

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]   File "/usr/local/lib/python3.8
  /dist-packages/libvirt.py", line 1515, in detachDeviceFlags

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]     raise
  libvirtError('virDomainDetachDeviceFlags() failed')

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431] libvirt.libvirtError: internal
  error: End of file from qemu monitor

  Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642
  nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance:
  7174e97c-8cf4-46c7-9498-2c5dbc452431]

  This was in a cinder-glance-multistore test, but I'm not sure it has
  anything to do with those. That test backs glance with cinder, but
  otherwise I wouldn't think nova would notice unless it was a failure
  to snapshot, use a volume, etc.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1914664/+subscriptions