← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #89128

[Bug 1978958] [NEW] Attempting to rescue volume-based instance doesn't return error but leaves VM in error state

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Description
===========

When attempting to rescue a volume-based instance using an image without
the hw_rescue_device and hw_rescue_bus properties set (i.e. which
doesn't support the stable rescue mode) I would expect the API call to
fail leaving the state in its existing state. Instead Nova appears to go
ahead with the rescue which fails (since non-stable rescues for volume
backed instances aren't supported) leading to an unrecoverable, broken
instance.


Steps to reproduce
==================

1. Create an instance booting from a volume (in my case the instance is
booting from a volume in Ceph)

2. Attempt to trigger a rescue using an image *without* the
hw_rescue_device and hw_rescue_bus properties set using:

     $ openstack --os-compute-api-version 2.87 server rescue --image
<IMAGE UUID> <INSTANCE UUID>


Expected result
===============

I would have expected step 2 to fail with an error along the lines of
"Cannot rescue a volume-backed instance", leaving the instance
unchanged.

My crude skimming of the Nova source appears to try and perform this
check here:
https://github.com/openstack/nova/blob/f766db261634c8f95f874ba132159f148de9e8bf/nova/compute/api.py#L4514-L4517.


Actual result
=============

The rescue command returns normally and the instance enters the rescue
state very briefly before ending up in the Error state with the message:

    Instance 05043c69-3533-4396-a610-98e564fa3ead cannot be rescued:
Driver Error: internal error: qemu unexpectedly closed the monitor:
2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev
{"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-43

Looking at the nova-compute logs on the hypervisor gets the following


    2022-06-16 11:53:40.765 32936 INFO nova.virt.libvirt.driver [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Attempting rescue
    2022-06-16 11:53:40.768 32936 INFO nova.virt.libvirt.driver [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Creating image
    2022-06-16 11:53:40.943 32936 WARNING nova.compute.manager [req-a39b605e-e2db-45ce-8f7f-c92a192d9fbe 023555cbd3a2484f89984da17ae9bbfb 0c6d3af6910e429894c203af88fc96c4 - default default] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Received unexpected event network-vif-unplugged-464bd8c6-2b41-4648-abe8-d900c88dd996 for instance with vm_state active a>
    2022-06-16 11:53:50.665 32936 INFO nova.virt.libvirt.driver [-] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Instance destroyed successfully.
    2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] Error launching a defined domain with XML: <domain type='kvm'>
      <name>instance-00015604</name>
      <uuid>05043c69-3533-4396-a610-98e564fa3ead</uuid>
      ...snip...
        <disk type='network' device='disk'>
          <driver name='qemu' type='raw' cache='writeback' discard='unmap'/>
          <auth username='cinder'>
            <secret type='ceph' uuid='...'/>
          </auth>
          <source protocol='rbd' name='ephemeral-vms/05043c69-3533-4396-a610-98e564fa3ead_disk.rescue'>
            <host name='...snip...' port='6789'/>
            <host name='...snip...' port='6789'/>
            <host name='...snip...' port='6789'/>
          </source>
          <target dev='sda' bus='scsi'/>
          <address type='drive' controller='0' bus='0' target='0' unit='0'/>
        </disk>
        <disk type='network' device='disk'>
          <driver name='qemu' type='raw' cache='writeback' discard='unmap'/>
          <auth username='cinder'>
            <secret type='ceph' uuid='...'/>
          </auth>
          <source protocol='rbd' name='ephemeral-vms/05043c69-3533-4396-a610-98e564fa3ead_disk'>
            <host name='...snip...' port='6789'/>
            <host name='...snip...' port='6789'/>
            <host name='...snip...' port='6789'/>
          </source>
          <target dev='sdb' bus='scsi'/>
          <address type='drive' controller='0' bus='0' target='0' unit='1'/>
        </disk>
        ...snip...
    : libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory
    2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest Traceback (most recent call last):
    2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 159, in launch
    2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest     return self._domain.createWithFlags(flags)
    2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 193, in doit
    2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest     result = proxy_call(self._autowrap, f, *args, **kwargs)
    2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 151, in proxy_call
    2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest     rv = execute(f, *args, **kwargs)
    2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 132, in execute
    2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest     six.reraise(c, e, tb)
    2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/six.py", line 719, in reraise
    2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest     raise value
    2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 86, in tworker
    2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest     rv = meth(*args, **kwargs)
    2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/libvirt.py", line 1265, in createWithFlags
    2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
    2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory
    2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest 
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Error trying to Rescue Instance: libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] Traceback (most recent call last):
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 4207, in rescue_instance
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]     self.driver.rescue(context, instance, network_info,
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 4145, in rescue
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]     self._create_guest(
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 7197, in _create_guest
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]     guest.launch(pause=pause)
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 162, in launch
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]     LOG.exception('Error launching a defined domain with XML: %s',
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]     self.force_reraise()
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]     raise self.value
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 159, in launch
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]     return self._domain.createWithFlags(flags)
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 193, in doit
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]     result = proxy_call(self._autowrap, f, *args, **kwargs)
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 151, in proxy_call
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]     rv = execute(f, *args, **kwargs)
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 132, in execute
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]     six.reraise(c, e, tb)
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/six.py", line 719, in reraise
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]     raise value
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 86, in tworker
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]     rv = meth(*args, **kwargs)
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/libvirt.py", line 1265, in createWithFlags
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory
    2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] 
    2022-06-16 11:53:52.310 32936 INFO nova.compute.manager [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Successfully reverted task state from rescuing on failure for instance.
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] Exception during message handling: nova.exception.InstanceNotRescuable: Instance 05043c69-3533-4396-a610-98e564fa3ead cannot be rescued: Driver Error: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 4207, in rescue_instance
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     self.driver.rescue(context, instance, network_info,
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 4145, in rescue
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     self._create_guest(
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 7197, in _create_guest
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     guest.launch(pause=pause)
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 162, in launch
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     LOG.exception('Error launching a defined domain with XML: %s',
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     self.force_reraise()
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     raise self.value
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 159, in launch
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     return self._domain.createWithFlags(flags)
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 193, in doit
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 151, in proxy_call
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 132, in execute
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/six.py", line 719, in reraise
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     raise value
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 86, in tworker
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/libvirt.py", line 1265, in createWithFlags
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server 
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server 
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/exception_wrapper.py", line 71, in wrapped
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     _emit_versioned_exception_notification(
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     self.force_reraise()
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     raise self.value
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/exception_wrapper.py", line 63, in wrapped
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 183, in decorated_function
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     LOG.warning("Failed to revert task state for instance. "
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     self.force_reraise()
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     raise self.value
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 154, in decorated_function
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/utils.py", line 1433, in decorated_function
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 211, in decorated_function
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     compute_utils.add_instance_fault_from_exc(context,
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     self.force_reraise()
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     raise self.value
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 200, in decorated_function
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 4214, in rescue_instance
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     raise exception.InstanceNotRescuable(
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server nova.exception.InstanceNotRescuable: Instance 05043c69-3533-4396-a610-98e564fa3ead cannot be rescued: Driver Error: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory
    2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server 
    2022-06-16 11:53:55.555 32936 INFO nova.compute.manager [-] [instance: 05043c69-3533-4396-a610-98e564fa3ead] VM Stopped (Lifecycle Event)
    2022-06-16 11:53:55.610 32936 INFO nova.compute.manager [req-7790122f-6942-4296-bdfa-37362785baec - - - - -] [instance: 05043c69-3533-4396-a610-98e564fa3ead] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor.
    


Environment
===========

Nova f766db261634c8f95f874ba132159f148de9e8bf (installed from Git by
OSA)

Hypervisor: Libvirt + KVM  (Libvirt 6.0.0-0ubuntu8.16)

Storage type: Ceph

Networking: Neutron (using Linux networking, no OVS)

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1978958

Title:
  Attempting to rescue volume-based instance doesn't return error but
  leaves VM in error state

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========

  When attempting to rescue a volume-based instance using an image
  without the hw_rescue_device and hw_rescue_bus properties set (i.e.
  which doesn't support the stable rescue mode) I would expect the API
  call to fail leaving the state in its existing state. Instead Nova
  appears to go ahead with the rescue which fails (since non-stable
  rescues for volume backed instances aren't supported) leading to an
  unrecoverable, broken instance.

  
  Steps to reproduce
  ==================

  1. Create an instance booting from a volume (in my case the instance
  is booting from a volume in Ceph)

  2. Attempt to trigger a rescue using an image *without* the
  hw_rescue_device and hw_rescue_bus properties set using:

       $ openstack --os-compute-api-version 2.87 server rescue --image
  <IMAGE UUID> <INSTANCE UUID>

  
  Expected result
  ===============

  I would have expected step 2 to fail with an error along the lines of
  "Cannot rescue a volume-backed instance", leaving the instance
  unchanged.

  My crude skimming of the Nova source appears to try and perform this
  check here:
  https://github.com/openstack/nova/blob/f766db261634c8f95f874ba132159f148de9e8bf/nova/compute/api.py#L4514-L4517.

  
  Actual result
  =============

  The rescue command returns normally and the instance enters the rescue
  state very briefly before ending up in the Error state with the
  message:

      Instance 05043c69-3533-4396-a610-98e564fa3ead cannot be rescued:
  Driver Error: internal error: qemu unexpectedly closed the monitor:
  2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev
  {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-43

  Looking at the nova-compute logs on the hypervisor gets the following

  
      2022-06-16 11:53:40.765 32936 INFO nova.virt.libvirt.driver [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Attempting rescue
      2022-06-16 11:53:40.768 32936 INFO nova.virt.libvirt.driver [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Creating image
      2022-06-16 11:53:40.943 32936 WARNING nova.compute.manager [req-a39b605e-e2db-45ce-8f7f-c92a192d9fbe 023555cbd3a2484f89984da17ae9bbfb 0c6d3af6910e429894c203af88fc96c4 - default default] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Received unexpected event network-vif-unplugged-464bd8c6-2b41-4648-abe8-d900c88dd996 for instance with vm_state active a>
      2022-06-16 11:53:50.665 32936 INFO nova.virt.libvirt.driver [-] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Instance destroyed successfully.
      2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] Error launching a defined domain with XML: <domain type='kvm'>
        <name>instance-00015604</name>
        <uuid>05043c69-3533-4396-a610-98e564fa3ead</uuid>
        ...snip...
          <disk type='network' device='disk'>
            <driver name='qemu' type='raw' cache='writeback' discard='unmap'/>
            <auth username='cinder'>
              <secret type='ceph' uuid='...'/>
            </auth>
            <source protocol='rbd' name='ephemeral-vms/05043c69-3533-4396-a610-98e564fa3ead_disk.rescue'>
              <host name='...snip...' port='6789'/>
              <host name='...snip...' port='6789'/>
              <host name='...snip...' port='6789'/>
            </source>
            <target dev='sda' bus='scsi'/>
            <address type='drive' controller='0' bus='0' target='0' unit='0'/>
          </disk>
          <disk type='network' device='disk'>
            <driver name='qemu' type='raw' cache='writeback' discard='unmap'/>
            <auth username='cinder'>
              <secret type='ceph' uuid='...'/>
            </auth>
            <source protocol='rbd' name='ephemeral-vms/05043c69-3533-4396-a610-98e564fa3ead_disk'>
              <host name='...snip...' port='6789'/>
              <host name='...snip...' port='6789'/>
              <host name='...snip...' port='6789'/>
            </source>
            <target dev='sdb' bus='scsi'/>
            <address type='drive' controller='0' bus='0' target='0' unit='1'/>
          </disk>
          ...snip...
      : libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory
      2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest Traceback (most recent call last):
      2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 159, in launch
      2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest     return self._domain.createWithFlags(flags)
      2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 193, in doit
      2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest     result = proxy_call(self._autowrap, f, *args, **kwargs)
      2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 151, in proxy_call
      2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest     rv = execute(f, *args, **kwargs)
      2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 132, in execute
      2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest     six.reraise(c, e, tb)
      2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/six.py", line 719, in reraise
      2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest     raise value
      2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 86, in tworker
      2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest     rv = meth(*args, **kwargs)
      2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/libvirt.py", line 1265, in createWithFlags
      2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
      2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory
      2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest 
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Error trying to Rescue Instance: libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] Traceback (most recent call last):
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 4207, in rescue_instance
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]     self.driver.rescue(context, instance, network_info,
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 4145, in rescue
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]     self._create_guest(
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 7197, in _create_guest
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]     guest.launch(pause=pause)
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 162, in launch
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]     LOG.exception('Error launching a defined domain with XML: %s',
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]     self.force_reraise()
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]     raise self.value
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 159, in launch
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]     return self._domain.createWithFlags(flags)
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 193, in doit
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]     result = proxy_call(self._autowrap, f, *args, **kwargs)
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 151, in proxy_call
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]     rv = execute(f, *args, **kwargs)
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 132, in execute
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]     six.reraise(c, e, tb)
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/six.py", line 719, in reraise
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]     raise value
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 86, in tworker
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]     rv = meth(*args, **kwargs)
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/libvirt.py", line 1265, in createWithFlags
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory
      2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] 
      2022-06-16 11:53:52.310 32936 INFO nova.compute.manager [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Successfully reverted task state from rescuing on failure for instance.
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] Exception during message handling: nova.exception.InstanceNotRescuable: Instance 05043c69-3533-4396-a610-98e564fa3ead cannot be rescued: Driver Error: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 4207, in rescue_instance
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     self.driver.rescue(context, instance, network_info,
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 4145, in rescue
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     self._create_guest(
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 7197, in _create_guest
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     guest.launch(pause=pause)
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 162, in launch
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     LOG.exception('Error launching a defined domain with XML: %s',
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     self.force_reraise()
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     raise self.value
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 159, in launch
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     return self._domain.createWithFlags(flags)
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 193, in doit
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 151, in proxy_call
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 132, in execute
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/six.py", line 719, in reraise
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     raise value
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 86, in tworker
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/libvirt.py", line 1265, in createWithFlags
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server 
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server 
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/exception_wrapper.py", line 71, in wrapped
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     _emit_versioned_exception_notification(
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     self.force_reraise()
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     raise self.value
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/exception_wrapper.py", line 63, in wrapped
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 183, in decorated_function
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     LOG.warning("Failed to revert task state for instance. "
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     self.force_reraise()
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     raise self.value
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 154, in decorated_function
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/utils.py", line 1433, in decorated_function
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 211, in decorated_function
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     compute_utils.add_instance_fault_from_exc(context,
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     self.force_reraise()
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     raise self.value
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 200, in decorated_function
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 4214, in rescue_instance
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server     raise exception.InstanceNotRescuable(
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server nova.exception.InstanceNotRescuable: Instance 05043c69-3533-4396-a610-98e564fa3ead cannot be rescued: Driver Error: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory
      2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server 
      2022-06-16 11:53:55.555 32936 INFO nova.compute.manager [-] [instance: 05043c69-3533-4396-a610-98e564fa3ead] VM Stopped (Lifecycle Event)
      2022-06-16 11:53:55.610 32936 INFO nova.compute.manager [req-7790122f-6942-4296-bdfa-37362785baec - - - - -] [instance: 05043c69-3533-4396-a610-98e564fa3ead] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor.
      


  Environment
  ===========

  Nova f766db261634c8f95f874ba132159f148de9e8bf (installed from Git by
  OSA)

  Hypervisor: Libvirt + KVM  (Libvirt 6.0.0-0ubuntu8.16)

  Storage type: Ceph

  Networking: Neutron (using Linux networking, no OVS)

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1978958/+subscriptions
  Thread Previous • Date Previous • • Thread Next