yahoo-eng-team team mailing list archive

[OpenStack Infra <1926601@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/nova/+/852737
Committed: https://opendev.org/openstack/nova/commit/6eed55bf55469f4ceaa7d4d4eb1be635e14bc73b
Submitter: "Zuul (22348)"
Branch:    master

commit 6eed55bf55469f4ceaa7d4d4eb1be635e14bc73b
Author: Rajesh Tailor <ratailor@xxxxxxxxxx>
Date:   Wed Aug 10 18:15:04 2022 +0530

    Fix rescue volume-based instance
    
    As of now, when attempting to rescue a volume-based instance
    using an image without the hw_rescue_device and/or hw_rescue_bus
    properties set, the rescue api call fails (as non-stable rescue
    for volume-based instances are not supported) leaving the instance
    in error state.
    
    This change checks for hw_rescue_device/hw_rescue_bus image
    properties before attempting to rescue and if the property
    is not set, then fail with proper error message, without changing
    instance state.
    
    Related-Bug: #1978958
    Closes-Bug: #1926601
    Change-Id: Id4c8c5f3b32985ac7d3d7c833b82e0876f7367c1


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1926601

Title:
  Rescuing RBD volume-backed instance does not work

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Context:
  - Openstack Victoria deployment
  - Separate ceph pools for Cinder volumes and Nova VMs.
  - Trying to rescue a volume-backed instance with --image parameter.

  The rescue disk is not even created in the VMs pool, so the instance
  is put into ERROR because of the following exception:

  libvirt.libvirtError: internal error: process exited while connecting
  to monitor: 2021-04-29T11:19:00.527948Z qemu-system-x86_64: -blockdev
  {"driver":"rbd","pool":"vms","image":"f2dfce55-94cb-43e2-b799-9f9a2671c38b_disk","server":[{"host":"10.0.1.81","port":"6789"},{"host":"10.0.1.82","port":"6789"},{"host":"10.0.1.83","port":"6789"}],"user":"nova","auth-
  client-required":["cephx","none"],"key-secret":"libvirt-1-storage-
  secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-
  flush":false},"auto-read-only":true,"discard":"unmap"}: error reading
  header from f2dfce55-94cb-43e2-b799-9f9a2671c38b_disk: No such file or
  directory

  Stacktrace

  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b] Traceback (most recent call last):
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 4178, in rescue_instance
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     self.driver.rescue(context, instance, network_info,
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 3668, in rescue
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     self._create_guest(
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 6637, in _create_guest
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     guest.launch(pause=pause)
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 158, in launch
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     LOG.error('Error launching a defined domain '
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     self.force_reraise()
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     six.reraise(self.type_, self.value, self.tb)
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     raise value
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 155, in launch
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     return self._domain.createWithFlags(flags)
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 190, in doit
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 148, in proxy_call
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     rv = execute(f, *args, **kwargs)
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 129, in execute
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     six.reraise(c, e, tb)
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     raise value
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 83, in tworker
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     rv = meth(*args, **kwargs)
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/usr/lib/python3/dist-packages/libvirt.py", line 1265, in createWithFlags
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b] libvirt.libvirtError: internal error: process exited while connecting to monitor: 2021-04-29T11:19:00.527948Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"vms","image":"f2dfce55-94cb-43e2-b799-9f9a2671c38b_disk","server":[{"host":"10.0.1.81","port":"6789"},{"host":"10.0.1.82","port":"6789"},{"host":"10.0.1.83","port":"6789"}],"user":"nova","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from f2dfce55-94cb-43e2-b799-9f9a2671c38b_disk: No such file or directory
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1926601/+subscriptions