yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1926601] [NEW] Rescuing RBD volume-backed instance does not work

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Marius L <1926601@xxxxxxxxxxxxxxxxxx>
Date: Thu, 29 Apr 2021 11:26:06 -0000
Reply-to: Bug 1926601 <1926601@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Context:
- Openstack Victoria deployment
- Separate ceph pools for Cinder volumes and Nova VMs.
- Trying to rescue a volume-backed instance with --image parameter.

The rescue disk is not even created in the VMs pool, so the instance is
put into ERROR because of the following exception:

libvirt.libvirtError: internal error: process exited while connecting to
monitor: 2021-04-29T11:19:00.527948Z qemu-system-x86_64: -blockdev
{"driver":"rbd","pool":"vms","image":"f2dfce55-94cb-
43e2-b799-9f9a2671c38b_disk","server":[{"host":"10.0.1.81","port":"6789"},{"host":"10.0.1.82","port":"6789"},{"host":"10.0.1.83","port":"6789"}],"user":"nova
","auth-client-required":["cephx","none"],"key-secret":"libvirt-1
-storage-secret0","node-
name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false
},"auto-read-only":true,"discard":"unmap"}: error reading header from
f2dfce55-94cb-43e2-b799-9f9a2671c38b_disk: No such file or directory

Stacktrace

2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b] Traceback (most recent call last):
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 4178, in rescue_instance
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     self.driver.rescue(context, instance, network_info,
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 3668, in rescue
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     self._create_guest(
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 6637, in _create_guest
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     guest.launch(pause=pause)
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 158, in launch
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     LOG.error('Error launching a defined domain '
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     self.force_reraise()
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     six.reraise(self.type_, self.value, self.tb)
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     raise value
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 155, in launch
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     return self._domain.createWithFlags(flags)
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 190, in doit
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 148, in proxy_call
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     rv = execute(f, *args, **kwargs)
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 129, in execute
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     six.reraise(c, e, tb)
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     raise value
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 83, in tworker
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     rv = meth(*args, **kwargs)
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/usr/lib/python3/dist-packages/libvirt.py", line 1265, in createWithFlags
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b] libvirt.libvirtError: internal error: process exited while connecting to monitor: 2021-04-29T11:19:00.527948Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"vms","image":"f2dfce55-94cb-43e2-b799-9f9a2671c38b_disk","server":[{"host":"10.0.1.81","port":"6789"},{"host":"10.0.1.82","port":"6789"},{"host":"10.0.1.83","port":"6789"}],"user":"nova","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from f2dfce55-94cb-43e2-b799-9f9a2671c38b_disk: No such file or directory
2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1926601

Title:
  Rescuing RBD volume-backed instance does not work

Status in OpenStack Compute (nova):
  New

Bug description:
  Context:
  - Openstack Victoria deployment
  - Separate ceph pools for Cinder volumes and Nova VMs.
  - Trying to rescue a volume-backed instance with --image parameter.

  The rescue disk is not even created in the VMs pool, so the instance
  is put into ERROR because of the following exception:

  libvirt.libvirtError: internal error: process exited while connecting
  to monitor: 2021-04-29T11:19:00.527948Z qemu-system-x86_64: -blockdev
  {"driver":"rbd","pool":"vms","image":"f2dfce55-94cb-
  43e2-b799-9f9a2671c38b_disk","server":[{"host":"10.0.1.81","port":"6789"},{"host":"10.0.1.82","port":"6789"},{"host":"10.0.1.83","port":"6789"}],"user":"nova
  ","auth-client-required":["cephx","none"],"key-secret":"libvirt-1
  -storage-secret0","node-
  name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false
  },"auto-read-only":true,"discard":"unmap"}: error reading header from
  f2dfce55-94cb-43e2-b799-9f9a2671c38b_disk: No such file or directory

  Stacktrace

  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b] Traceback (most recent call last):
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 4178, in rescue_instance
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     self.driver.rescue(context, instance, network_info,
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 3668, in rescue
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     self._create_guest(
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 6637, in _create_guest
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     guest.launch(pause=pause)
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 158, in launch
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     LOG.error('Error launching a defined domain '
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     self.force_reraise()
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     six.reraise(self.type_, self.value, self.tb)
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     raise value
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 155, in launch
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     return self._domain.createWithFlags(flags)
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 190, in doit
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 148, in proxy_call
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     rv = execute(f, *args, **kwargs)
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 129, in execute
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     six.reraise(c, e, tb)
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     raise value
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 83, in tworker
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     rv = meth(*args, **kwargs)
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]   File "/usr/lib/python3/dist-packages/libvirt.py", line 1265, in createWithFlags
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b] libvirt.libvirtError: internal error: process exited while connecting to monitor: 2021-04-29T11:19:00.527948Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"vms","image":"f2dfce55-94cb-43e2-b799-9f9a2671c38b_disk","server":[{"host":"10.0.1.81","port":"6789"},{"host":"10.0.1.82","port":"6789"},{"host":"10.0.1.83","port":"6789"}],"user":"nova","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from f2dfce55-94cb-43e2-b799-9f9a2671c38b_disk: No such file or directory
  2021-04-29 11:19:00.774 6 ERROR nova.compute.manager [instance: f2dfce55-94cb-43e2-b799-9f9a2671c38b]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1926601/+subscriptions
Follow ups

[Bug 1926601] Re: Rescuing RBD volume-backed instance does not work
From: OpenStack Infra, 2023-01-30