← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 2110738] [NEW] Stable rescue fails when necessary image properties not set

 

Public bug reported:

>From https://issues.redhat.com/browse/OSPRH-13142:

Description of problem:

For a boot-from-volume instances, 'openstack server rescue <vm> --image
<image>' fails with the following issues:

1.  It attempts to attach two disks: <instance_uuid>_disk &
<instance_uuid>_disk.rescue.  Only, <instance_uuid>_disk.rescue is
created so it fails with the following error:

2024-01-23 16:32:14.338 2 ERROR oslo_messaging.rpc.server
nova.exception.InstanceNotRescuable: Instance
dc0812ba-b4ca-4ffa-a7e5-2157e52f35d2 cannot be rescued: Driver Error:
internal error: process exited while connecting to monitor:
2024-01-23T16:32:13.017966Z qemu-kvm: -blockdev
{"driver":"rbd","pool":"vms","image":"dc0812ba-b4ca-4ffa-a7e5-2157e52f35d2_disk","server":[{"host":"172.16.1.100","port":"6789"}],"user":"openstack","auth-
client-required":["cephx","none"],"key-secret":"libvirt-1-storage-auth-
secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-
flush":false},"auto-read-only":true,"discard":"unmap"}: error reading
header from dc0812ba-b4ca-4ffa-a7e5-2157e52f35d2_disk: No such file or
directory

If you look in ceph, only the .rescue image exists.

# rbd --id openstack -p vms ls -l
NAME                                              SIZE    PARENT  FMT  PROT  LOCK
dc0812ba-b4ca-4ffa-a7e5-2157e52f35d2_disk.rescue  10 GiB            2        excl

However we see the instance configured with both disks.


# virsh domblklist instance-00000003
 Target   Source
----------------------------------------------------------------
 vda      vms/dc0812ba-b4ca-4ffa-a7e5-2157e52f35d2_disk.rescue
 vdb      vms/dc0812ba-b4ca-4ffa-a7e5-2157e52f35d2_disk

If I manually copy, the UUID_disk.rescue to UUID_disk, the instance will
boot into RESCUE mode.  It seems the UUID_disk volume is not needed and
should not be configured in this RESCUE situation.

2.  The RESCUED instance doesn't attach the cinder root volume.  The
cinder root also doesnt re-attach after "unrescuing" the instance.

Reproducer:

$ openstack volume create --size 10 --image rhel8 rootvol1

$ openstack volume list
+--------------------------------------+----------+-----------+------+-------------+
| ID                                   | Name     | Status    | Size | Attached to |
+--------------------------------------+----------+-----------+------+-------------+
| f855dfe6-ad5a-4497-87ff-16ac5856f596 | rootvol1 | available |   10 |             |
+--------------------------------------+----------+-----------+------+-------------+


$ openstack server create --key-name default --flavor rhel --volume rootvol1 --network external test1

$ openstack server show test1 -c status -c image -c volumes_attached
+------------------+--------------------------------------------------------------------------+
| Field            | Value                                                                    |
+------------------+--------------------------------------------------------------------------+
| image            | N/A (booted from volume)                                                 |
| status           | ACTIVE                                                                   |
| volumes_attached | delete_on_termination='False', id='f855dfe6-ad5a-4497-87ff-16ac5856f596' |
+------------------+--------------------------------------------------------------------------+


$ openstack server rescue test1 --image rhel8


$ openstack server show test1 -c status -c image -c volumes_attached -c fault --fit
+------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| Field            | Value                                                                                                                                     |
+------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| fault            | {'code': 400, 'created': '2024-01-23T20:12:17Z', 'message': 'Instance ac3d46c0-c8d5-45df-bd17-d467baaa5a98 cannot be rescued: Driver      |
|                  | Error: internal error: process exited while connecting to monitor: 2024-01-23T20:12:17.612453Z qemu-kvm: -blockdev                        |
|                  | {"driver":"rbd","pool":"vms","image":"ac3d46c0-c8d5-45df-bd17-d467ba'}                                                                    |
| image            | N/A (booted from volume)                                                                                                                  |
| status           | ERROR                                                                                                                                     |
| volumes_attached | delete_on_termination='False', id='f855dfe6-ad5a-4497-87ff-16ac5856f596'                                                                  |
+------------------+-------------------------------------------------------------------------------------------------------------------------------------------+


# virsh domblklist instance-00000004
 Target   Source
----------------------------------------------------------------
 vda      vms/ac3d46c0-c8d5-45df-bd17-d467baaa5a98_disk.rescue
 vdb      vms/ac3d46c0-c8d5-45df-bd17-d467baaa5a98_disk


# rbd --id openstack -p vms ls -l
NAME                                              SIZE    PARENT  FMT  PROT  LOCK
ac3d46c0-c8d5-45df-bd17-d467baaa5a98_disk.rescue  10 GiB            2     

NOTE: here if manually create the _disk volume, the instance will boot
into rescue mode; however, the cinder volume is not attached.

# rbd --id openstack cp vms/ac3d46c0-c8d5-45df-bd17-d467baaa5a98_disk.rescue vms/ac3d46c0-c8d5-45df-bd17-d467baaa5a98_disk
Image copy: 100% complete...done.

RESCUE now completes and the instance is accessible (without cinder root
vol attached).

$ openstack server show test1 -c status -c image -c volumes_attached -c fault --fit
+------------------+--------------------------------------------------------------------------+
| Field            | Value                                                                    |
+------------------+--------------------------------------------------------------------------+
| image            | N/A (booted from volume)                                                 |
| status           | RESCUE                                                                   |
| volumes_attached | delete_on_termination='False', id='f855dfe6-ad5a-4497-87ff-16ac5856f596' |
+------------------+--------------------------------------------------------------------------+

volume still shows in-use

$ openstack volume list
+--------------------------------------+----------+--------+------+--------------------------------+
| ID                                   | Name     | Status | Size | Attached to                    |
+--------------------------------------+----------+--------+------+--------------------------------+
| f855dfe6-ad5a-4497-87ff-16ac5856f596 | rootvol1 | in-use |   10 | Attached to test1 on /dev/vda  |
+--------------------------------------+----------+--------+------+--------------------------------+

But not attached.

# virsh domblklist instance-00000004
 Target   Source
----------------------------------------------------------------
 vda      vms/ac3d46c0-c8d5-45df-bd17-d467baaa5a98_disk.rescue
 vdb      vms/ac3d46c0-c8d5-45df-bd17-d467baaa5a98_disk


The other ugly thing, the unrescue does not revert this back to original disk config.

$ openstack server unrescue test1
$ openstack server show test1 -c status -c image -c volumes_attached -c fault --fit
+------------------+--------------------------------------------------------------------------+
| Field            | Value                                                                    |
+------------------+--------------------------------------------------------------------------+
| image            | N/A (booted from volume)                                                 |
| status           | ACTIVE                                                                   |
| volumes_attached | delete_on_termination='False', id='f855dfe6-ad5a-4497-87ff-16ac5856f596' |
+------------------+--------------------------------------------------------------------------+

The above looks good, but the instance is still booted on rescue disks.

# virsh domblklist instance-00000004
 Target   Source
----------------------------------------------------------------
 vda      vms/ac3d46c0-c8d5-45df-bd17-d467baaa5a98_disk.rescue
 vdb      vms/ac3d46c0-c8d5-45df-bd17-d467baaa5a98_disk

A hard reboot will fix it:

$ openstack server reboot --hard test1

Now the instance is back to boot from vol:

# virsh domblklist instance-00000004
 Target   Source
---------------------------------------------------------------
 vda      volumes/volume-f855dfe6-ad5a-4497-87ff-16ac5856f596


Version-Release number of selected component (if applicable):
Wallaby

How reproducible:
100%

Steps to Reproduce:
1. See above
2.
3.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2110738

Title:
  Stable rescue fails when necessary image properties not set

Status in OpenStack Compute (nova):
  New

Bug description:
  From https://issues.redhat.com/browse/OSPRH-13142:

  Description of problem:

  For a boot-from-volume instances, 'openstack server rescue <vm>
  --image <image>' fails with the following issues:

  1.  It attempts to attach two disks: <instance_uuid>_disk &
  <instance_uuid>_disk.rescue.  Only, <instance_uuid>_disk.rescue is
  created so it fails with the following error:

  2024-01-23 16:32:14.338 2 ERROR oslo_messaging.rpc.server
  nova.exception.InstanceNotRescuable: Instance
  dc0812ba-b4ca-4ffa-a7e5-2157e52f35d2 cannot be rescued: Driver Error:
  internal error: process exited while connecting to monitor:
  2024-01-23T16:32:13.017966Z qemu-kvm: -blockdev
  {"driver":"rbd","pool":"vms","image":"dc0812ba-b4ca-4ffa-a7e5-2157e52f35d2_disk","server":[{"host":"172.16.1.100","port":"6789"}],"user":"openstack","auth-
  client-required":["cephx","none"],"key-secret":"libvirt-1-storage-
  auth-secret0","node-
  name":"libvirt-1-storage","cache":{"direct":false,"no-
  flush":false},"auto-read-only":true,"discard":"unmap"}: error reading
  header from dc0812ba-b4ca-4ffa-a7e5-2157e52f35d2_disk: No such file or
  directory

  If you look in ceph, only the .rescue image exists.

  # rbd --id openstack -p vms ls -l
  NAME                                              SIZE    PARENT  FMT  PROT  LOCK
  dc0812ba-b4ca-4ffa-a7e5-2157e52f35d2_disk.rescue  10 GiB            2        excl

  However we see the instance configured with both disks.

  
  # virsh domblklist instance-00000003
   Target   Source
  ----------------------------------------------------------------
   vda      vms/dc0812ba-b4ca-4ffa-a7e5-2157e52f35d2_disk.rescue
   vdb      vms/dc0812ba-b4ca-4ffa-a7e5-2157e52f35d2_disk

  If I manually copy, the UUID_disk.rescue to UUID_disk, the instance
  will boot into RESCUE mode.  It seems the UUID_disk volume is not
  needed and should not be configured in this RESCUE situation.

  2.  The RESCUED instance doesn't attach the cinder root volume.  The
  cinder root also doesnt re-attach after "unrescuing" the instance.

  Reproducer:

  $ openstack volume create --size 10 --image rhel8 rootvol1

  $ openstack volume list
  +--------------------------------------+----------+-----------+------+-------------+
  | ID                                   | Name     | Status    | Size | Attached to |
  +--------------------------------------+----------+-----------+------+-------------+
  | f855dfe6-ad5a-4497-87ff-16ac5856f596 | rootvol1 | available |   10 |             |
  +--------------------------------------+----------+-----------+------+-------------+

  
  $ openstack server create --key-name default --flavor rhel --volume rootvol1 --network external test1

  $ openstack server show test1 -c status -c image -c volumes_attached
  +------------------+--------------------------------------------------------------------------+
  | Field            | Value                                                                    |
  +------------------+--------------------------------------------------------------------------+
  | image            | N/A (booted from volume)                                                 |
  | status           | ACTIVE                                                                   |
  | volumes_attached | delete_on_termination='False', id='f855dfe6-ad5a-4497-87ff-16ac5856f596' |
  +------------------+--------------------------------------------------------------------------+

  
  $ openstack server rescue test1 --image rhel8

  
  $ openstack server show test1 -c status -c image -c volumes_attached -c fault --fit
  +------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
  | Field            | Value                                                                                                                                     |
  +------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
  | fault            | {'code': 400, 'created': '2024-01-23T20:12:17Z', 'message': 'Instance ac3d46c0-c8d5-45df-bd17-d467baaa5a98 cannot be rescued: Driver      |
  |                  | Error: internal error: process exited while connecting to monitor: 2024-01-23T20:12:17.612453Z qemu-kvm: -blockdev                        |
  |                  | {"driver":"rbd","pool":"vms","image":"ac3d46c0-c8d5-45df-bd17-d467ba'}                                                                    |
  | image            | N/A (booted from volume)                                                                                                                  |
  | status           | ERROR                                                                                                                                     |
  | volumes_attached | delete_on_termination='False', id='f855dfe6-ad5a-4497-87ff-16ac5856f596'                                                                  |
  +------------------+-------------------------------------------------------------------------------------------------------------------------------------------+

  
  # virsh domblklist instance-00000004
   Target   Source
  ----------------------------------------------------------------
   vda      vms/ac3d46c0-c8d5-45df-bd17-d467baaa5a98_disk.rescue
   vdb      vms/ac3d46c0-c8d5-45df-bd17-d467baaa5a98_disk

  
  # rbd --id openstack -p vms ls -l
  NAME                                              SIZE    PARENT  FMT  PROT  LOCK
  ac3d46c0-c8d5-45df-bd17-d467baaa5a98_disk.rescue  10 GiB            2     

  NOTE: here if manually create the _disk volume, the instance will boot
  into rescue mode; however, the cinder volume is not attached.

  # rbd --id openstack cp vms/ac3d46c0-c8d5-45df-bd17-d467baaa5a98_disk.rescue vms/ac3d46c0-c8d5-45df-bd17-d467baaa5a98_disk
  Image copy: 100% complete...done.

  RESCUE now completes and the instance is accessible (without cinder
  root vol attached).

  $ openstack server show test1 -c status -c image -c volumes_attached -c fault --fit
  +------------------+--------------------------------------------------------------------------+
  | Field            | Value                                                                    |
  +------------------+--------------------------------------------------------------------------+
  | image            | N/A (booted from volume)                                                 |
  | status           | RESCUE                                                                   |
  | volumes_attached | delete_on_termination='False', id='f855dfe6-ad5a-4497-87ff-16ac5856f596' |
  +------------------+--------------------------------------------------------------------------+

  volume still shows in-use

  $ openstack volume list
  +--------------------------------------+----------+--------+------+--------------------------------+
  | ID                                   | Name     | Status | Size | Attached to                    |
  +--------------------------------------+----------+--------+------+--------------------------------+
  | f855dfe6-ad5a-4497-87ff-16ac5856f596 | rootvol1 | in-use |   10 | Attached to test1 on /dev/vda  |
  +--------------------------------------+----------+--------+------+--------------------------------+

  But not attached.

  # virsh domblklist instance-00000004
   Target   Source
  ----------------------------------------------------------------
   vda      vms/ac3d46c0-c8d5-45df-bd17-d467baaa5a98_disk.rescue
   vdb      vms/ac3d46c0-c8d5-45df-bd17-d467baaa5a98_disk

  
  The other ugly thing, the unrescue does not revert this back to original disk config.

  $ openstack server unrescue test1
  $ openstack server show test1 -c status -c image -c volumes_attached -c fault --fit
  +------------------+--------------------------------------------------------------------------+
  | Field            | Value                                                                    |
  +------------------+--------------------------------------------------------------------------+
  | image            | N/A (booted from volume)                                                 |
  | status           | ACTIVE                                                                   |
  | volumes_attached | delete_on_termination='False', id='f855dfe6-ad5a-4497-87ff-16ac5856f596' |
  +------------------+--------------------------------------------------------------------------+

  The above looks good, but the instance is still booted on rescue
  disks.

  # virsh domblklist instance-00000004
   Target   Source
  ----------------------------------------------------------------
   vda      vms/ac3d46c0-c8d5-45df-bd17-d467baaa5a98_disk.rescue
   vdb      vms/ac3d46c0-c8d5-45df-bd17-d467baaa5a98_disk

  A hard reboot will fix it:

  $ openstack server reboot --hard test1

  Now the instance is back to boot from vol:

  # virsh domblklist instance-00000004
   Target   Source
  ---------------------------------------------------------------
   vda      volumes/volume-f855dfe6-ad5a-4497-87ff-16ac5856f596

  
  Version-Release number of selected component (if applicable):
  Wallaby

  How reproducible:
  100%

  Steps to Reproduce:
  1. See above
  2.
  3.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2110738/+subscriptions