yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #68636
[Bug 1724573] [NEW] When using resume_guests_state_on_host_boot encrypted volumes are directly attached to instances after a host reboot
Public bug reported:
Description
===========
When using resume_guests_state_on_host_boot encrypted volumes are
directly attached to instances after a host reboot. These volumes should
be decrypted by the os-brick encryptors that provide libvirt with
decrypted dm devices for use by the instance/domain.
This is due to the following encryptor.attach_volume call being skipped
in the _hard_reboot where reboot=True as it is assumed the dm devices
are already present on the host:
5204 def _create_domain_and_network(self, context, xml, instance, network_info,
5205 block_device_info=None,
5206 power_on=True, reboot=False,
5207 vifs_already_plugged=False,
5208 post_xml_callback=None,
5209 destroy_disks_on_failure=False):
[..]
5218 if (not reboot and 'data' in connection_info and
5219 'volume_id' in connection_info['data']):
5220 volume_id = connection_info['data']['volume_id']
5221 encryption = encryptors.get_encryption_metadata(
5222 context, self._volume_api, volume_id, connection_info)
5223
5224 if encryption:
5225 encryptor = self._get_volume_encryptor(connection_info,
5226 encryption)
5227 encryptor.attach_volume(context, **encryption)
Steps to reproduce
==================
- Ensure resume_guests_state_on_host_boot is set to True within the
n-cpu config:
$ grep resume_guests_state_on_host_boot /etc/nova/nova-cpu.conf
resume_guests_state_on_host_boot = True
- Create an instance with an attached encrypted volume:
$ cinder type-create LUKS
$ cinder encryption-type-create --cipher aes-xts-plain64 --key_size 512 --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor
$ cinder create --display-name 'encrypted volume' --volume-type LUKS 1
$ nova boot --image cirros-0.3.5-x86_64-disk --flavor 1 test
$ nova volume-attach c762ef8d-13ab-4aee-bd20-c6a002bdd172 3f2cfdf2-11d7-4ac7-883a-76217136f751
- Before continuing note that the instance is connected to the decrypted
dm device:
$ sudo virsh domblklist c762ef8d-13ab-4aee-bd20-c6a002bdd172
Target Source
------------------------------------------------
vda /opt/stack/data/nova/instances/c762ef8d-13ab-4aee-bd20-c6a002bdd172/disk
vdb /dev/disk/by-id/scsi-360014054c6bbc8645494397ad372e0e6
$ ll /dev/disk/by-id/scsi-360014054c6bbc8645494397ad372e0e6
lrwxrwxrwx. 1 root root 56 Oct 18 08:28 /dev/disk/by-id/scsi-360014054c6bbc8645494397ad372e0e6 -> /dev/mapper/crypt-scsi-360014054c6bbc8645494397ad372e0e6
- Restart the n-cpu host _or_ fake a host reset by stopping the n-cpu
service, destroying the domain, removing the decrypted dm device,
unlinking the volume path before finally restarting n-cpu:
$ sudo systemctl stop devstack@n-cpu
$ sudo virsh destroy c762ef8d-13ab-4aee-bd20-c6a002bdd172
$ sudo cryptsetup luksClose /dev/disk/by-id/scsi-360014054c6bbc8645494397ad372e0e6
$ sudo unlink /dev/disk/by-id/scsi-360014054c6bbc8645494397ad372e0e6
$ sudo systemctl start devstack@n-cpu
- The instance is restarted but now points at the original encrypted
block device:
$ sudo virsh domblklist c762ef8d-13ab-4aee-bd20-c6a002bdd172
Target Source
------------------------------------------------
vda /opt/stack/data/nova/instances/c762ef8d-13ab-4aee-bd20-c6a002bdd172/disk
vdb /dev/disk/by-id/scsi-360014054c6bbc8645494397ad372e0e6
$ ll /dev/disk/by-id/scsi-360014054c6bbc8645494397ad372e0e6
lrwxrwxrwx. 1 root root 9 Oct 18 08:32 /dev/disk/by-id/scsi-360014054c6bbc8645494397ad372e0e6 -> ../../sde
- Additional stop and start requests will not correct this:
$ nova stop c762ef8d-13ab-4aee-bd20-c6a002bdd172
$ nova start c762ef8d-13ab-4aee-bd20-c6a002bdd172
$ sudo virsh domblklist c762ef8d-13ab-4aee-bd20-c6a002bdd172
Target Source
------------------------------------------------
vda /opt/stack/data/nova/instances/c762ef8d-13ab-4aee-bd20-c6a002bdd172/disk
vdb /dev/disk/by-id/scsi-360014054c6bbc8645494397ad372e0e6
$ ll /dev/disk/by-id/scsi-360014054c6bbc8645494397ad372e0e6
lrwxrwxrwx. 1 root root 9 Oct 18 08:32 /dev/disk/by-id/scsi-360014054c6bbc8645494397ad372e0e6 -> ../../sde
Expected result
===============
The decrypted volume is attached to the instance once it is restarted.
Actual result
=============
The encrypted volume is attached to the instance once it is restarted.
Environment
===========
1. Exact version of OpenStack you are running. See the following
list for all releases: http://docs.openstack.org/releases/
# git rev-parse HEAD
fce56ce8c04b20174cd89dfbc2c06f0068324b55
2. Which hypervisor did you use?
(For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
What's the version of that?
Libvirt + KVM
2. Which storage type did you use?
(For example: Ceph, LVM, GPFS, ...)
What's the version of that?
LVM+iSCSI
3. Which networking type did you use?
(For example: nova-network, Neutron with OpenVSwitch, ...)
N/A
Logs & Configs
==============
See above.
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1724573
Title:
When using resume_guests_state_on_host_boot encrypted volumes are
directly attached to instances after a host reboot
Status in OpenStack Compute (nova):
New
Bug description:
Description
===========
When using resume_guests_state_on_host_boot encrypted volumes are
directly attached to instances after a host reboot. These volumes
should be decrypted by the os-brick encryptors that provide libvirt
with decrypted dm devices for use by the instance/domain.
This is due to the following encryptor.attach_volume call being
skipped in the _hard_reboot where reboot=True as it is assumed the dm
devices are already present on the host:
5204 def _create_domain_and_network(self, context, xml, instance, network_info,
5205 block_device_info=None,
5206 power_on=True, reboot=False,
5207 vifs_already_plugged=False,
5208 post_xml_callback=None,
5209 destroy_disks_on_failure=False):
[..]
5218 if (not reboot and 'data' in connection_info and
5219 'volume_id' in connection_info['data']):
5220 volume_id = connection_info['data']['volume_id']
5221 encryption = encryptors.get_encryption_metadata(
5222 context, self._volume_api, volume_id, connection_info)
5223
5224 if encryption:
5225 encryptor = self._get_volume_encryptor(connection_info,
5226 encryption)
5227 encryptor.attach_volume(context, **encryption)
Steps to reproduce
==================
- Ensure resume_guests_state_on_host_boot is set to True within the
n-cpu config:
$ grep resume_guests_state_on_host_boot /etc/nova/nova-cpu.conf
resume_guests_state_on_host_boot = True
- Create an instance with an attached encrypted volume:
$ cinder type-create LUKS
$ cinder encryption-type-create --cipher aes-xts-plain64 --key_size 512 --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor
$ cinder create --display-name 'encrypted volume' --volume-type LUKS 1
$ nova boot --image cirros-0.3.5-x86_64-disk --flavor 1 test
$ nova volume-attach c762ef8d-13ab-4aee-bd20-c6a002bdd172 3f2cfdf2-11d7-4ac7-883a-76217136f751
- Before continuing note that the instance is connected to the
decrypted dm device:
$ sudo virsh domblklist c762ef8d-13ab-4aee-bd20-c6a002bdd172
Target Source
------------------------------------------------
vda /opt/stack/data/nova/instances/c762ef8d-13ab-4aee-bd20-c6a002bdd172/disk
vdb /dev/disk/by-id/scsi-360014054c6bbc8645494397ad372e0e6
$ ll /dev/disk/by-id/scsi-360014054c6bbc8645494397ad372e0e6
lrwxrwxrwx. 1 root root 56 Oct 18 08:28 /dev/disk/by-id/scsi-360014054c6bbc8645494397ad372e0e6 -> /dev/mapper/crypt-scsi-360014054c6bbc8645494397ad372e0e6
- Restart the n-cpu host _or_ fake a host reset by stopping the n-cpu
service, destroying the domain, removing the decrypted dm device,
unlinking the volume path before finally restarting n-cpu:
$ sudo systemctl stop devstack@n-cpu
$ sudo virsh destroy c762ef8d-13ab-4aee-bd20-c6a002bdd172
$ sudo cryptsetup luksClose /dev/disk/by-id/scsi-360014054c6bbc8645494397ad372e0e6
$ sudo unlink /dev/disk/by-id/scsi-360014054c6bbc8645494397ad372e0e6
$ sudo systemctl start devstack@n-cpu
- The instance is restarted but now points at the original encrypted
block device:
$ sudo virsh domblklist c762ef8d-13ab-4aee-bd20-c6a002bdd172
Target Source
------------------------------------------------
vda /opt/stack/data/nova/instances/c762ef8d-13ab-4aee-bd20-c6a002bdd172/disk
vdb /dev/disk/by-id/scsi-360014054c6bbc8645494397ad372e0e6
$ ll /dev/disk/by-id/scsi-360014054c6bbc8645494397ad372e0e6
lrwxrwxrwx. 1 root root 9 Oct 18 08:32 /dev/disk/by-id/scsi-360014054c6bbc8645494397ad372e0e6 -> ../../sde
- Additional stop and start requests will not correct this:
$ nova stop c762ef8d-13ab-4aee-bd20-c6a002bdd172
$ nova start c762ef8d-13ab-4aee-bd20-c6a002bdd172
$ sudo virsh domblklist c762ef8d-13ab-4aee-bd20-c6a002bdd172
Target Source
------------------------------------------------
vda /opt/stack/data/nova/instances/c762ef8d-13ab-4aee-bd20-c6a002bdd172/disk
vdb /dev/disk/by-id/scsi-360014054c6bbc8645494397ad372e0e6
$ ll /dev/disk/by-id/scsi-360014054c6bbc8645494397ad372e0e6
lrwxrwxrwx. 1 root root 9 Oct 18 08:32 /dev/disk/by-id/scsi-360014054c6bbc8645494397ad372e0e6 -> ../../sde
Expected result
===============
The decrypted volume is attached to the instance once it is restarted.
Actual result
=============
The encrypted volume is attached to the instance once it is restarted.
Environment
===========
1. Exact version of OpenStack you are running. See the following
list for all releases: http://docs.openstack.org/releases/
# git rev-parse HEAD
fce56ce8c04b20174cd89dfbc2c06f0068324b55
2. Which hypervisor did you use?
(For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
What's the version of that?
Libvirt + KVM
2. Which storage type did you use?
(For example: Ceph, LVM, GPFS, ...)
What's the version of that?
LVM+iSCSI
3. Which networking type did you use?
(For example: nova-network, Neutron with OpenVSwitch, ...)
N/A
Logs & Configs
==============
See above.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1724573/+subscriptions
Follow ups
