← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #63794

[Bug 1688342] [NEW] Encryption key is not properly formatted before being passed to dmcrypt

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Description
===========
A TypeError occurs when using Nova to boot an instance with ephemeral storage encryption enabled.

When key management was moved from Nova to Castellan in the Newton
release, the key retrieval return value was changed from being formatted
as a list of unsigned ints (in the case of an octet stream) [1] to not
being formatted at all [2]. Nova's dmcrypt still expects the retrieved
key [3][4] to be formatted as an array of unsigned bytes [5].

References:
[1] https://github.com/openstack/nova/blob/9702e5d2e9433c67067847c02fb100cd808a3596/nova/keymgr/barbican.py#L297-L323
[2] https://github.com/openstack/castellan/blob/8d3f1cf375d28047cfd5f251afbb3f6d3ea7cda5/castellan/key_manager/barbican_key_manager.py#L527-L549
[3] https://github.com/openstack/nova/blob/d219a3dcdc9c51bff3ebf2df086ea61a840ea3e9/nova/virt/libvirt/imagebackend.py#L714-L715
[4] https://github.com/openstack/nova/blob/d219a3dcdc9c51bff3ebf2df086ea61a840ea3e9/nova/virt/libvirt/imagebackend.py#L674-L680
[5] https://github.com/openstack/nova/blob/d219a3dcdc9c51bff3ebf2df086ea61a840ea3e9/nova/virt/libvirt/storage/dmcrypt.py#L48-L70


Steps to reproduce
==================
1. Set up an LVM device:
Create a backing file:
  $ truncate nova-lvm -s 2G

Mount the backing file on a loop device:
  $ sudo losetup /dev/loop1 nova-lvm

Prepare the device for LVM:
  $ sudo pvcreate /dev/loop1

Create the LVM group on the loop device:
  $ sudo vgcreate nova-lvm /dev/loop1

2. Set up a devstack environment with ephemeral storage encryption enabled by adding the following lines to `lib/nova`:
  iniset $NOVA_CONF ephemeral_storage_encryption enabled "True"
  iniset $NOVA_CONF ephemeral_storage_encryption cipher "aes-xts-plain64"
  iniset $NOVA_CONF ephemeral_storage_encryption key_size "256"
  iniset $NOVA_CONF libvirt images_type "lvm"
  iniset $NOVA_CONF libvirt images_volume_group "nova-lvm"

3. Stack:
  $ ./stack

4. Use Nova to boot an instance:
  $ nova boot --flavor 1 --image {image_id}


Expected result
===============
Ephemeral storage encryption succeeds and Nova successfully boots the instance.


Actual result
=============
Ephemeral storage encryption fails with a TypeError (similar results can be seen from Barbican Tempest gate failures [1]):

2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] Traceback (most recent call last):
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/compute/manager.py", line 2122, in _build_resources
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     yield resources
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/compute/manager.py", line 1927, in _build_and_run_instance
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     block_device_info=block_device_info)
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2688, in spawn
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     block_device_info=block_device_info)
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3095, in _create_image
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     fallback_from_host)
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3211, in _create_and_inject_local_root
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     instance, size, fallback_from_host)
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6780, in _try_fetch_image_cache
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     size=size)
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 227, in cache
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     *args, **kwargs)
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 735, in create_image
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     create_lvm_image(base, size)
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     return f(*args, **kwargs)
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 693, in create_lvm_image
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     encrypt_lvm_image()
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 680, in encrypt_lvm_image
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     key)
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/storage/dmcrypt.py", line 64, in create_volume
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     key = ''.join(map(lambda byte: "%02x" % byte, key))
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/storage/dmcrypt.py", line 64, in <lambda>
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     key = ''.join(map(lambda byte: "%02x" % byte, key))
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] TypeError: %x format: a number is required, not str


Nova fails to boot the instance.


[1] http://logs.openstack.org/59/455459/16/check/gate-barbican-simple-crypto-dsvm-tempest-ubuntu-xenial-nv/9c0fa48/logs/screen-n-cpu.txt.gz#_2017-04-24_21_01_37_841


Environment
===========

Latest `master` branch for all projects, except for Nova, which has
patch [1] applied to get around the error addressed by the patch.
However, remnants of the TypeError can still be seen without this patch
[2].

[1] https://review.openstack.org/#/c/462348/
[2] http://logs.openstack.org/59/455459/17/check/gate-barbican-simple-crypto-dsvm-tempest-ubuntu-xenial-nv/26e8659/logs/screen-n-cpu.txt.gz#_2017-04-30_16_16_43_532

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1688342

Title:
  Encryption key is not properly formatted before being passed to
  dmcrypt

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  A TypeError occurs when using Nova to boot an instance with ephemeral storage encryption enabled.

  When key management was moved from Nova to Castellan in the Newton
  release, the key retrieval return value was changed from being
  formatted as a list of unsigned ints (in the case of an octet stream)
  [1] to not being formatted at all [2]. Nova's dmcrypt still expects
  the retrieved key [3][4] to be formatted as an array of unsigned bytes
  [5].

  References:
  [1] https://github.com/openstack/nova/blob/9702e5d2e9433c67067847c02fb100cd808a3596/nova/keymgr/barbican.py#L297-L323
  [2] https://github.com/openstack/castellan/blob/8d3f1cf375d28047cfd5f251afbb3f6d3ea7cda5/castellan/key_manager/barbican_key_manager.py#L527-L549
  [3] https://github.com/openstack/nova/blob/d219a3dcdc9c51bff3ebf2df086ea61a840ea3e9/nova/virt/libvirt/imagebackend.py#L714-L715
  [4] https://github.com/openstack/nova/blob/d219a3dcdc9c51bff3ebf2df086ea61a840ea3e9/nova/virt/libvirt/imagebackend.py#L674-L680
  [5] https://github.com/openstack/nova/blob/d219a3dcdc9c51bff3ebf2df086ea61a840ea3e9/nova/virt/libvirt/storage/dmcrypt.py#L48-L70

  
  Steps to reproduce
  ==================
  1. Set up an LVM device:
  Create a backing file:
    $ truncate nova-lvm -s 2G

  Mount the backing file on a loop device:
    $ sudo losetup /dev/loop1 nova-lvm

  Prepare the device for LVM:
    $ sudo pvcreate /dev/loop1

  Create the LVM group on the loop device:
    $ sudo vgcreate nova-lvm /dev/loop1

  2. Set up a devstack environment with ephemeral storage encryption enabled by adding the following lines to `lib/nova`:
    iniset $NOVA_CONF ephemeral_storage_encryption enabled "True"
    iniset $NOVA_CONF ephemeral_storage_encryption cipher "aes-xts-plain64"
    iniset $NOVA_CONF ephemeral_storage_encryption key_size "256"
    iniset $NOVA_CONF libvirt images_type "lvm"
    iniset $NOVA_CONF libvirt images_volume_group "nova-lvm"

  3. Stack:
    $ ./stack

  4. Use Nova to boot an instance:
    $ nova boot --flavor 1 --image {image_id}

  
  Expected result
  ===============
  Ephemeral storage encryption succeeds and Nova successfully boots the instance.

  
  Actual result
  =============
  Ephemeral storage encryption fails with a TypeError (similar results can be seen from Barbican Tempest gate failures [1]):

  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] Traceback (most recent call last):
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/compute/manager.py", line 2122, in _build_resources
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     yield resources
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/compute/manager.py", line 1927, in _build_and_run_instance
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     block_device_info=block_device_info)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2688, in spawn
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     block_device_info=block_device_info)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3095, in _create_image
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     fallback_from_host)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3211, in _create_and_inject_local_root
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     instance, size, fallback_from_host)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6780, in _try_fetch_image_cache
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     size=size)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 227, in cache
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     *args, **kwargs)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 735, in create_image
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     create_lvm_image(base, size)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     return f(*args, **kwargs)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 693, in create_lvm_image
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     encrypt_lvm_image()
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 680, in encrypt_lvm_image
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     key)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/storage/dmcrypt.py", line 64, in create_volume
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     key = ''.join(map(lambda byte: "%02x" % byte, key))
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/storage/dmcrypt.py", line 64, in <lambda>
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     key = ''.join(map(lambda byte: "%02x" % byte, key))
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] TypeError: %x format: a number is required, not str

  
  Nova fails to boot the instance.

  
  [1] http://logs.openstack.org/59/455459/16/check/gate-barbican-simple-crypto-dsvm-tempest-ubuntu-xenial-nv/9c0fa48/logs/screen-n-cpu.txt.gz#_2017-04-24_21_01_37_841

  
  Environment
  ===========

  Latest `master` branch for all projects, except for Nova, which has
  patch [1] applied to get around the error addressed by the patch.
  However, remnants of the TypeError can still be seen without this
  patch [2].

  [1] https://review.openstack.org/#/c/462348/
  [2] http://logs.openstack.org/59/455459/17/check/gate-barbican-simple-crypto-dsvm-tempest-ubuntu-xenial-nv/26e8659/logs/screen-n-cpu.txt.gz#_2017-04-30_16_16_43_532

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1688342/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next