yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1688342] Re: Encryption key is not properly formatted before being passed to dmcrypt

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Tue, 23 May 2017 14:50:15 -0000
Reply-to: Bug 1688342 <1688342@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Also affects: nova/newton
   Importance: Undecided
       Status: New

** Also affects: nova/ocata
   Importance: Undecided
       Status: New

** Changed in: nova/newton
       Status: New => In Progress

** Changed in: nova/ocata
       Status: New => In Progress

** Changed in: nova
   Importance: Undecided => Medium

** Changed in: nova/newton
   Importance: Undecided => Medium

** Changed in: nova/ocata
   Importance: Undecided => Medium

** Changed in: nova/newton
     Assignee: (unassigned) => Lee Yarwood (lyarwood)

** Changed in: nova/ocata
     Assignee: (unassigned) => Lee Yarwood (lyarwood)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1688342

Title:
  Encryption key is not properly formatted before being passed to
  dmcrypt

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) newton series:
  In Progress
Status in OpenStack Compute (nova) ocata series:
  In Progress

Bug description:
  Description
  ===========
  A TypeError occurs when using Nova to boot an instance with ephemeral storage encryption enabled.

  When key management was moved from Nova to Castellan in the Newton
  release, the key retrieval return value was changed from being
  formatted as a list of unsigned ints (in the case of an octet stream)
  [1] to not being formatted at all [2]. Nova's dmcrypt still expects
  the retrieved key [3][4] to be formatted as an array of unsigned bytes
  [5].

  References:
  [1] https://github.com/openstack/nova/blob/9702e5d2e9433c67067847c02fb100cd808a3596/nova/keymgr/barbican.py#L297-L323
  [2] https://github.com/openstack/castellan/blob/8d3f1cf375d28047cfd5f251afbb3f6d3ea7cda5/castellan/key_manager/barbican_key_manager.py#L527-L549
  [3] https://github.com/openstack/nova/blob/d219a3dcdc9c51bff3ebf2df086ea61a840ea3e9/nova/virt/libvirt/imagebackend.py#L714-L715
  [4] https://github.com/openstack/nova/blob/d219a3dcdc9c51bff3ebf2df086ea61a840ea3e9/nova/virt/libvirt/imagebackend.py#L674-L680
  [5] https://github.com/openstack/nova/blob/d219a3dcdc9c51bff3ebf2df086ea61a840ea3e9/nova/virt/libvirt/storage/dmcrypt.py#L48-L70

  
  Steps to reproduce
  ==================
  1. Set up an LVM device:
  Create a backing file:
    $ truncate nova-lvm -s 2G

  Mount the backing file on a loop device:
    $ sudo losetup /dev/loop1 nova-lvm

  Prepare the device for LVM:
    $ sudo pvcreate /dev/loop1

  Create the LVM group on the loop device:
    $ sudo vgcreate nova-lvm /dev/loop1

  2. Set up a devstack environment with ephemeral storage encryption enabled by adding the following lines to `lib/nova`:
    iniset $NOVA_CONF ephemeral_storage_encryption enabled "True"
    iniset $NOVA_CONF ephemeral_storage_encryption cipher "aes-xts-plain64"
    iniset $NOVA_CONF ephemeral_storage_encryption key_size "256"
    iniset $NOVA_CONF libvirt images_type "lvm"
    iniset $NOVA_CONF libvirt images_volume_group "nova-lvm"

  3. Stack:
    $ ./stack

  4. Use Nova to boot an instance:
    $ nova boot --flavor 1 --image {image_id}

  
  Expected result
  ===============
  Ephemeral storage encryption succeeds and Nova successfully boots the instance.

  
  Actual result
  =============
  Ephemeral storage encryption fails with a TypeError (similar results can be seen from Barbican Tempest gate failures [1]):

  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] Traceback (most recent call last):
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/compute/manager.py", line 2122, in _build_resources
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     yield resources
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/compute/manager.py", line 1927, in _build_and_run_instance
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     block_device_info=block_device_info)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2688, in spawn
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     block_device_info=block_device_info)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3095, in _create_image
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     fallback_from_host)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3211, in _create_and_inject_local_root
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     instance, size, fallback_from_host)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6780, in _try_fetch_image_cache
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     size=size)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 227, in cache
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     *args, **kwargs)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 735, in create_image
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     create_lvm_image(base, size)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     return f(*args, **kwargs)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 693, in create_lvm_image
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     encrypt_lvm_image()
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 680, in encrypt_lvm_image
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     key)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/storage/dmcrypt.py", line 64, in create_volume
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     key = ''.join(map(lambda byte: "%02x" % byte, key))
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/storage/dmcrypt.py", line 64, in <lambda>
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     key = ''.join(map(lambda byte: "%02x" % byte, key))
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] TypeError: %x format: a number is required, not str

  
  Nova fails to boot the instance.

  
  [1] http://logs.openstack.org/59/455459/16/check/gate-barbican-simple-crypto-dsvm-tempest-ubuntu-xenial-nv/9c0fa48/logs/screen-n-cpu.txt.gz#_2017-04-24_21_01_37_841

  
  Environment
  ===========

  Latest `master` branch for all projects, except for Nova, which has
  patch [1] applied to get around the error addressed by the patch.
  However, remnants of the TypeError can still be seen without this
  patch [2].

  [1] https://review.openstack.org/#/c/462348/
  [2] http://logs.openstack.org/59/455459/17/check/gate-barbican-simple-crypto-dsvm-tempest-ubuntu-xenial-nv/26e8659/logs/screen-n-cpu.txt.gz#_2017-04-30_16_16_43_532

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1688342/+subscriptions
References

[Bug 1688342] [NEW] Encryption key is not properly formatted before being passed to dmcrypt
From: Jackie Truong, 2017-05-04