yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1688342] Re: Encryption key is not properly formatted before being passed to dmcrypt

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1688342@xxxxxxxxxxxxxxxxxx>
Date: Fri, 19 May 2017 15:49:40 -0000
Reply-to: Bug 1688342 <1688342@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/462674
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=53a71c1241aac70018c16d174032427a172378ed
Submitter: Jenkins
Branch:    master

commit 53a71c1241aac70018c16d174032427a172378ed
Author: Jackie Truong <jacklyn.truong@xxxxxxxxxx>
Date:   Thu May 4 12:51:22 2017 -0400

    Fix decoding of encryption key passed to dmcrypt
    
    This patch fixes the decoding of the encryption key passed to dmcrypt.
    During the key management move from Nova to Castellan, in the Newton
    release, conversion of the encryption key (from a string to list of
    unsigned ints) was removed from the key retrieval method. This patch
    updates dmcrypt to decode an encryption key string, rather than a list
    of unsigned ints. See the linked bug for more information.
    
    The method used to decode the encryption key has been updated to use
    binascii, as done in os-brick [1], to maintain consistency. The key
    generation and decoding portions of test_dmcrypt have been updated to
    reflect this change and ensure compatibility with both, Python 2 and
    Python 3.
    
    [1] https://github.com/openstack/os-brick/blob/6cf9b1cd689f70a2c50c0fa83a9a9f7c502712a1/os_brick/encryptors/cryptsetup.py#L100-L102
    
    Depends-On: I5fe3e5d5e5a9694d0dbe5b59248e5eaf89858c62
    Closes-Bug: #1688342
    Change-Id: I050585ecb55742a972038cf72b0650321ded2856


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1688342

Title:
  Encryption key is not properly formatted before being passed to
  dmcrypt

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Description
  ===========
  A TypeError occurs when using Nova to boot an instance with ephemeral storage encryption enabled.

  When key management was moved from Nova to Castellan in the Newton
  release, the key retrieval return value was changed from being
  formatted as a list of unsigned ints (in the case of an octet stream)
  [1] to not being formatted at all [2]. Nova's dmcrypt still expects
  the retrieved key [3][4] to be formatted as an array of unsigned bytes
  [5].

  References:
  [1] https://github.com/openstack/nova/blob/9702e5d2e9433c67067847c02fb100cd808a3596/nova/keymgr/barbican.py#L297-L323
  [2] https://github.com/openstack/castellan/blob/8d3f1cf375d28047cfd5f251afbb3f6d3ea7cda5/castellan/key_manager/barbican_key_manager.py#L527-L549
  [3] https://github.com/openstack/nova/blob/d219a3dcdc9c51bff3ebf2df086ea61a840ea3e9/nova/virt/libvirt/imagebackend.py#L714-L715
  [4] https://github.com/openstack/nova/blob/d219a3dcdc9c51bff3ebf2df086ea61a840ea3e9/nova/virt/libvirt/imagebackend.py#L674-L680
  [5] https://github.com/openstack/nova/blob/d219a3dcdc9c51bff3ebf2df086ea61a840ea3e9/nova/virt/libvirt/storage/dmcrypt.py#L48-L70

  
  Steps to reproduce
  ==================
  1. Set up an LVM device:
  Create a backing file:
    $ truncate nova-lvm -s 2G

  Mount the backing file on a loop device:
    $ sudo losetup /dev/loop1 nova-lvm

  Prepare the device for LVM:
    $ sudo pvcreate /dev/loop1

  Create the LVM group on the loop device:
    $ sudo vgcreate nova-lvm /dev/loop1

  2. Set up a devstack environment with ephemeral storage encryption enabled by adding the following lines to `lib/nova`:
    iniset $NOVA_CONF ephemeral_storage_encryption enabled "True"
    iniset $NOVA_CONF ephemeral_storage_encryption cipher "aes-xts-plain64"
    iniset $NOVA_CONF ephemeral_storage_encryption key_size "256"
    iniset $NOVA_CONF libvirt images_type "lvm"
    iniset $NOVA_CONF libvirt images_volume_group "nova-lvm"

  3. Stack:
    $ ./stack

  4. Use Nova to boot an instance:
    $ nova boot --flavor 1 --image {image_id}

  
  Expected result
  ===============
  Ephemeral storage encryption succeeds and Nova successfully boots the instance.

  
  Actual result
  =============
  Ephemeral storage encryption fails with a TypeError (similar results can be seen from Barbican Tempest gate failures [1]):

  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] Traceback (most recent call last):
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/compute/manager.py", line 2122, in _build_resources
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     yield resources
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/compute/manager.py", line 1927, in _build_and_run_instance
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     block_device_info=block_device_info)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2688, in spawn
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     block_device_info=block_device_info)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3095, in _create_image
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     fallback_from_host)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3211, in _create_and_inject_local_root
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     instance, size, fallback_from_host)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6780, in _try_fetch_image_cache
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     size=size)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 227, in cache
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     *args, **kwargs)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 735, in create_image
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     create_lvm_image(base, size)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     return f(*args, **kwargs)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 693, in create_lvm_image
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     encrypt_lvm_image()
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 680, in encrypt_lvm_image
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     key)
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/storage/dmcrypt.py", line 64, in create_volume
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     key = ''.join(map(lambda byte: "%02x" % byte, key))
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]   File "/opt/stack/nova/nova/virt/libvirt/storage/dmcrypt.py", line 64, in <lambda>
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56]     key = ''.join(map(lambda byte: "%02x" % byte, key))
  2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] TypeError: %x format: a number is required, not str

  
  Nova fails to boot the instance.

  
  [1] http://logs.openstack.org/59/455459/16/check/gate-barbican-simple-crypto-dsvm-tempest-ubuntu-xenial-nv/9c0fa48/logs/screen-n-cpu.txt.gz#_2017-04-24_21_01_37_841

  
  Environment
  ===========

  Latest `master` branch for all projects, except for Nova, which has
  patch [1] applied to get around the error addressed by the patch.
  However, remnants of the TypeError can still be seen without this
  patch [2].

  [1] https://review.openstack.org/#/c/462348/
  [2] http://logs.openstack.org/59/455459/17/check/gate-barbican-simple-crypto-dsvm-tempest-ubuntu-xenial-nv/26e8659/logs/screen-n-cpu.txt.gz#_2017-04-30_16_16_43_532

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1688342/+subscriptions
References

[Bug 1688342] [NEW] Encryption key is not properly formatted before being passed to dmcrypt
From: Jackie Truong, 2017-05-04