yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1651887] Re: Ephemeral storage encryption is broken with interface mismatch

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1651887@xxxxxxxxxxxxxxxxxx>
Date: Thu, 18 May 2017 21:32:03 -0000
Reply-to: Bug 1651887 <1651887@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/446072
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1d3acad111c5106592b0921628480fcf41e0fb4c
Submitter: Jenkins
Branch:    master

commit 1d3acad111c5106592b0921628480fcf41e0fb4c
Author: Kaitlin Farr <kaitlin.farr@xxxxxxxxxx>
Date:   Fri Mar 10 18:09:49 2017 -0500

    Parse algorithm from cipher for ephemeral disk encryption
    
    Nova's keymgr implementation used to have default values
    for the algorithm and bit length.  Castellan does not have
    default values, and when Castellan replaced keymgr in
    Ib563b0ea4b8b4bc1833bf52bf49a68546c384996, the parameters
    to the create_key method were not updated. This change
    parses the algorithm from the cipher value and passes it
    to Castellan's key manager interface.
    
    Closes-Bug: #1651887
    Change-Id: Ib90bc7571aef59325be0efe123fcf12e86252b85


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1651887

Title:
  Ephemeral storage encryption is broken with interface mismatch

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) newton series:
  Confirmed
Status in OpenStack Compute (nova) ocata series:
  Confirmed

Bug description:
  Description
  ===========
  Ephemeral storage encryption is broken because of interface mismatch. The default key manager (Castellan with Barbican)'s create_key() interface required at least 4 arguments. See

  https://github.com/openstack/castellan/blob/0.4.0/castellan/key_manager/barbican_key_manager.py#L200

  However, Nova is only passing in 3. Looks like the 'algorithm'
  argument is missing. See

  https://github.com/openstack/nova/blob/stable/newton/nova/compute/api.py#L1401

  This will result in "TypeError: create_key() takes exactly 4 arguments
  (3 given)" on server create.

  Steps to reproduce
  ==================

  1. Install devstack with Barbican plugin enabled. i.e.

  cat local.conf
  [[local|localrc]]
  enable_plugin barbican https://git.openstack.org/openstack/barbican stable/newton

  2. After devstack is installed, enable ephemeral storage encryption in
  nova.conf. i.e.

  [libvirt]
  images_type = lvm
  images_volume_group = vg-comp

  [ephemeral_storage_encryption]
  key_size = 256
  cipher = aes-xts-plain64
  enabled = True

  3. restart nova-api

  4. using the nova user account, try to create a server. i.e.

  gyee@abacus:~$ env | grep OS_
  OS_PROJECT_DOMAIN_ID=default
  OS_USER_DOMAIN_ID=default
  OS_PROJECT_NAME=service
  OS_IDENTITY_API_VERSION=3
  OS_PASSWORD=secrete
  OS_AUTH_URL=http://localhost:5000
  OS_USERNAME=nova
  gyee@abacus:~$ openstack flavor list
  +----+-----------+-------+------+-----------+-------+-----------+
  | ID | Name      |   RAM | Disk | Ephemeral | VCPUs | Is Public |
  +----+-----------+-------+------+-----------+-------+-----------+
  | 1  | m1.tiny   |   512 |    1 |         0 |     1 | True      |
  | 2  | m1.small  |  2048 |   20 |         0 |     1 | True      |
  | 3  | m1.medium |  4096 |   40 |         0 |     2 | True      |
  | 4  | m1.large  |  8192 |   80 |         0 |     4 | True      |
  | 42 | m1.nano   |    64 |    0 |         0 |     1 | True      |
  | 5  | m1.xlarge | 16384 |  160 |         0 |     8 | True      |
  | 84 | m1.micro  |   128 |    0 |         0 |     1 | True      |
  | c1 | cirros256 |   256 |    0 |         0 |     1 | True      |
  | d1 | ds512M    |   512 |    5 |         0 |     1 | True      |
  | d2 | ds1G      |  1024 |   10 |         0 |     1 | True      |
  | d3 | ds2G      |  2048 |   10 |         0 |     2 | True      |
  | d4 | ds4G      |  4096 |   20 |         0 |     4 | True      |
  +----+-----------+-------+------+-----------+-------+-----------+
  gyee@abacus:~$ openstack image list
  +----------------------------------+---------------------------------+--------+
  | ID                               | Name                            | Status |
  +----------------------------------+---------------------------------+--------+
  | da447cd9-619a-                   | cirros-0.3.4-x86_64-uec         | active |
  | 41b3-9772-4a9a80fa55f9           |                                 |        |
  | 718fff25-9d61-4a37-a974-fdef2f1f | cirros-0.3.4-x86_64-uec-ramdisk | active |
  | 570a                             |                                 |        |
  | 91c06518-a752-48ec-a7fd-         | cirros-0.3.4-x86_64-uec-kernel  | active |
  | 3c0ad020d9a4                     |                                 |        |
  +----------------------------------+---------------------------------+--------+
  gyee@abacus:~$ openstack server create --image 91c06518-a752-48ec-a7fd-3c0ad020d9a4 --flavor 1 test_eph_enc
  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <type 'exceptions.TypeError'> (HTTP 500) (Request-ID: req-6d2eb531-b239-429d-8d25-f06b4fe6309c)

  
  5. And you'll see a traceback similiar to this.

  2016-12-21 14:04:40.903 ERROR nova.api.openstack.extensions [req-6d2eb531-b239-429d-8d25-f06b4fe6309c nova service] Unexpected exception in API method
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions Traceback (most recent call last):
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/extensions.py", line 338, in wrapped
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     return f(*args, **kwargs)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 631, in create
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     **create_kwargs)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/hooks.py", line 154, in inner
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     rv = f(*args, **kwargs)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 1528, in create
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 1159, in _create_instance
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     key_pair)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 1022, in _provision_instances
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     quotas.rollback()
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     self.force_reraise()
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 944, in _provision_instances
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     shutdown_terminate, create_instance=False)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 1451, in create_db_entry_for_new_instance
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     num_instances, shutdown_terminate)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 1403, in _populate_instance_for_create
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     length=CONF.ephemeral_storage_encryption.key_size)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions TypeError: create_key() takes exactly 4 arguments (3 given)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions 
  2016-12-21 14:04:40.905 INFO nova.api.openstack.wsgi [req-6d2eb531-b239-429d-8d25-f06b4fe6309c nova service] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <type 'exceptions.TypeError'>

  
  Expected result
  ===============
  server should be successfully created

  Actual result
  =============
  HTTP 500, with the following traceback in the log.

  2016-12-21 14:04:40.903 ERROR nova.api.openstack.extensions [req-6d2eb531-b239-429d-8d25-f06b4fe6309c nova service] Unexpected exception in API method
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions Traceback (most recent call last):
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/extensions.py", line 338, in wrapped
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     return f(*args, **kwargs)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 631, in create
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     **create_kwargs)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/hooks.py", line 154, in inner
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     rv = f(*args, **kwargs)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 1528, in create
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 1159, in _create_instance
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     key_pair)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 1022, in _provision_instances
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     quotas.rollback()
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     self.force_reraise()
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 944, in _provision_instances
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     shutdown_terminate, create_instance=False)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 1451, in create_db_entry_for_new_instance
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     num_instances, shutdown_terminate)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 1403, in _populate_instance_for_create
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions     length=CONF.ephemeral_storage_encryption.key_size)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions TypeError: create_key() takes exactly 4 arguments (3 given)
  2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions 
  2016-12-21 14:04:40.905 INFO nova.api.openstack.wsgi [req-6d2eb531-b239-429d-8d25-f06b4fe6309c nova service] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <type 'exceptions.TypeError'>

  
  Environment
  ===========
  1. Ubuntu 16.04

  cat /etc/lsb-release 
  DISTRIB_ID=Ubuntu
  DISTRIB_RELEASE=16.04
  DISTRIB_CODENAME=xenial
  DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"

  2. Devstack + Barbican plugin, both on stable/newton

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1651887/+subscriptions
References

[Bug 1651887] [NEW] Ephemeral storage encryption is broken with interface mismatch
From: Guang Yee, 2016-12-21