yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1651898] [NEW] Key manager configuration for ephemeral storage encryption is not backward compatible

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Guang Yee <1651898@xxxxxxxxxxxxxxxxxx>
Date: Wed, 21 Dec 2016 23:08:16 -0000
Reply-to: Bug 1651898 <1651898@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========
With the move to Castellan, Nova's key manager configuration is no longer backward compatible. Furthermore, looks like it hasn't been tested with grenade gate either. Otherwise, it would've easily break theory #1: New code should work with old configs.

The old config only have the [keymgr] section, not the [key_manager]
section. However, this line of code adds a default key manager in
[key_manager] section, which basically ignores the old config.

https://github.com/openstack/nova/blob/stable/newton/nova/keymgr/__init__.py#L29

In other words, the NoSucoOptError would've never raised.

https://github.com/openstack/nova/blob/stable/newton/nova/keymgr/__init__.py#L37


Steps to reproduce
==================

1. Install devstack with Barbican plugin enabled. i.e.

cat local.conf
[[local|localrc]]
enable_plugin barbican https://git.openstack.org/openstack/barbican stable/newton

2. After devstack is installed, revert back to the old config for key
manager and enable ephemeral storage encryption in nova.conf. i.e.

[keymgr]
api_class = nova.keymgr.barbican.BarbicanKeyManager

[barbican]
endpoint_template = http://localhost:9311/v1
os_region_name = RegionOne

[libvirt]
images_type = lvm
images_volume_group = vg-comp

[ephemeral_storage_encryption]
key_size = 256
cipher = aes-xts-plain64
enabled = True

3. try to restart nova-api and it will fail with a traceback that look
similar to this

2016-12-21 14:54:05.406 CRITICAL nova [req-04e1b733-5b50-41ae-
aa98-1a6b4f550cd7 None None] ValueError: keymgr.fixed_key not defined

2016-12-21 14:54:05.406 TRACE nova Traceback (most recent call last):
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/bin/nova-api", line 10, in <module>
2016-12-21 14:54:05.406 TRACE nova     sys.exit(main())
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/cmd/api.py", line 60, in main
2016-12-21 14:54:05.406 TRACE nova     server = service.WSGIService(api, use_ssl=should_use_ssl)
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/service.py", line 288, in __init__
2016-12-21 14:54:05.406 TRACE nova     self.app = self.loader.load_app(name)
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/wsgi.py", line 497, in load_app
2016-12-21 14:54:05.406 TRACE nova     return deploy.loadapp("config:%s" % self.config_path, name=name)
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2016-12-21 14:54:05.406 TRACE nova     return loadobj(APP, uri, name=name, **kw)
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2016-12-21 14:54:05.406 TRACE nova     return context.create()
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-12-21 14:54:05.406 TRACE nova     return self.object_type.invoke(self)
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2016-12-21 14:54:05.406 TRACE nova     **context.local_conf)
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-12-21 14:54:05.406 TRACE nova     val = callable(*args, **kw)
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/urlmap.py", line 160, in urlmap_factory
2016-12-21 14:54:05.406 TRACE nova     app = loader.get_app(app_name, global_conf=global_conf)
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2016-12-21 14:54:05.406 TRACE nova     name=name, global_conf=global_conf).create()
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-12-21 14:54:05.406 TRACE nova     return self.object_type.invoke(self)
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2016-12-21 14:54:05.406 TRACE nova     **context.local_conf)
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-12-21 14:54:05.406 TRACE nova     val = callable(*args, **kw)
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/auth.py", line 58, in pipeline_factory_v21
2016-12-21 14:54:05.406 TRACE nova     return _load_pipeline(loader, local_conf[CONF.auth_strategy].split())
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/auth.py", line 39, in _load_pipeline
2016-12-21 14:54:05.406 TRACE nova     app = loader.get_app(pipeline[-1])
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2016-12-21 14:54:05.406 TRACE nova     name=name, global_conf=global_conf).create()
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-12-21 14:54:05.406 TRACE nova     return self.object_type.invoke(self)
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 146, in invoke
2016-12-21 14:54:05.406 TRACE nova     return fix_call(context.object, context.global_conf, **context.local_conf)
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-12-21 14:54:05.406 TRACE nova     val = callable(*args, **kw)
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 219, in factory
2016-12-21 14:54:05.406 TRACE nova     return cls()
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/compute/__init__.py", line 35, in __init__
2016-12-21 14:54:05.406 TRACE nova     super(APIRouterV21, self).__init__(init_only)
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 244, in __init__
2016-12-21 14:54:05.406 TRACE nova     self._register_resources_check_inherits(mapper)
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 260, in _register_resources_check_inherits
2016-12-21 14:54:05.406 TRACE nova     for resource in ext.obj.get_resources():
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/compute/remote_consoles.py", line 207, in get_resources
2016-12-21 14:54:05.406 TRACE nova     'remote-consoles', RemoteConsolesController(), parent=parent,
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/compute/remote_consoles.py", line 32, in __init__
2016-12-21 14:54:05.406 TRACE nova     self.compute_api = compute.API()
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/compute/__init__.py", line 39, in API
2016-12-21 14:54:05.406 TRACE nova     return importutils.import_object(class_name, *args, **kwargs)
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 44, in import_object
2016-12-21 14:54:05.406 TRACE nova     return import_class(import_str)(*args, **kwargs)
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/compute/api.py", line 211, in __init__
2016-12-21 14:54:05.406 TRACE nova     self.key_manager = keymgr.API()
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/keymgr/__init__.py", line 75, in API
2016-12-21 14:54:05.406 TRACE nova     return cls(conf)
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/keymgr/conf_key_mgr.py", line 67, in __init__
2016-12-21 14:54:05.406 TRACE nova     raise ValueError(_('keymgr.fixed_key not defined'))
2016-12-21 14:54:05.406 TRACE nova ValueError: keymgr.fixed_key not defined


Expected result
===============
server should start correctly with an old config

Actual result
=============
server failed to start with the following traceback

2016-12-21 14:54:05.406 CRITICAL nova [req-04e1b733-5b50-41ae-
aa98-1a6b4f550cd7 None None] ValueError: keymgr.fixed_key not defined

2016-12-21 14:54:05.406 TRACE nova Traceback (most recent call last):
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/bin/nova-api", line 10, in <module>
2016-12-21 14:54:05.406 TRACE nova     sys.exit(main())
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/cmd/api.py", line 60, in main
2016-12-21 14:54:05.406 TRACE nova     server = service.WSGIService(api, use_ssl=should_use_ssl)
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/service.py", line 288, in __init__
2016-12-21 14:54:05.406 TRACE nova     self.app = self.loader.load_app(name)
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/wsgi.py", line 497, in load_app
2016-12-21 14:54:05.406 TRACE nova     return deploy.loadapp("config:%s" % self.config_path, name=name)
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2016-12-21 14:54:05.406 TRACE nova     return loadobj(APP, uri, name=name, **kw)
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2016-12-21 14:54:05.406 TRACE nova     return context.create()
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-12-21 14:54:05.406 TRACE nova     return self.object_type.invoke(self)
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2016-12-21 14:54:05.406 TRACE nova     **context.local_conf)
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-12-21 14:54:05.406 TRACE nova     val = callable(*args, **kw)
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/urlmap.py", line 160, in urlmap_factory
2016-12-21 14:54:05.406 TRACE nova     app = loader.get_app(app_name, global_conf=global_conf)
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2016-12-21 14:54:05.406 TRACE nova     name=name, global_conf=global_conf).create()
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-12-21 14:54:05.406 TRACE nova     return self.object_type.invoke(self)
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2016-12-21 14:54:05.406 TRACE nova     **context.local_conf)
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-12-21 14:54:05.406 TRACE nova     val = callable(*args, **kw)
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/auth.py", line 58, in pipeline_factory_v21
2016-12-21 14:54:05.406 TRACE nova     return _load_pipeline(loader, local_conf[CONF.auth_strategy].split())
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/auth.py", line 39, in _load_pipeline
2016-12-21 14:54:05.406 TRACE nova     app = loader.get_app(pipeline[-1])
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2016-12-21 14:54:05.406 TRACE nova     name=name, global_conf=global_conf).create()
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-12-21 14:54:05.406 TRACE nova     return self.object_type.invoke(self)
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 146, in invoke
2016-12-21 14:54:05.406 TRACE nova     return fix_call(context.object, context.global_conf, **context.local_conf)
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-12-21 14:54:05.406 TRACE nova     val = callable(*args, **kw)
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 219, in factory
2016-12-21 14:54:05.406 TRACE nova     return cls()
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/compute/__init__.py", line 35, in __init__
2016-12-21 14:54:05.406 TRACE nova     super(APIRouterV21, self).__init__(init_only)
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 244, in __init__
2016-12-21 14:54:05.406 TRACE nova     self._register_resources_check_inherits(mapper)
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 260, in _register_resources_check_inherits
2016-12-21 14:54:05.406 TRACE nova     for resource in ext.obj.get_resources():
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/compute/remote_consoles.py", line 207, in get_resources
2016-12-21 14:54:05.406 TRACE nova     'remote-consoles', RemoteConsolesController(), parent=parent,
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/compute/remote_consoles.py", line 32, in __init__
2016-12-21 14:54:05.406 TRACE nova     self.compute_api = compute.API()
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/compute/__init__.py", line 39, in API
2016-12-21 14:54:05.406 TRACE nova     return importutils.import_object(class_name, *args, **kwargs)
2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 44, in import_object
2016-12-21 14:54:05.406 TRACE nova     return import_class(import_str)(*args, **kwargs)
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/compute/api.py", line 211, in __init__
2016-12-21 14:54:05.406 TRACE nova     self.key_manager = keymgr.API()
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/keymgr/__init__.py", line 75, in API
2016-12-21 14:54:05.406 TRACE nova     return cls(conf)
2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/keymgr/conf_key_mgr.py", line 67, in __init__
2016-12-21 14:54:05.406 TRACE nova     raise ValueError(_('keymgr.fixed_key not defined'))
2016-12-21 14:54:05.406 TRACE nova ValueError: keymgr.fixed_key not defined


Environment
===========
1. Ubuntu 16.04

cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"

2. Devstack + Barbican plugin, both on stable/newton

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1651898

Title:
  Key manager configuration for ephemeral storage encryption is not
  backward compatible

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  With the move to Castellan, Nova's key manager configuration is no longer backward compatible. Furthermore, looks like it hasn't been tested with grenade gate either. Otherwise, it would've easily break theory #1: New code should work with old configs.

  The old config only have the [keymgr] section, not the [key_manager]
  section. However, this line of code adds a default key manager in
  [key_manager] section, which basically ignores the old config.

  https://github.com/openstack/nova/blob/stable/newton/nova/keymgr/__init__.py#L29

  In other words, the NoSucoOptError would've never raised.

  https://github.com/openstack/nova/blob/stable/newton/nova/keymgr/__init__.py#L37

  
  Steps to reproduce
  ==================

  1. Install devstack with Barbican plugin enabled. i.e.

  cat local.conf
  [[local|localrc]]
  enable_plugin barbican https://git.openstack.org/openstack/barbican stable/newton

  2. After devstack is installed, revert back to the old config for key
  manager and enable ephemeral storage encryption in nova.conf. i.e.

  [keymgr]
  api_class = nova.keymgr.barbican.BarbicanKeyManager

  [barbican]
  endpoint_template = http://localhost:9311/v1
  os_region_name = RegionOne

  [libvirt]
  images_type = lvm
  images_volume_group = vg-comp

  [ephemeral_storage_encryption]
  key_size = 256
  cipher = aes-xts-plain64
  enabled = True

  3. try to restart nova-api and it will fail with a traceback that look
  similar to this

  2016-12-21 14:54:05.406 CRITICAL nova [req-04e1b733-5b50-41ae-
  aa98-1a6b4f550cd7 None None] ValueError: keymgr.fixed_key not defined

  2016-12-21 14:54:05.406 TRACE nova Traceback (most recent call last):
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/bin/nova-api", line 10, in <module>
  2016-12-21 14:54:05.406 TRACE nova     sys.exit(main())
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/cmd/api.py", line 60, in main
  2016-12-21 14:54:05.406 TRACE nova     server = service.WSGIService(api, use_ssl=should_use_ssl)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/service.py", line 288, in __init__
  2016-12-21 14:54:05.406 TRACE nova     self.app = self.loader.load_app(name)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/wsgi.py", line 497, in load_app
  2016-12-21 14:54:05.406 TRACE nova     return deploy.loadapp("config:%s" % self.config_path, name=name)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
  2016-12-21 14:54:05.406 TRACE nova     return loadobj(APP, uri, name=name, **kw)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
  2016-12-21 14:54:05.406 TRACE nova     return context.create()
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
  2016-12-21 14:54:05.406 TRACE nova     return self.object_type.invoke(self)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
  2016-12-21 14:54:05.406 TRACE nova     **context.local_conf)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
  2016-12-21 14:54:05.406 TRACE nova     val = callable(*args, **kw)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/urlmap.py", line 160, in urlmap_factory
  2016-12-21 14:54:05.406 TRACE nova     app = loader.get_app(app_name, global_conf=global_conf)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
  2016-12-21 14:54:05.406 TRACE nova     name=name, global_conf=global_conf).create()
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
  2016-12-21 14:54:05.406 TRACE nova     return self.object_type.invoke(self)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
  2016-12-21 14:54:05.406 TRACE nova     **context.local_conf)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
  2016-12-21 14:54:05.406 TRACE nova     val = callable(*args, **kw)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/auth.py", line 58, in pipeline_factory_v21
  2016-12-21 14:54:05.406 TRACE nova     return _load_pipeline(loader, local_conf[CONF.auth_strategy].split())
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/auth.py", line 39, in _load_pipeline
  2016-12-21 14:54:05.406 TRACE nova     app = loader.get_app(pipeline[-1])
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
  2016-12-21 14:54:05.406 TRACE nova     name=name, global_conf=global_conf).create()
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
  2016-12-21 14:54:05.406 TRACE nova     return self.object_type.invoke(self)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 146, in invoke
  2016-12-21 14:54:05.406 TRACE nova     return fix_call(context.object, context.global_conf, **context.local_conf)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
  2016-12-21 14:54:05.406 TRACE nova     val = callable(*args, **kw)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 219, in factory
  2016-12-21 14:54:05.406 TRACE nova     return cls()
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/compute/__init__.py", line 35, in __init__
  2016-12-21 14:54:05.406 TRACE nova     super(APIRouterV21, self).__init__(init_only)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 244, in __init__
  2016-12-21 14:54:05.406 TRACE nova     self._register_resources_check_inherits(mapper)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 260, in _register_resources_check_inherits
  2016-12-21 14:54:05.406 TRACE nova     for resource in ext.obj.get_resources():
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/compute/remote_consoles.py", line 207, in get_resources
  2016-12-21 14:54:05.406 TRACE nova     'remote-consoles', RemoteConsolesController(), parent=parent,
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/compute/remote_consoles.py", line 32, in __init__
  2016-12-21 14:54:05.406 TRACE nova     self.compute_api = compute.API()
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/compute/__init__.py", line 39, in API
  2016-12-21 14:54:05.406 TRACE nova     return importutils.import_object(class_name, *args, **kwargs)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 44, in import_object
  2016-12-21 14:54:05.406 TRACE nova     return import_class(import_str)(*args, **kwargs)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/compute/api.py", line 211, in __init__
  2016-12-21 14:54:05.406 TRACE nova     self.key_manager = keymgr.API()
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/keymgr/__init__.py", line 75, in API
  2016-12-21 14:54:05.406 TRACE nova     return cls(conf)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/keymgr/conf_key_mgr.py", line 67, in __init__
  2016-12-21 14:54:05.406 TRACE nova     raise ValueError(_('keymgr.fixed_key not defined'))
  2016-12-21 14:54:05.406 TRACE nova ValueError: keymgr.fixed_key not defined


  Expected result
  ===============
  server should start correctly with an old config

  Actual result
  =============
  server failed to start with the following traceback

  2016-12-21 14:54:05.406 CRITICAL nova [req-04e1b733-5b50-41ae-
  aa98-1a6b4f550cd7 None None] ValueError: keymgr.fixed_key not defined

  2016-12-21 14:54:05.406 TRACE nova Traceback (most recent call last):
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/bin/nova-api", line 10, in <module>
  2016-12-21 14:54:05.406 TRACE nova     sys.exit(main())
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/cmd/api.py", line 60, in main
  2016-12-21 14:54:05.406 TRACE nova     server = service.WSGIService(api, use_ssl=should_use_ssl)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/service.py", line 288, in __init__
  2016-12-21 14:54:05.406 TRACE nova     self.app = self.loader.load_app(name)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/wsgi.py", line 497, in load_app
  2016-12-21 14:54:05.406 TRACE nova     return deploy.loadapp("config:%s" % self.config_path, name=name)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
  2016-12-21 14:54:05.406 TRACE nova     return loadobj(APP, uri, name=name, **kw)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
  2016-12-21 14:54:05.406 TRACE nova     return context.create()
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
  2016-12-21 14:54:05.406 TRACE nova     return self.object_type.invoke(self)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
  2016-12-21 14:54:05.406 TRACE nova     **context.local_conf)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
  2016-12-21 14:54:05.406 TRACE nova     val = callable(*args, **kw)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/urlmap.py", line 160, in urlmap_factory
  2016-12-21 14:54:05.406 TRACE nova     app = loader.get_app(app_name, global_conf=global_conf)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
  2016-12-21 14:54:05.406 TRACE nova     name=name, global_conf=global_conf).create()
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
  2016-12-21 14:54:05.406 TRACE nova     return self.object_type.invoke(self)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
  2016-12-21 14:54:05.406 TRACE nova     **context.local_conf)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
  2016-12-21 14:54:05.406 TRACE nova     val = callable(*args, **kw)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/auth.py", line 58, in pipeline_factory_v21
  2016-12-21 14:54:05.406 TRACE nova     return _load_pipeline(loader, local_conf[CONF.auth_strategy].split())
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/auth.py", line 39, in _load_pipeline
  2016-12-21 14:54:05.406 TRACE nova     app = loader.get_app(pipeline[-1])
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
  2016-12-21 14:54:05.406 TRACE nova     name=name, global_conf=global_conf).create()
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
  2016-12-21 14:54:05.406 TRACE nova     return self.object_type.invoke(self)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 146, in invoke
  2016-12-21 14:54:05.406 TRACE nova     return fix_call(context.object, context.global_conf, **context.local_conf)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
  2016-12-21 14:54:05.406 TRACE nova     val = callable(*args, **kw)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 219, in factory
  2016-12-21 14:54:05.406 TRACE nova     return cls()
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/compute/__init__.py", line 35, in __init__
  2016-12-21 14:54:05.406 TRACE nova     super(APIRouterV21, self).__init__(init_only)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 244, in __init__
  2016-12-21 14:54:05.406 TRACE nova     self._register_resources_check_inherits(mapper)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 260, in _register_resources_check_inherits
  2016-12-21 14:54:05.406 TRACE nova     for resource in ext.obj.get_resources():
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/compute/remote_consoles.py", line 207, in get_resources
  2016-12-21 14:54:05.406 TRACE nova     'remote-consoles', RemoteConsolesController(), parent=parent,
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/compute/remote_consoles.py", line 32, in __init__
  2016-12-21 14:54:05.406 TRACE nova     self.compute_api = compute.API()
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/compute/__init__.py", line 39, in API
  2016-12-21 14:54:05.406 TRACE nova     return importutils.import_object(class_name, *args, **kwargs)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 44, in import_object
  2016-12-21 14:54:05.406 TRACE nova     return import_class(import_str)(*args, **kwargs)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/compute/api.py", line 211, in __init__
  2016-12-21 14:54:05.406 TRACE nova     self.key_manager = keymgr.API()
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/keymgr/__init__.py", line 75, in API
  2016-12-21 14:54:05.406 TRACE nova     return cls(conf)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/keymgr/conf_key_mgr.py", line 67, in __init__
  2016-12-21 14:54:05.406 TRACE nova     raise ValueError(_('keymgr.fixed_key not defined'))
  2016-12-21 14:54:05.406 TRACE nova ValueError: keymgr.fixed_key not defined

  
  Environment
  ===========
  1. Ubuntu 16.04

  cat /etc/lsb-release
  DISTRIB_ID=Ubuntu
  DISTRIB_RELEASE=16.04
  DISTRIB_CODENAME=xenial
  DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"

  2. Devstack + Barbican plugin, both on stable/newton

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1651898/+subscriptions
Follow ups

[Bug 1651898] Re: Key manager configuration for ephemeral storage encryption is not backward compatible
From: Lee Yarwood, 2019-11-07