yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1651898] Re: Key manager configuration for ephemeral storage encryption is not backward compatible

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Lee Yarwood <lyarwood@xxxxxxxxxx>
Date: Thu, 07 Nov 2019 10:11:10 -0000
Reply-to: Bug 1651898 <1651898@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: Confirmed => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1651898

Title:
  Key manager configuration for ephemeral storage encryption is not
  backward compatible

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Description
  ===========
  With the move to Castellan, Nova's key manager configuration is no longer backward compatible. Furthermore, looks like it hasn't been tested with grenade gate either. Otherwise, it would've easily break theory #1: New code should work with old configs.

  The old config only have the [keymgr] section, not the [key_manager]
  section. However, this line of code adds a default key manager in
  [key_manager] section, which basically ignores the old config.

  https://github.com/openstack/nova/blob/stable/newton/nova/keymgr/__init__.py#L29

  In other words, the NoSucoOptError would've never raised.

  https://github.com/openstack/nova/blob/stable/newton/nova/keymgr/__init__.py#L37

  
  Steps to reproduce
  ==================

  1. Install devstack with Barbican plugin enabled. i.e.

  cat local.conf
  [[local|localrc]]
  enable_plugin barbican https://git.openstack.org/openstack/barbican stable/newton

  2. After devstack is installed, revert back to the old config for key
  manager and enable ephemeral storage encryption in nova.conf. i.e.

  [keymgr]
  api_class = nova.keymgr.barbican.BarbicanKeyManager

  [barbican]
  endpoint_template = http://localhost:9311/v1
  os_region_name = RegionOne

  [libvirt]
  images_type = lvm
  images_volume_group = vg-comp

  [ephemeral_storage_encryption]
  key_size = 256
  cipher = aes-xts-plain64
  enabled = True

  3. try to restart nova-api and it will fail with a traceback that look
  similar to this

  2016-12-21 14:54:05.406 CRITICAL nova [req-04e1b733-5b50-41ae-
  aa98-1a6b4f550cd7 None None] ValueError: keymgr.fixed_key not defined

  2016-12-21 14:54:05.406 TRACE nova Traceback (most recent call last):
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/bin/nova-api", line 10, in <module>
  2016-12-21 14:54:05.406 TRACE nova     sys.exit(main())
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/cmd/api.py", line 60, in main
  2016-12-21 14:54:05.406 TRACE nova     server = service.WSGIService(api, use_ssl=should_use_ssl)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/service.py", line 288, in __init__
  2016-12-21 14:54:05.406 TRACE nova     self.app = self.loader.load_app(name)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/wsgi.py", line 497, in load_app
  2016-12-21 14:54:05.406 TRACE nova     return deploy.loadapp("config:%s" % self.config_path, name=name)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
  2016-12-21 14:54:05.406 TRACE nova     return loadobj(APP, uri, name=name, **kw)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
  2016-12-21 14:54:05.406 TRACE nova     return context.create()
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
  2016-12-21 14:54:05.406 TRACE nova     return self.object_type.invoke(self)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
  2016-12-21 14:54:05.406 TRACE nova     **context.local_conf)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
  2016-12-21 14:54:05.406 TRACE nova     val = callable(*args, **kw)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/urlmap.py", line 160, in urlmap_factory
  2016-12-21 14:54:05.406 TRACE nova     app = loader.get_app(app_name, global_conf=global_conf)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
  2016-12-21 14:54:05.406 TRACE nova     name=name, global_conf=global_conf).create()
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
  2016-12-21 14:54:05.406 TRACE nova     return self.object_type.invoke(self)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
  2016-12-21 14:54:05.406 TRACE nova     **context.local_conf)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
  2016-12-21 14:54:05.406 TRACE nova     val = callable(*args, **kw)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/auth.py", line 58, in pipeline_factory_v21
  2016-12-21 14:54:05.406 TRACE nova     return _load_pipeline(loader, local_conf[CONF.auth_strategy].split())
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/auth.py", line 39, in _load_pipeline
  2016-12-21 14:54:05.406 TRACE nova     app = loader.get_app(pipeline[-1])
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
  2016-12-21 14:54:05.406 TRACE nova     name=name, global_conf=global_conf).create()
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
  2016-12-21 14:54:05.406 TRACE nova     return self.object_type.invoke(self)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 146, in invoke
  2016-12-21 14:54:05.406 TRACE nova     return fix_call(context.object, context.global_conf, **context.local_conf)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
  2016-12-21 14:54:05.406 TRACE nova     val = callable(*args, **kw)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 219, in factory
  2016-12-21 14:54:05.406 TRACE nova     return cls()
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/compute/__init__.py", line 35, in __init__
  2016-12-21 14:54:05.406 TRACE nova     super(APIRouterV21, self).__init__(init_only)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 244, in __init__
  2016-12-21 14:54:05.406 TRACE nova     self._register_resources_check_inherits(mapper)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 260, in _register_resources_check_inherits
  2016-12-21 14:54:05.406 TRACE nova     for resource in ext.obj.get_resources():
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/compute/remote_consoles.py", line 207, in get_resources
  2016-12-21 14:54:05.406 TRACE nova     'remote-consoles', RemoteConsolesController(), parent=parent,
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/compute/remote_consoles.py", line 32, in __init__
  2016-12-21 14:54:05.406 TRACE nova     self.compute_api = compute.API()
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/compute/__init__.py", line 39, in API
  2016-12-21 14:54:05.406 TRACE nova     return importutils.import_object(class_name, *args, **kwargs)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 44, in import_object
  2016-12-21 14:54:05.406 TRACE nova     return import_class(import_str)(*args, **kwargs)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/compute/api.py", line 211, in __init__
  2016-12-21 14:54:05.406 TRACE nova     self.key_manager = keymgr.API()
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/keymgr/__init__.py", line 75, in API
  2016-12-21 14:54:05.406 TRACE nova     return cls(conf)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/keymgr/conf_key_mgr.py", line 67, in __init__
  2016-12-21 14:54:05.406 TRACE nova     raise ValueError(_('keymgr.fixed_key not defined'))
  2016-12-21 14:54:05.406 TRACE nova ValueError: keymgr.fixed_key not defined


  Expected result
  ===============
  server should start correctly with an old config

  Actual result
  =============
  server failed to start with the following traceback

  2016-12-21 14:54:05.406 CRITICAL nova [req-04e1b733-5b50-41ae-
  aa98-1a6b4f550cd7 None None] ValueError: keymgr.fixed_key not defined

  2016-12-21 14:54:05.406 TRACE nova Traceback (most recent call last):
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/bin/nova-api", line 10, in <module>
  2016-12-21 14:54:05.406 TRACE nova     sys.exit(main())
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/cmd/api.py", line 60, in main
  2016-12-21 14:54:05.406 TRACE nova     server = service.WSGIService(api, use_ssl=should_use_ssl)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/service.py", line 288, in __init__
  2016-12-21 14:54:05.406 TRACE nova     self.app = self.loader.load_app(name)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/wsgi.py", line 497, in load_app
  2016-12-21 14:54:05.406 TRACE nova     return deploy.loadapp("config:%s" % self.config_path, name=name)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
  2016-12-21 14:54:05.406 TRACE nova     return loadobj(APP, uri, name=name, **kw)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
  2016-12-21 14:54:05.406 TRACE nova     return context.create()
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
  2016-12-21 14:54:05.406 TRACE nova     return self.object_type.invoke(self)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
  2016-12-21 14:54:05.406 TRACE nova     **context.local_conf)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
  2016-12-21 14:54:05.406 TRACE nova     val = callable(*args, **kw)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/urlmap.py", line 160, in urlmap_factory
  2016-12-21 14:54:05.406 TRACE nova     app = loader.get_app(app_name, global_conf=global_conf)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
  2016-12-21 14:54:05.406 TRACE nova     name=name, global_conf=global_conf).create()
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
  2016-12-21 14:54:05.406 TRACE nova     return self.object_type.invoke(self)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
  2016-12-21 14:54:05.406 TRACE nova     **context.local_conf)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
  2016-12-21 14:54:05.406 TRACE nova     val = callable(*args, **kw)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/auth.py", line 58, in pipeline_factory_v21
  2016-12-21 14:54:05.406 TRACE nova     return _load_pipeline(loader, local_conf[CONF.auth_strategy].split())
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/auth.py", line 39, in _load_pipeline
  2016-12-21 14:54:05.406 TRACE nova     app = loader.get_app(pipeline[-1])
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
  2016-12-21 14:54:05.406 TRACE nova     name=name, global_conf=global_conf).create()
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
  2016-12-21 14:54:05.406 TRACE nova     return self.object_type.invoke(self)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 146, in invoke
  2016-12-21 14:54:05.406 TRACE nova     return fix_call(context.object, context.global_conf, **context.local_conf)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
  2016-12-21 14:54:05.406 TRACE nova     val = callable(*args, **kw)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 219, in factory
  2016-12-21 14:54:05.406 TRACE nova     return cls()
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/compute/__init__.py", line 35, in __init__
  2016-12-21 14:54:05.406 TRACE nova     super(APIRouterV21, self).__init__(init_only)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 244, in __init__
  2016-12-21 14:54:05.406 TRACE nova     self._register_resources_check_inherits(mapper)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 260, in _register_resources_check_inherits
  2016-12-21 14:54:05.406 TRACE nova     for resource in ext.obj.get_resources():
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/compute/remote_consoles.py", line 207, in get_resources
  2016-12-21 14:54:05.406 TRACE nova     'remote-consoles', RemoteConsolesController(), parent=parent,
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/api/openstack/compute/remote_consoles.py", line 32, in __init__
  2016-12-21 14:54:05.406 TRACE nova     self.compute_api = compute.API()
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/compute/__init__.py", line 39, in API
  2016-12-21 14:54:05.406 TRACE nova     return importutils.import_object(class_name, *args, **kwargs)
  2016-12-21 14:54:05.406 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 44, in import_object
  2016-12-21 14:54:05.406 TRACE nova     return import_class(import_str)(*args, **kwargs)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/compute/api.py", line 211, in __init__
  2016-12-21 14:54:05.406 TRACE nova     self.key_manager = keymgr.API()
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/keymgr/__init__.py", line 75, in API
  2016-12-21 14:54:05.406 TRACE nova     return cls(conf)
  2016-12-21 14:54:05.406 TRACE nova   File "/opt/stack/nova/nova/keymgr/conf_key_mgr.py", line 67, in __init__
  2016-12-21 14:54:05.406 TRACE nova     raise ValueError(_('keymgr.fixed_key not defined'))
  2016-12-21 14:54:05.406 TRACE nova ValueError: keymgr.fixed_key not defined

  
  Environment
  ===========
  1. Ubuntu 16.04

  cat /etc/lsb-release
  DISTRIB_ID=Ubuntu
  DISTRIB_RELEASE=16.04
  DISTRIB_CODENAME=xenial
  DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"

  2. Devstack + Barbican plugin, both on stable/newton

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1651898/+subscriptions
References

[Bug 1651898] [NEW] Key manager configuration for ephemeral storage encryption is not backward compatible
From: Guang Yee, 2016-12-21