yahoo-eng-team team mailing list archive

[OpenStack Infra <1642457@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/413495
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=5b7c9a66f0aed860ea0776d4c5b42710d88fcb5f
Submitter: Jenkins
Branch:    master

commit 5b7c9a66f0aed860ea0776d4c5b42710d88fcb5f
Author: johnlinp <johnlinp@xxxxxxxxx>
Date:   Wed Dec 21 15:17:01 2016 +0800

    Handle disk write failure when doing Fernet key rotation
    
    _create_new_key() is broke down into 2 parts:
    
    1. _create_tmp_new_key()
    2. _become_valid_new_key()
    
    This can avoid empty Fernet keys when the write to the
    staged key fails. The _become_valid_new_key() is called
    only after a successful call to _create_tmp_new_key().
    
    Change-Id: Iaf33e2b291f13b9eb9464ef345a8664a634121ff
    Closes-Bug: #1642457
    Signed-off-by: John Lin <johnlinp@xxxxxxxxx>


** Changed in: keystone
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1642457

Title:
  Fernet rotate doesn't prevent rotation when disk is full

Status in OpenStack Identity (keystone):
  Fix Released

Bug description:
  When the root partition on any control node is full, the Fernet key on
  all control nodes will be empty. This will cause Keystone cannot auth
  anyone (500 Internal Server Error). Is that caused by Fernet key
  rotation?

  When I check the files in /etc/keystone/fernet-keys:

  root@control1:/etc/keystone/fernet-keys# ll
  total 40
  drwxr-s--- 2 keystone keystone 4096 Nov 17 00:00 ./
  drwxr-xr-x 5 keystone keystone 4096 Nov 10 11:24 ../
  -rw------- 1 keystone keystone    0 Nov 17 00:00 0
  -rw------- 1 keystone keystone   44 Nov 16 13:57 10
  -rw------- 1 keystone keystone   44 Nov  9 00:00 3
  -rw------- 1 keystone keystone   44 Nov 10 00:00 4
  -rw------- 1 keystone keystone   44 Nov 11 00:00 5
  -rw------- 1 keystone keystone   44 Nov 12 00:00 6
  -rw------- 1 keystone keystone   44 Nov 13 00:00 7
  -rw------- 1 keystone keystone   44 Nov 14 00:00 8
  -rw------- 1 keystone keystone   44 Nov 15 00:00 9

  Here is some of the Keystone logs when the master Fernet token is
  empty.

  [req-37cfe30f-5ff0-4d28-a187-066bf8031ad4 - - - - -] Fernet key must be 32 url-safe base64-encoded bytes.
  Traceback (most recent call last):
    File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/common/wsgi.py", line 249, in __call__
      result = method(context, **params)
    File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/auth/controllers.py", line 416, in authenticate_for_token
      parent_audit_id=token_audit_id)
    File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/common/manager.py", line 124, in wrapped
      __ret_val = __f(*args, **kwargs)
    File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/provider.py", line 388, in issue_v3_token
      parent_audit_id)
    File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/providers/fernet/core.py", line 44, in issue_v3_token
      *args, **kwargs)
    File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/providers/common.py", line 623, in issue_v3_token
      token_id = self._get_token_id(token_data)
    File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/providers/fernet/core.py", line 201, in _get_token_id
      access_token_id=access_token_id
    File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/providers/fernet/token_formatters.py", line 165, in create_token
      token = self.pack(serialized_payload)
    File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/providers/fernet/token_formatters.py", line 75, in pack
      return self.crypto.encrypt(payload).rstrip(b'=').decode('utf-8')
    File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/providers/fernet/token_formatters.py", line 64, in crypto
      fernet_instances = [fernet.Fernet(key) for key in keys]
    File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/cryptography/fernet.py", line 37, in __init__
      "Fernet key must be 32 url-safe base64-encoded bytes."
  ValueError: Fernet key must be 32 url-safe base64-encoded bytes.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1642457/+subscriptions