yahoo-eng-team team mailing list archive

[John Lin <1642457@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

When the root partition on any control node is full, the Fernet key on
all control nodes will be empty. This will cause Keystone cannot auth
anyone (500 Internal Server Error). Is that caused by Fernet key
rotation?

When I check the files in /etc/keystone/fernet-keys:

root@control1:/etc/keystone/fernet-keys# ll
total 40
drwxr-s--- 2 keystone keystone 4096 Nov 17 00:00 ./
drwxr-xr-x 5 keystone keystone 4096 Nov 10 11:24 ../
-rw------- 1 keystone keystone    0 Nov 17 00:00 0
-rw------- 1 keystone keystone   44 Nov 16 13:57 10
-rw------- 1 keystone keystone   44 Nov  9 00:00 3
-rw------- 1 keystone keystone   44 Nov 10 00:00 4
-rw------- 1 keystone keystone   44 Nov 11 00:00 5
-rw------- 1 keystone keystone   44 Nov 12 00:00 6
-rw------- 1 keystone keystone   44 Nov 13 00:00 7
-rw------- 1 keystone keystone   44 Nov 14 00:00 8
-rw------- 1 keystone keystone   44 Nov 15 00:00 9

Here is some of the Keystone logs when the master Fernet token is empty.

[req-37cfe30f-5ff0-4d28-a187-066bf8031ad4 - - - - -] Fernet key must be 32 url-safe base64-encoded bytes.
Traceback (most recent call last):
  File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/common/wsgi.py", line 249, in __call__
    result = method(context, **params)
  File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/auth/controllers.py", line 416, in authenticate_for_token
    parent_audit_id=token_audit_id)
  File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/common/manager.py", line 124, in wrapped
    __ret_val = __f(*args, **kwargs)
  File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/provider.py", line 388, in issue_v3_token
    parent_audit_id)
  File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/providers/fernet/core.py", line 44, in issue_v3_token
    *args, **kwargs)
  File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/providers/common.py", line 623, in issue_v3_token
    token_id = self._get_token_id(token_data)
  File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/providers/fernet/core.py", line 201, in _get_token_id
    access_token_id=access_token_id
  File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/providers/fernet/token_formatters.py", line 165, in create_token
    token = self.pack(serialized_payload)
  File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/providers/fernet/token_formatters.py", line 75, in pack
    return self.crypto.encrypt(payload).rstrip(b'=').decode('utf-8')
  File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/providers/fernet/token_formatters.py", line 64, in crypto
    fernet_instances = [fernet.Fernet(key) for key in keys]
  File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/cryptography/fernet.py", line 37, in __init__
    "Fernet key must be 32 url-safe base64-encoded bytes."
ValueError: Fernet key must be 32 url-safe base64-encoded bytes.

** Affects: keystone
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1642457

Title:
  Root partition full on any control node makes master Fernet keys empty
  on all nodes

Status in OpenStack Identity (keystone):
  New

Bug description:
  When the root partition on any control node is full, the Fernet key on
  all control nodes will be empty. This will cause Keystone cannot auth
  anyone (500 Internal Server Error). Is that caused by Fernet key
  rotation?

  When I check the files in /etc/keystone/fernet-keys:

  root@control1:/etc/keystone/fernet-keys# ll
  total 40
  drwxr-s--- 2 keystone keystone 4096 Nov 17 00:00 ./
  drwxr-xr-x 5 keystone keystone 4096 Nov 10 11:24 ../
  -rw------- 1 keystone keystone    0 Nov 17 00:00 0
  -rw------- 1 keystone keystone   44 Nov 16 13:57 10
  -rw------- 1 keystone keystone   44 Nov  9 00:00 3
  -rw------- 1 keystone keystone   44 Nov 10 00:00 4
  -rw------- 1 keystone keystone   44 Nov 11 00:00 5
  -rw------- 1 keystone keystone   44 Nov 12 00:00 6
  -rw------- 1 keystone keystone   44 Nov 13 00:00 7
  -rw------- 1 keystone keystone   44 Nov 14 00:00 8
  -rw------- 1 keystone keystone   44 Nov 15 00:00 9

  Here is some of the Keystone logs when the master Fernet token is
  empty.

  [req-37cfe30f-5ff0-4d28-a187-066bf8031ad4 - - - - -] Fernet key must be 32 url-safe base64-encoded bytes.
  Traceback (most recent call last):
    File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/common/wsgi.py", line 249, in __call__
      result = method(context, **params)
    File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/auth/controllers.py", line 416, in authenticate_for_token
      parent_audit_id=token_audit_id)
    File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/common/manager.py", line 124, in wrapped
      __ret_val = __f(*args, **kwargs)
    File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/provider.py", line 388, in issue_v3_token
      parent_audit_id)
    File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/providers/fernet/core.py", line 44, in issue_v3_token
      *args, **kwargs)
    File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/providers/common.py", line 623, in issue_v3_token
      token_id = self._get_token_id(token_data)
    File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/providers/fernet/core.py", line 201, in _get_token_id
      access_token_id=access_token_id
    File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/providers/fernet/token_formatters.py", line 165, in create_token
      token = self.pack(serialized_payload)
    File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/providers/fernet/token_formatters.py", line 75, in pack
      return self.crypto.encrypt(payload).rstrip(b'=').decode('utf-8')
    File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/providers/fernet/token_formatters.py", line 64, in crypto
      fernet_instances = [fernet.Fernet(key) for key in keys]
    File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/cryptography/fernet.py", line 37, in __init__
      "Fernet key must be 32 url-safe base64-encoded bytes."
  ValueError: Fernet key must be 32 url-safe base64-encoded bytes.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1642457/+subscriptions