yahoo-eng-team team mailing list archive

[OpenStack Infra <1728907@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/546785
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=f2a210e3fe991885833a627a9e45b6f0e4669b34
Submitter: Zuul
Branch:    master

commit f2a210e3fe991885833a627a9e45b6f0e4669b34
Author: Gage Hugo <gagehugo@xxxxxxxxx>
Date:   Wed Feb 21 15:45:17 2018 -0600

    Handle empty token key files
    
    In some rare cases, an empty key file can get created within the fernet
    key repository. When keystone tries to load the keys from disk, it will
    fail with an invalid fernet key ValueError.
    
    This change adds a check for empty files with a valid numerical name
    within the key repository when rotating keys and loading keys. If an
    empty file exists, it will be ignored when loading keys, reported in the
    logs, and overwritten with a valid key upon rotation.
    
    Change-Id: Ic19dd02d38e8f6a05c8951ec3dd13659aab98259
    Closes-Bug: 1728907


** Changed in: keystone
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1728907

Title:
  Empty Fernet Key Files causing problems with token issue

Status in OpenStack Identity (keystone):
  Fix Released

Bug description:
  The problem being reported is very similar to the one reported at
  https://bugs.launchpad.net/keystone/+bug/1642457 but not the same.

  Step to reproduce (Not sure of the chances of being able to reproduce
  this again but these were the steps that happened when the problem was
  found):

  1. Fernet token rotation is configured in this environment to run via
  a cron job every 3 hours. The primary key when things were working was
  58. The system (where OpenStack was installed) went out of memory and
  in an attempt to recover, a reboot was initiated. As fate would have
  it, the reboot was initiated at 14.58 and the key rotate was to happen
  at 15:02. Keystone logs don't have any logging between 14.58 and
  15:15. When the system was up, token issue was failing with

  
  File "/usr/lib64/python2.7/site-packages/cryptography/fernet.py", line 37, in __init__
  2017-10-26 15:46:30.613 4767 ERROR keystone.common.wsgi   "Fernet key must be 32 url-safe base64-encoded bytes."
  2017-10-26 15:46:30.613 4767 ERROR keystone.common.wsgi ValueError: Fernet key must be 32 url-safe base64-encoded bytes.

  2. Soon after the above was noticed , a key rotation was attempted to
  see if that fixes anything (/usr/bin/keystone-manage fernet_rotate
  --keystone-user keystone --keystone-group keystone). And it did not.

  3. When the fernet-keys directory was checked after step 3, an empty
  primary key file was found(60) . No other files were empty. This file
  was manually deleted after which the primary key became 59 and token
  issue continued to work.

  
  System has no problem with disk space.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1728907/+subscriptions