yahoo-eng-team team mailing list archive

[Divya K Konoor <dikonoor@xxxxxxxxxx>]

Public bug reported:

The problem being reported is very similar to the one reported at
https://bugs.launchpad.net/keystone/+bug/1642457 but not the same.

Step to reproduce (Not sure of the chances of being able to reproduce
this again but these were the steps that happened when the problem was
found):

1. Fernet token rotation is configured in this environment to run via a
cron job every 3 hours. The primary key when things were working was 58.
The system (where OpenStack was installed) went out of memory and in an
attempt to recover, a reboot was initiated. As fate would have it, the
reboot was initiated at 14.58 and the key rotate was to happen at 15:02.
Keystone logs don't have any logging between 14.58 and 15:15. When the
system was up, token issue was failing with


File "/usr/lib64/python2.7/site-packages/cryptography/fernet.py", line 37, in __init__
2017-10-26 15:46:30.613 4767 ERROR keystone.common.wsgi   "Fernet key must be 32 url-safe base64-encoded bytes."
2017-10-26 15:46:30.613 4767 ERROR keystone.common.wsgi ValueError: Fernet key must be 32 url-safe base64-encoded bytes.

2. Soon after the above was noticed , a key rotation was attempted to
see if that fixes anything (/usr/bin/keystone-manage fernet_rotate
--keystone-user keystone --keystone-group keystone). And it did not.

3. When the fernet-keys directory was checked after step 3, an empty
primary key file was found(60) . No other files were empty. This file
was manually deleted after which the primary key became 59 and token
issue continued to work.


System has no problem with disk space.

** Affects: keystone
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1728907

Title:
  Empty Fernet Key Files causing problems with token issue

Status in OpenStack Identity (keystone):
  New

Bug description:
  The problem being reported is very similar to the one reported at
  https://bugs.launchpad.net/keystone/+bug/1642457 but not the same.

  Step to reproduce (Not sure of the chances of being able to reproduce
  this again but these were the steps that happened when the problem was
  found):

  1. Fernet token rotation is configured in this environment to run via
  a cron job every 3 hours. The primary key when things were working was
  58. The system (where OpenStack was installed) went out of memory and
  in an attempt to recover, a reboot was initiated. As fate would have
  it, the reboot was initiated at 14.58 and the key rotate was to happen
  at 15:02. Keystone logs don't have any logging between 14.58 and
  15:15. When the system was up, token issue was failing with

  
  File "/usr/lib64/python2.7/site-packages/cryptography/fernet.py", line 37, in __init__
  2017-10-26 15:46:30.613 4767 ERROR keystone.common.wsgi   "Fernet key must be 32 url-safe base64-encoded bytes."
  2017-10-26 15:46:30.613 4767 ERROR keystone.common.wsgi ValueError: Fernet key must be 32 url-safe base64-encoded bytes.

  2. Soon after the above was noticed , a key rotation was attempted to
  see if that fixes anything (/usr/bin/keystone-manage fernet_rotate
  --keystone-user keystone --keystone-group keystone). And it did not.

  3. When the fernet-keys directory was checked after step 3, an empty
  primary key file was found(60) . No other files were empty. This file
  was manually deleted after which the primary key became 59 and token
  issue continued to work.

  
  System has no problem with disk space.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1728907/+subscriptions