yahoo-eng-team team mailing list archive

[OpenStack Infra <1895723@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/759210
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=ba8c27f554682e1f2720fad1bff5cfa1b35017f2
Submitter: Zuul
Branch:    master

commit ba8c27f554682e1f2720fad1bff5cfa1b35017f2
Author: Mark Goddard <mark@xxxxxxxxxxxx>
Date:   Thu Oct 22 09:18:32 2020 +0100

    Fix keystone-startup.sh - remove Fernet key age check
    
    Currently we check the age of the primary Fernet key on Keystone
    startup, and fail if it is older than the rotation interval. While this
    may seem sensible, there are various reasons why the key may be older
    than this:
    
    * if the rotation interval is not a factor of the number of seconds in a
      week, the rotation schedule will be lumpy, with the last rotation
      being up to twice the nominal rotation interval
    * if a keystone host is unavailable at its scheduled rotation time,
      rotation will not happen. This may happen multiple times
    
    We could do several things to avoid this issue:
    
    1. remove the check on the age of the key
    2. multiply the rotation interval by some factor to determine the
       allowed key age
    
    This change goes for the more simple option 1. It also cleans up some
    terminology in the keystone-startup.sh script.
    
    Closes-Bug: #1895723
    
    Change-Id: I2c35f59ae9449cb1646e402e0a9f28ad61f918a8


** Changed in: kolla-ansible
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1895723

Title:
  Keystone is restarting due to stale primary key

Status in OpenStack Identity (keystone):
  Invalid
Status in kolla-ansible:
  Fix Released
Status in kolla-ansible train series:
  Triaged
Status in kolla-ansible ussuri series:
  Triaged
Status in kolla-ansible victoria series:
  Fix Released

Bug description:
  After restart of keystone's container, it keeps restarting. I found only this error in docker logs keystone:
  Running command: '/usr/bin/keystone-startup.sh -DFOREGROUND'
  + exec /usr/bin/keystone-startup.sh -DFOREGROUND
  + set -o errexit
  + set -o pipefail
  + TOKEN_DIR=/etc/keystone/fernet-keys
  + n=0
  + '[' '!' -f /etc/keystone/fernet-keys/0 ']'
  ++ ls -1 /etc/keystone/fernet-keys
  ++ sort -hr
  ++ head -n 1
  + TOKEN_PRIMARY=5
  ++ date +%s
  ++ date +%s -r /etc/keystone/fernet-keys/5
  + TOKEN_AGE=589164
  + '[' 589164 -gt 86400 ']'
  + echo 'ERROR: Primary token 5 is stale.'
  + exit 1

  Workaround is change expiration from 86400 to 864000 in
  /etc/kolla/keystone/keystone-startup.sh:

  # Compare if it's older than fernet_token_expiry and run key rotation if needed
  if [ "${TOKEN_AGE}" -gt "864000" ]; then
      echo "ERROR: Primary token ${TOKEN_PRIMARY} is stale."
      exit 1
  fi

  Regarding the comment in code, It should also run rotation of primary
  key. But this part is missing, it only throws an exception as
  mentioned. Or I would like to ask, why the primary key wasn't rotated
  automatically when it was needed.

  I am using 2 weeks old deployment of Ussuri, deployd by kolla-ansible
  on CentOS8.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1895723/+subscriptions