yahoo-eng-team team mailing list archive

[OpenStack Infra <1895723@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/754085
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=d78673e77fc376274192a8f7b547ce90d6bdc151
Submitter: Zuul
Branch:    master

commit d78673e77fc376274192a8f7b547ce90d6bdc151
Author: Michal Nasiadka <mnasiadka@xxxxxxxxx>
Date:   Thu Sep 24 15:16:55 2020 +0200

    Fix keystone-startup.sh
    
    keystone-startup.sh is using fernet_token_expiry instead of
    fernet_key_rotation_interval - which effects in restart loop of keystone
    containers - when restarted after 2-3 days.
    
    Closes-Bug: #1895723
    
    Change-Id: Ifff77af3d25d9dc659fff34f2ae3c6f2670df0f4


** Changed in: kolla-ansible
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1895723

Title:
  Keystone is restarting due to stale primary key

Status in OpenStack Identity (keystone):
  Invalid
Status in kolla-ansible:
  Fix Released
Status in kolla-ansible train series:
  In Progress
Status in kolla-ansible ussuri series:
  In Progress
Status in kolla-ansible victoria series:
  Fix Released

Bug description:
  After restart of keystone's container, it keeps restarting. I found only this error in docker logs keystone:
  Running command: '/usr/bin/keystone-startup.sh -DFOREGROUND'
  + exec /usr/bin/keystone-startup.sh -DFOREGROUND
  + set -o errexit
  + set -o pipefail
  + TOKEN_DIR=/etc/keystone/fernet-keys
  + n=0
  + '[' '!' -f /etc/keystone/fernet-keys/0 ']'
  ++ ls -1 /etc/keystone/fernet-keys
  ++ sort -hr
  ++ head -n 1
  + TOKEN_PRIMARY=5
  ++ date +%s
  ++ date +%s -r /etc/keystone/fernet-keys/5
  + TOKEN_AGE=589164
  + '[' 589164 -gt 86400 ']'
  + echo 'ERROR: Primary token 5 is stale.'
  + exit 1

  Workaround is change expiration from 86400 to 864000 in
  /etc/kolla/keystone/keystone-startup.sh:

  # Compare if it's older than fernet_token_expiry and run key rotation if needed
  if [ "${TOKEN_AGE}" -gt "864000" ]; then
      echo "ERROR: Primary token ${TOKEN_PRIMARY} is stale."
      exit 1
  fi

  Regarding the comment in code, It should also run rotation of primary
  key. But this part is missing, it only throws an exception as
  mentioned. Or I would like to ask, why the primary key wasn't rotated
  automatically when it was needed.

  I am using 2 weeks old deployment of Ussuri, deployd by kolla-ansible
  on CentOS8.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1895723/+subscriptions