yahoo-eng-team team mailing list archive

[Radosław Piliszek <1895723@xxxxxxxxxxxxxxxxxx>]

** Also affects: kolla-ansible/ussuri
   Importance: Undecided
       Status: New

** Also affects: kolla-ansible/train
   Importance: Undecided
       Status: New

** Also affects: kolla-ansible/victoria
   Importance: High
     Assignee: Michal Nasiadka (mnasiadka)
       Status: In Progress

** Changed in: kolla-ansible/ussuri
   Importance: Undecided => High

** Changed in: kolla-ansible/train
   Importance: Undecided => High

** Changed in: kolla-ansible/ussuri
       Status: New => Triaged

** Changed in: kolla-ansible/train
       Status: New => Triaged

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1895723

Title:
  Keystone is restarting due to stale primary key

Status in OpenStack Identity (keystone):
  Invalid
Status in kolla-ansible:
  In Progress
Status in kolla-ansible train series:
  Triaged
Status in kolla-ansible ussuri series:
  Triaged
Status in kolla-ansible victoria series:
  In Progress

Bug description:
  After restart of keystone's container, it keeps restarting. I found only this error in docker logs keystone:
  Running command: '/usr/bin/keystone-startup.sh -DFOREGROUND'
  + exec /usr/bin/keystone-startup.sh -DFOREGROUND
  + set -o errexit
  + set -o pipefail
  + TOKEN_DIR=/etc/keystone/fernet-keys
  + n=0
  + '[' '!' -f /etc/keystone/fernet-keys/0 ']'
  ++ ls -1 /etc/keystone/fernet-keys
  ++ sort -hr
  ++ head -n 1
  + TOKEN_PRIMARY=5
  ++ date +%s
  ++ date +%s -r /etc/keystone/fernet-keys/5
  + TOKEN_AGE=589164
  + '[' 589164 -gt 86400 ']'
  + echo 'ERROR: Primary token 5 is stale.'
  + exit 1

  Workaround is change expiration from 86400 to 864000 in
  /etc/kolla/keystone/keystone-startup.sh:

  # Compare if it's older than fernet_token_expiry and run key rotation if needed
  if [ "${TOKEN_AGE}" -gt "864000" ]; then
      echo "ERROR: Primary token ${TOKEN_PRIMARY} is stale."
      exit 1
  fi

  Regarding the comment in code, It should also run rotation of primary
  key. But this part is missing, it only throws an exception as
  mentioned. Or I would like to ask, why the primary key wasn't rotated
  automatically when it was needed.

  I am using 2 weeks old deployment of Ussuri, deployd by kolla-ansible
  on CentOS8.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1895723/+subscriptions