yahoo-eng-team team mailing list archive

[Colleen Murphy <colleen@xxxxxxxxxxx>]

I'm going to mark this as invalid for keystone. Keystone does expect the
setup to happen in a specific order, and it is up to the operator or
deployment tool to meet those requirements. Keystone can't accommodate
implementation differences between different deployment tools.

** Changed in: keystone
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1832005

Title:
  Race during Keystone deploy (fernet)

Status in OpenStack Identity (keystone):
  Invalid
Status in kolla-ansible:
  In Progress

Bug description:
  RedHat 7.6 OpenStack Ocata
  Custom build Docker images using binary type.
  Keystone configured to use fernet tokens.

  
  When keystone container is started it expects directory and tokens to be present.
  This is checked by the following code https://github.com/openstack/keystone/blob/3d2b293d7edfb0bd4bdec9b33abc63d1308e10bd/keystone/token/providers/fernet/core.py#L36

  In some rare scenarios, keystone container fails with

   2019-05-31 17:26:39.620011 File "/usr/lib/python2.7/site-packages/keystone/token/providers/fernet/core.py", line 45, in _init_
   2019-05-31 17:26:39.620106 'Fernet keys.') % subs)
   2019-05-31 17:26:39.620126 SystemExit: /etc/keystone/fernet-keys/ does not contain keys, use keystone-manage fernet_setup to create Fernet keys.

  When inspecting directory, keys are there

   (keystone)[root@osc1 fernet-keys]# ls -la
   total 12
   drwxrwx---. 2 keystone keystone 33 May 31 17:26 .
   drwxr-x---. 1 root keystone 61 May 31 17:26 ..
   rw------. 1 keystone keystone 44 May 31 17:26 0
   rw------. 1 keystone keystone 44 May 31 17:26 1
   rw------. 1 keystone keystone 44 May 31 17:26 2

  Please note that the files creation time is the same as error message
  time (17:26).

  Upon inspection of the ansible/roles/keystone/tasks/deploy.yml one can find that
  init_fernet.yml task is executed after flush_handlers. When handlers are run, containers are created or restarted. 

  The obvious option would be to move init_fernet before handlers, but
  this task does require keystone_ssh and keystone_fernet to be up and
  running.

  The solutions could include:
   - Changes in keystone itself to retry initialization as long as the keys are missing
   - Changes in keystone to fail in a way that the container will restart
   - Changes in kolla-ansible to enforce fernet init before keystone container starts.

  
  The bug is found on Ocata but upon Ansible manifests inspection it could happen on master as well.

  
  Workaround:
  Restart Keystone container.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1832005/+subscriptions