yahoo-eng-team team mailing list archive

[Lance Bragstad <lbragstad@xxxxxxxxx>]

Public bug reported:

The keystone-manage bootstrap command was designed to be idempotent.
Most everything in the bootstrap command is wrapped with a try/except to
handle cases where specific entities already exist (i.e. there is
already an admin project or an admin user from a previous bootstrap
run). This is important because bootstrap handles the creation of
administrator-like things in order to "bootstrap" a deployment. If
bootstrap wasn't idempotent, the side-effect of running it multiple
times would be catastrophic.

During an upgrade scenario, using OpenStack Ansible's rolling upgrade
support [0], from stable/newton to master, I noticed a very specific
case where bootstrap was not idempotent. Even if the admin user passed
to bootstrap already exists, the command will still attempt to update
it's password [1], even if the admin password hasn't changed. It does
the same thing with the user's enabled property. This somehow creates a
revocation event to be stored for that specific user [2]. As a result,
all tokens for the user specified in the bootstrap command will be
invalid once the upgrade happens, since OpenStack Ansible relies on
`keystone-manage bootstrap` during the upgrade.

This only affects the bootstrap user, but it can be considered a service
interruption since it is being done during an upgrade. We could look
into only updating the user's password, or enabled field, if and only if
they have changed. In that case, a revocation event *should* be
persisted since the bootstrap command is changing something about the
account. In the case where there is no change in password or enabled
status, tokens should still be able to be validated across releases.

I have documented the upgrade procedure and process in a separate
repository [3]

[0] https://review.openstack.org/#/c/384269/
[1] https://github.com/openstack/keystone/blob/1c60b1539cf63bba79711e237df496dfa094b2c5/keystone/cmd/cli.py#L226-L232
[2] http://cdn.pasteraw.com/9gz9964mwufyw3f98rv1mv1hqxezpis
[3] https://github.com/lbragstad/keystone-performance-upgrade

** Affects: keystone
     Importance: High
         Status: Confirmed

** Affects: keystone/newton
     Importance: Undecided
         Status: New


** Tags: newton-backport-potential upgrades

** Description changed:

  The keystone-manage bootstrap command was designed to be idempotent.
  Most everything in the bootstrap command is wrapped with a try/except to
  handle cases where specific entities already exist (i.e. there is
  already an admin project or an admin user from a previous bootstrap
  run). This is important because bootstrap handles the creation of
  administrator-like things in order to "bootstrap" a deployment. If
  bootstrap wasn't idempotent, the side-effect of running it multiple
  times would be catastrophic.
  
- During an upgrade scenario, using OpenStack Ansibles rolling upgrade
- support [1], from stable/newton to master, I noticed a very specific
+ During an upgrade scenario, using OpenStack Ansible's rolling upgrade
+ support [0], from stable/newton to master, I noticed a very specific
  case where bootstrap was not idempotent. Even if the admin user passed
  to bootstrap already exists, the command will still attempt to update
- it's password [0], even if the admin password hasn't changed. It does
+ it's password [1], even if the admin password hasn't changed. It does
  the same thing with the user's enabled property. This somehow creates a
- revocation event to be stored for that specific user [1]. As a result,
+ revocation event to be stored for that specific user [2]. As a result,
  all tokens for the user specified in the bootstrap command will be
  invalid once the upgrade happens, since OpenStack Ansible relies on
  `keystone-manage bootstrap` during the upgrade.
  
  This only affects the bootstrap user, but it can be considered a service
  interruption since it is being done during an upgrade. We could look
  into only updating the user's password, or enabled field, if and only if
  they have changed. In that case, a revocation event *should* be
- persisted. In the case where there is no change in password or enabled
+ persisted since the bootstrap command is changing something about the
+ account. In the case where there is no change in password or enabled
  status, tokens should still be able to be validated across releases.
  
+ I have documented the upgrade procedure and process in a separate
+ repository [3]
  
- [0] https://github.com/openstack/keystone/blob/1c60b1539cf63bba79711e237df496dfa094b2c5/keystone/cmd/cli.py#L226-L232
- [1] https://review.openstack.org/#/c/384269/
+ [0] https://review.openstack.org/#/c/384269/
+ [1] https://github.com/openstack/keystone/blob/1c60b1539cf63bba79711e237df496dfa094b2c5/keystone/cmd/cli.py#L226-L232
  [2] http://cdn.pasteraw.com/9gz9964mwufyw3f98rv1mv1hqxezpis
+ [3] https://github.com/lbragstad/keystone-performance-upgrade

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1647800

Title:
  keystone-manage bootstrap isn't completely idempotent

Status in OpenStack Identity (keystone):
  Confirmed
Status in OpenStack Identity (keystone) newton series:
  New

Bug description:
  The keystone-manage bootstrap command was designed to be idempotent.
  Most everything in the bootstrap command is wrapped with a try/except
  to handle cases where specific entities already exist (i.e. there is
  already an admin project or an admin user from a previous bootstrap
  run). This is important because bootstrap handles the creation of
  administrator-like things in order to "bootstrap" a deployment. If
  bootstrap wasn't idempotent, the side-effect of running it multiple
  times would be catastrophic.

  During an upgrade scenario, using OpenStack Ansible's rolling upgrade
  support [0], from stable/newton to master, I noticed a very specific
  case where bootstrap was not idempotent. Even if the admin user passed
  to bootstrap already exists, the command will still attempt to update
  it's password [1], even if the admin password hasn't changed. It does
  the same thing with the user's enabled property. This somehow creates
  a revocation event to be stored for that specific user [2]. As a
  result, all tokens for the user specified in the bootstrap command
  will be invalid once the upgrade happens, since OpenStack Ansible
  relies on `keystone-manage bootstrap` during the upgrade.

  This only affects the bootstrap user, but it can be considered a
  service interruption since it is being done during an upgrade. We
  could look into only updating the user's password, or enabled field,
  if and only if they have changed. In that case, a revocation event
  *should* be persisted since the bootstrap command is changing
  something about the account. In the case where there is no change in
  password or enabled status, tokens should still be able to be
  validated across releases.

  I have documented the upgrade procedure and process in a separate
  repository [3]

  [0] https://review.openstack.org/#/c/384269/
  [1] https://github.com/openstack/keystone/blob/1c60b1539cf63bba79711e237df496dfa094b2c5/keystone/cmd/cli.py#L226-L232
  [2] http://cdn.pasteraw.com/9gz9964mwufyw3f98rv1mv1hqxezpis
  [3] https://github.com/lbragstad/keystone-performance-upgrade

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1647800/+subscriptions