yahoo-eng-team team mailing list archive

[Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>]

** Changed in: keystone
       Status: Fix Committed => Fix Released

** Changed in: keystone
    Milestone: None => icehouse-1

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Keystone.
https://bugs.launchpad.net/bugs/1246489

Title:
  races in assignment sql backend cause spurious 404s and transient
  errors while granting roles

Status in OpenStack Identity (Keystone):
  Fix Released

Bug description:
  There are numerous schedules that lead to inconsistent data in the
  various grant tables and role table. These races arise from a lack of
  synchronization during grant table modification (i.e., during grant
  record creation + deletion and role deletion) and deletions of records
  that grant records refer to (i.e., user, group, domain, and project
  records).

  For example, consider a project and a role that are related by a
  UserProjectGrant, which are concurrently deleted. In the
  implementation of delete_role, the project ids that are enumerated in
  one transaction (in
  keystone.assignment.backends.sql.Assignment.delete_role) are asserted
  to exist during another transaction (the call to self._get_project in
  delete_grant in the same class) and are referred to in two more
  transactions (_get_metadata and _create_metadata or _update_metadata).
  Suppose that after the enumeration of a project_id, but before the
  assertion of its existence, a transaction is committed that deletes
  the project_id. The result will be a 404 (project not found) and the
  role isn't deleted. Note that a subsequent delete_role API call will
  remove the Role record without hitting a 404 since the
  UserProjectGrant record will have been deleted by the delete_project
  transaction.

  Suppose two roles are being granted to the same user on the same
  project concurrently. Further suppose that prior to these two grants,
  the user didn't have any roles on that project. In
  Assignment.create_grant, a UserProjectGrant record is either updated
  or created anew depending on whether or not a UserProjectGrant record
  already exists for the (user, project) pair passed in. Since the
  existence check and the creation of a new record are performed in
  separate transactions, two (user, project) UserProjectGrant records
  can be created given the correct interleaving of the concurrent
  create_grant executions. Having the same user and project values,
  these two records would violate the primary key constraints on the
  UserProjectGrant table. Assuming the DBMS enforces the primary key
  constraints, then one of the create_grant requests will fail.

  Another example is described in
  https://review.openstack.org/#/c/50767/.

  Although I haven't been able to come up with an example yet,  I
  suspect that grants or roles might become undeletable because of
  inconsistent data.

  The races pertaining to data owned by the keystone.assignment module
  (i.e., tenants, roles & grants) can be fixed by judicious use of
  transactions in keystone.assignment.backends.sql.Assignment. In
  particular, using a single transaction per API-level operation. The
  races pertaining to data owned by another module (i.e., groups and
  users are owned by keystone.identity) can't be fixed with transactions
  since the other module might be using a different backend (e.g.,
  LDAP). Those races can't be outright eliminated but they can be turned
  into transient problems by removing existence assertions during
  deletion operations (such as https://review.openstack.org/#/c/50767/)
  and adding existence assertions to creation operations.

  Running multiple keystone processes, which can be done either via
  apache or https://review.openstack.org/#/c/42967/,  aggravates the
  likelihood of these races.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1246489/+subscriptions