yahoo-eng-team team mailing list archive

[Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>]

** Changed in: keystone
       Status: Fix Committed => Fix Released

** Changed in: keystone
    Milestone: None => juno-3

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Keystone.
https://bugs.launchpad.net/bugs/1188378

Title:
  keystone.token.backends.sql uses a single delete command to flush
  expired tokens causing replication lag and potential deadlocks

Status in OpenStack Identity (Keystone):
  Fix Released

Bug description:
      def flush_expired_tokens(self):
          session = self.get_session()

          query = session.query(TokenModel)
          query = query.filter(TokenModel.expires < timeutils.utcnow())
          query.delete(synchronize_session=False)

          session.flush()

  
  This will basically result in this command being run (for MySQL anyway):

  DELETE FROM token WHERE expires < '2013-06-05 21:02:00';

  If there are millions of rows to delete, this will take a long time,
  which in turn will translate into replication lag if there is an
  active replication slave.

  Also, because the keystone tokens are somewhat random 64 byte strings,
  there will be gaps in the InnoDB key storage. In order to preserve
  transactional integrity while deleting these rows, InnoDB has to lock
  these gaps. So new tokens that fit into these gaps will have to wait
  for the entire delete to finish to be inserted.

  A much more healthy approach is to walk through the table deleting in
  small batches:

  q = session.query(TokenModel).filter_by(TokenModel.expires < timeutils.utcnow())).limit(10).subquery()
  while(session.query(TokenModel).filter(TokenModel.id.in_(q)).delete())
      pass

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1188378/+subscriptions