yahoo-eng-team team mailing list archive

[OpenStack Infra <1549726@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/287020
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=0b18edab226f6e9dc531febd4eb6f65ccd3c031e
Submitter: Jenkins
Branch:    master

commit 0b18edab226f6e9dc531febd4eb6f65ccd3c031e
Author: Divya <dikonoor@xxxxxxxxxx>
Date:   Wed Mar 2 08:05:42 2016 +0100

    Race condition in keystone domain config
    
    This bug fixes a race condition in the domains_config
    decorator. The race condition occurs when more than
    one thread accesses the decorator. The first thread
    sets the configured flag to True before proceeding with
    driver load leading the second thread to use the default
    driver. This fix ensures that the second thread waits for
    the first thread to finish configuration before it uses
    the driver.
    
    Change-Id: I0289a4d38e0d30d39c67e29bf77b0a89d1dd23f6
    Closes-Bug: 1549726


** Changed in: keystone
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1549726

Title:
  Race condition in keystone domain config

Status in OpenStack Identity (keystone):
  Fix Released

Bug description:
  This is a very difficult to reproduce bug but occurs nevertheless and
  can be observed when we switch the backend drivers.

  Steps to reproduce:
  1. Switch the backened driver in keystone conf file from one driver to another . Restart keystone
  2. Immediately (if you wait for few seconds, this cannot be reproduced) , make calls that in turn access the keystone methods  /keystone/identity/core >> get_group and list_users_in_group. It doesn't have to be exactly these two. It can be any two similar methods in identity.core.py which uses the @domains_configured decorator.

  https://github.com/openstack/keystone/blob/master/keystone/identity/core.py#L100

  Invoke two methods that use this decorators and these method
  invocations must be almost parallel. Both the methods hit the
  following flow where the race condition occurs:

  
  def domains_configured(f):
     """Wraps API calls to lazy load domain configs after init.
      """
      @functools.wraps(f)
      def wrapper(self, *args, **kwargs):
          if (not self.domain_configs.configured and
                  CONF.identity.domain_specific_drivers_enabled):
              self.domain_configs.setup_domain_drivers(
                  self.driver, self.resource_api)
          else:
              LOG.error('domains will not be configured')
          return f(self, *args, **kwargs)
      return wrapper

  def setup_domain_drivers(self, standard_driver, resource_api):
          # This is called by the api call wrapper
         self.configured = True
          self.driver = standard_driver

  .....

  When the first call is placed, it sets self.configured to True and
  then proceeds towards loading the driver that corresponds to the
  domain-..However, the second request call assumes the the driver load
  is already complete (purely based on the value set to self.configured
  - which is True even though driver is not really loaded). It thus ends
  up using the default driver (ie driver which is not domain specific )
  and retrieves the values.

  There should be some synchronization logic added inside
  domains_configured (or one of the internal methods) so that incorrect
  backend driver is not used by a request.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1549726/+subscriptions