yahoo-eng-team team mailing list archive

[Divya K Konoor <dikonoor@xxxxxxxxxx>]

Public bug reported:

This is a very difficult to reproduce bug but occurs nevertheless and
can be observed when we switch the backend drivers.

Steps to reproduce:
1. Switch the backened driver in keystone conf file from one driver to another . Restart keystone
2. Immediately (if you wait for few seconds, this cannot be reproduced) , make calls that in turn access the keystone methods  /keystone/identity/core >> get_group and list_users_in_group. It doesn't have to be exactly these two. It can be any two similar methods in identity.core.py which uses the @domains_configured decorator.

https://github.com/openstack/keystone/blob/master/keystone/identity/core.py#L100

Invoke two methods that use this decorators and these method invocations
must be almost parallel. Both the methods hit the following flow where
the race condition occurs:


def domains_configured(f):
   """Wraps API calls to lazy load domain configs after init.
    """
    @functools.wraps(f)
    def wrapper(self, *args, **kwargs):
        if (not self.domain_configs.configured and
                CONF.identity.domain_specific_drivers_enabled):
            self.domain_configs.setup_domain_drivers(
                self.driver, self.resource_api)
        else:
            LOG.error('domains will not be configured')
        return f(self, *args, **kwargs)
    return wrapper

def setup_domain_drivers(self, standard_driver, resource_api):
        # This is called by the api call wrapper
       self.configured = True
        self.driver = standard_driver

.....

When the first call is placed, it sets self.configured to True and then
proceeds towards loading the driver that corresponds to the
domain-..However, the second request call assumes the the driver load is
already complete (purely based on the value set to self.configured -
which is True even though driver is not really loaded). It thus ends up
using the default driver (ie driver which is not domain specific ) and
retrieves the values.

There should be some synchronization logic added inside
domains_configured (or one of the internal methods) so that incorrect
backend driver is not used by a request.

** Affects: keystone
     Importance: Undecided
     Assignee: Divya K Konoor (dikonoor)
         Status: New

** Changed in: keystone
     Assignee: (unassigned) => Divya K Konoor (dikonoor)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1549726

Title:
  Race condition in keystone domain config

Status in OpenStack Identity (keystone):
  New

Bug description:
  This is a very difficult to reproduce bug but occurs nevertheless and
  can be observed when we switch the backend drivers.

  Steps to reproduce:
  1. Switch the backened driver in keystone conf file from one driver to another . Restart keystone
  2. Immediately (if you wait for few seconds, this cannot be reproduced) , make calls that in turn access the keystone methods  /keystone/identity/core >> get_group and list_users_in_group. It doesn't have to be exactly these two. It can be any two similar methods in identity.core.py which uses the @domains_configured decorator.

  https://github.com/openstack/keystone/blob/master/keystone/identity/core.py#L100

  Invoke two methods that use this decorators and these method
  invocations must be almost parallel. Both the methods hit the
  following flow where the race condition occurs:

  
  def domains_configured(f):
     """Wraps API calls to lazy load domain configs after init.
      """
      @functools.wraps(f)
      def wrapper(self, *args, **kwargs):
          if (not self.domain_configs.configured and
                  CONF.identity.domain_specific_drivers_enabled):
              self.domain_configs.setup_domain_drivers(
                  self.driver, self.resource_api)
          else:
              LOG.error('domains will not be configured')
          return f(self, *args, **kwargs)
      return wrapper

  def setup_domain_drivers(self, standard_driver, resource_api):
          # This is called by the api call wrapper
         self.configured = True
          self.driver = standard_driver

  .....

  When the first call is placed, it sets self.configured to True and
  then proceeds towards loading the driver that corresponds to the
  domain-..However, the second request call assumes the the driver load
  is already complete (purely based on the value set to self.configured
  - which is True even though driver is not really loaded). It thus ends
  up using the default driver (ie driver which is not domain specific )
  and retrieves the values.

  There should be some synchronization logic added inside
  domains_configured (or one of the internal methods) so that incorrect
  backend driver is not used by a request.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1549726/+subscriptions