yahoo-eng-team team mailing list archive

[Pavlo Shchelokovskyy <1938323@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

NOTE - this is happening only on Queens, and possibly earlier, and is
already silently fixed in Rocky as part of the major refactor of token
model. I am posing this issue here so that anyone still running Queens
has a reference and probably a patch to apply.


In Queens release, if I create a token using a nocatalog option:

  curl -X POST <keystone-endpoint>/v3/auth/tokens?nocatalog

and then use this token to e.g. list servers with details

  curl -X GET <nova-endpoint>/v2.1/servers/detail

I get 500 error from nova, with nova api logs containing

  ERROR nova.api.openstack EmptyCatalog: The service catalog is empty.

When repeating the same request with the same token after 5-10 minutes, the token starts to work.
Tokens generated with catalog are working as well.

AFAIU this goes down to token caching - in Queens tokens are
cached/memoized with catalog - or without it if token was requested w/o
catalog. Then on token validation, the token validation response takes
the token - with or without catalog - from cache and returns it to
caller with minimal processing - e.g. removing catalog if token
validation call asked for it. It does not however ensures that the
catalog is present otherwise.

This breaks some other services like Nova, that expects the catalog to
be present in the request context constructed from the
keystonemiddleware results. Nova needs this for example to make API
requests to other services - exactly what happens in server/details call
where it has to ask Neutron for some network info about instances.

After the cache is invalidated, the catalog starts to be generated for
token validation response anew, and everything starts to work as
expected.

** Affects: keystone
     Importance: Undecided
     Assignee: Pavlo Shchelokovskyy (pshchelo)
         Status: In Progress

** Changed in: keystone
       Status: New => In Progress

** Changed in: keystone
     Assignee: (unassigned) => Pavlo Shchelokovskyy (pshchelo)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1938323

Title:
  [Queens] tokens generated with nocatalog are not usable in some
  requests

Status in OpenStack Identity (keystone):
  In Progress

Bug description:
  NOTE - this is happening only on Queens, and possibly earlier, and is
  already silently fixed in Rocky as part of the major refactor of token
  model. I am posing this issue here so that anyone still running Queens
  has a reference and probably a patch to apply.

  
  In Queens release, if I create a token using a nocatalog option:

    curl -X POST <keystone-endpoint>/v3/auth/tokens?nocatalog

  and then use this token to e.g. list servers with details

    curl -X GET <nova-endpoint>/v2.1/servers/detail

  I get 500 error from nova, with nova api logs containing

    ERROR nova.api.openstack EmptyCatalog: The service catalog is empty.

  When repeating the same request with the same token after 5-10 minutes, the token starts to work.
  Tokens generated with catalog are working as well.

  AFAIU this goes down to token caching - in Queens tokens are
  cached/memoized with catalog - or without it if token was requested
  w/o catalog. Then on token validation, the token validation response
  takes the token - with or without catalog - from cache and returns it
  to caller with minimal processing - e.g. removing catalog if token
  validation call asked for it. It does not however ensures that the
  catalog is present otherwise.

  This breaks some other services like Nova, that expects the catalog to
  be present in the request context constructed from the
  keystonemiddleware results. Nova needs this for example to make API
  requests to other services - exactly what happens in server/details
  call where it has to ask Neutron for some network info about
  instances.

  After the cache is invalidated, the catalog starts to be generated for
  token validation response anew, and everything starts to work as
  expected.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1938323/+subscriptions