yahoo-eng-team team mailing list archive

[yatin <ykarel@xxxxxxxxxx>]

Public bug reported:

This happens with oslo.config >= 6.3.0([1]) + python3 + "keystone" paste_deploy + current glance(before https://review.openstack.org/#/c/532503/10/glance/common/store_utils.py@30 it works)
Testing in devstack: https://review.openstack.org/#/c/596380/

The glance api service fails to start with below Error, reproducing here: https://review.openstack.org/#/c/596380/:-
ERROR: dictionary changed size during iteration , see logs below

Failure logs from job:- http://logs.openstack.org/80/596380/2/check
/tempest-full-
py3/514fa29/controller/logs/screen-g-api.txt.gz#_Aug_27_07_26_10_698243


The Runtime Error is returned at keystonemiddleware:- https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token/__init__.py#L551
Adding code snippet here:-
if self._conf.oslo_conf_obj != cfg.CONF:   <-- Fails here
    oslo_cache.configure(self._conf.oslo_conf_obj)

So with pdb found that an additional key(fatal_deprecations) was added
to cfg.CONF at ^^, so Error is returned in python3. With python2 same
key is added but no Error.

There are multiple ways to avoid it, like use the paste_deploy configuration that works(ex: keystone+cachemanagement), use oslo.config <= 6.2.0, Use python2 or update glance(https://review.openstack.org/#/c/532503/10/glance/common/store_utils.py@30 as use_user_token is deprecated since long)
with keystone+cachemanagement, all the config items were added before reaching the Failure point in keystonemiddleware and self._conf.oslo_conf_obj != cfg.CONF didn't raised an error and returned Boolean. Don't know why.

But it seems a real issue to me as it may happen in python3 at different places. So it would be good if Teams from affected projects(oslo.config, keystonemiddleware, glance) can look at it and fix(not avoid) at the best place.
To me it looks like keystonemiddleware is not handling(comparing the dict) it properly for python3, as the conf is dynamically updated(how ? and when ?).

- so can oslo.config Team check if glance and keystonmiddleware are handling/using oslo.config properly.
- i checked keystone+cachemanagement is default in devstack from last 6 years, is "keystone" flavor supported? if yes it should be fixed. Also it would be good to cleanup the deprecated options those are deprecated since Mitaka.
- If it's wrongly used in keystonemiddleware/glance, it would be good to fix there.


Initially detected while testing with Fedora[2], but later digged on why it's working in CI with Ubuntu and started [3].


[1] https://review.openstack.org/#/c/560094/
[2] https://review.rdoproject.org/r/#/c/14921/
[3] https://review.openstack.org/#/c/596380/

** Affects: glance
     Importance: Undecided
         Status: New

** Affects: keystonemiddleware
     Importance: Undecided
         Status: New

** Affects: oslo.config
     Importance: Undecided
         Status: New

** Also affects: oslo.config
   Importance: Undecided
       Status: New

** Also affects: keystonemiddleware
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1789351

Title:
  Glance deployment with python3 + "keystone" paste_deploy flavor Fails

Status in Glance:
  New
Status in keystonemiddleware:
  New
Status in oslo.config:
  New

Bug description:
  This happens with oslo.config >= 6.3.0([1]) + python3 + "keystone" paste_deploy + current glance(before https://review.openstack.org/#/c/532503/10/glance/common/store_utils.py@30 it works)
  Testing in devstack: https://review.openstack.org/#/c/596380/

  The glance api service fails to start with below Error, reproducing here: https://review.openstack.org/#/c/596380/:-
  ERROR: dictionary changed size during iteration , see logs below

  Failure logs from job:- http://logs.openstack.org/80/596380/2/check
  /tempest-full-
  py3/514fa29/controller/logs/screen-g-api.txt.gz#_Aug_27_07_26_10_698243

  
  The Runtime Error is returned at keystonemiddleware:- https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token/__init__.py#L551
  Adding code snippet here:-
  if self._conf.oslo_conf_obj != cfg.CONF:   <-- Fails here
      oslo_cache.configure(self._conf.oslo_conf_obj)

  So with pdb found that an additional key(fatal_deprecations) was added
  to cfg.CONF at ^^, so Error is returned in python3. With python2 same
  key is added but no Error.

  There are multiple ways to avoid it, like use the paste_deploy configuration that works(ex: keystone+cachemanagement), use oslo.config <= 6.2.0, Use python2 or update glance(https://review.openstack.org/#/c/532503/10/glance/common/store_utils.py@30 as use_user_token is deprecated since long)
  with keystone+cachemanagement, all the config items were added before reaching the Failure point in keystonemiddleware and self._conf.oslo_conf_obj != cfg.CONF didn't raised an error and returned Boolean. Don't know why.

  But it seems a real issue to me as it may happen in python3 at different places. So it would be good if Teams from affected projects(oslo.config, keystonemiddleware, glance) can look at it and fix(not avoid) at the best place.
  To me it looks like keystonemiddleware is not handling(comparing the dict) it properly for python3, as the conf is dynamically updated(how ? and when ?).

  - so can oslo.config Team check if glance and keystonmiddleware are handling/using oslo.config properly.
  - i checked keystone+cachemanagement is default in devstack from last 6 years, is "keystone" flavor supported? if yes it should be fixed. Also it would be good to cleanup the deprecated options those are deprecated since Mitaka.
  - If it's wrongly used in keystonemiddleware/glance, it would be good to fix there.

  
  Initially detected while testing with Fedora[2], but later digged on why it's working in CI with Ubuntu and started [3].

  
  [1] https://review.openstack.org/#/c/560094/
  [2] https://review.rdoproject.org/r/#/c/14921/
  [3] https://review.openstack.org/#/c/596380/

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1789351/+subscriptions