yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1829304] Re: Neutron returns HttpException: 500 on certain operations with modified list of policies for non-admin users

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1829304@xxxxxxxxxxxxxxxxxx>
Date: Thu, 16 May 2019 23:25:08 -0000
Reply-to: Bug 1829304 <1829304@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.opendev.org/659397
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2c1098b3afb4f5d7cd1d10eaa722652624e6bb74
Submitter: Zuul
Branch:    master

commit 2c1098b3afb4f5d7cd1d10eaa722652624e6bb74
Author: Nate Johnston <nate.johnston@xxxxxxxxxx>
Date:   Wed May 15 19:18:52 2019 -0400

    Use six.viewkeys instead of dict.keys to avoid py2 to py3 problems
    
    This change fixes an 'RuntimeError: dictionary changed size during
    iteration' error that is raised because of different behaviour between
    python2 and python3. We use the six library to ensure that the behavior
    is compatible across versions.
    
    Change-Id: I0723ae10825e1e2d86789627895e3286d8c97602
    Resolves-Bug: #1829304


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1829304

Title:
  Neutron returns HttpException: 500 on certain operations with modified
  list of policies for non-admin users

Status in neutron:
  Fix Released

Bug description:
  Description of problem:

  When deploying with a modified list of Neutron API policies, post
  deployment, policies which worked on previous versions will result in
  Neutron API Server returning 'HttpException: 500' when using the API
  with non admin users.

  Additional API policies which were passed during deployment:
  http://paste.openstack.org/show/751347/

  Example:

  1. Source credentials with non admin user
  [stack@undercloud-0 ~]$ source /home/stack/overcloudrc_user_tenant
  2. Query port list with as non admin user
  (overcloud) [stack@undercloud-0 ~]$ openstack port list

  At this point, neutron will return:
  HttpException: 500: Server Error for url: http://10.35.141.150:9696/v2.0/ports, Internal Server Error

  And the following exception will be generated inside server.log on controller nodes:
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors [req-98bfd77f-1fc1-4d13-a0fd-02e82f6caa53 2236f6cc04c04964a0b435599ffb7acb ef4de28cbec04ea785b855010e7f46a1 - default default] An error occurred during processing the request: POST /v2.0/ports HTTP/1.0
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors Traceback (most recent call last):
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/oslo_middleware/catch_errors.py", line 40, in __call__
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     response = req.get_response(self.application)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     application, catch_exc_info=False)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     app_iter = application(self.environ, start_response)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     resp = self.call_func(req, *args, **kw)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return self.func(req, *args, **kwargs)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/osprofiler/web.py", line 112, in __call__
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return request.get_response(self.application)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     application, catch_exc_info=False)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     app_iter = application(self.environ, start_response)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     resp = self.call_func(req, *args, **kw)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return self.func(req, *args, **kwargs)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 333, in __call__
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     response = req.get_response(self._app)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     application, catch_exc_info=False)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     app_iter = application(self.environ, start_response)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return resp(environ, start_response)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return resp(environ, start_response)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/routes/middleware.py", line 141, in __call__
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     response = self.app(environ, start_response)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return resp(environ, start_response)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/pecan/middleware/recursive.py", line 56, in __call__
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return self.application(environ, start_response)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/pecan/core.py", line 840, in __call__
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return super(Pecan, self).__call__(environ, start_response)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/pecan/core.py", line 736, in __call__
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     state
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/pecan/core.py", line 865, in handle_hooks
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return super(Pecan, self).handle_hooks(hooks, *args, **kw)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/pecan/core.py", line 342, in handle_hooks
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     result = getattr(hook, hook_type)(*args)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 185, in after
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     for item in to_process
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 189, in <listcomp>
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     pluralized=collection))]
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 207, in _get_filtered_item
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     neutron_context, controller, resource, collection, data)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 226, in _exclude_attributes_by_policy
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     for attr_name in data.keys():
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors RuntimeError: dictionary changed size during iteration
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors 
  }

  Version-Release number of selected component (if applicable):
  Compose: RHOS_TRUNK-15.0-RHEL-8-20190509.n.1
  rpm -qa | grep neutron
  puppet-neutron-14.4.1-0.20190420042323.400fd54.el8ost.noarch
  python3-neutronclient-6.12.0-0.20190312100012.680b417.el8ost.noarch

  
  How reproducible:
  Always

  
  Steps to Reproduce:
  1. Deploy Overcloud with modified Neutron APIs
  2. Create non admin user/tenant
  3. Attempt to list ports

  Actual results:
  Fail to retrieve ports and receive python exceptions

  Expected results:
  List of ports is returned

  Additional info:

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1829304/+subscriptions
References

[Bug 1829304] [NEW] Neutron returns HttpException: 500 on certain operations with modified list of policies for non-admin users
From: Nate Johnston, 2019-05-15