yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1829304] [NEW] Neutron returns HttpException: 500 on certain operations with modified list of policies for non-admin users

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Nate Johnston <1829304@xxxxxxxxxxxxxxxxxx>
Date: Wed, 15 May 2019 22:57:11 -0000
Reply-to: Bug 1829304 <1829304@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description of problem:

When deploying with a modified list of Neutron API policies, post
deployment, policies which worked on previous versions will result in
Neutron API Server returning 'HttpException: 500' when using the API
with non admin users.

Additional API policies which were passed during deployment:
http://paste.openstack.org/show/751347/

Example:

1. Source credentials with non admin user
[stack@undercloud-0 ~]$ source /home/stack/overcloudrc_user_tenant
2. Query port list with as non admin user
(overcloud) [stack@undercloud-0 ~]$ openstack port list

At this point, neutron will return:
HttpException: 500: Server Error for url: http://10.35.141.150:9696/v2.0/ports, Internal Server Error

And the following exception will be generated inside server.log on controller nodes:
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors [req-98bfd77f-1fc1-4d13-a0fd-02e82f6caa53 2236f6cc04c04964a0b435599ffb7acb ef4de28cbec04ea785b855010e7f46a1 - default default] An error occurred during processing the request: POST /v2.0/ports HTTP/1.0
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors Traceback (most recent call last):
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/oslo_middleware/catch_errors.py", line 40, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     response = req.get_response(self.application)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     application, catch_exc_info=False)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     app_iter = application(self.environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     resp = self.call_func(req, *args, **kw)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return self.func(req, *args, **kwargs)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/osprofiler/web.py", line 112, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return request.get_response(self.application)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     application, catch_exc_info=False)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     app_iter = application(self.environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     resp = self.call_func(req, *args, **kw)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return self.func(req, *args, **kwargs)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 333, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     response = req.get_response(self._app)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     application, catch_exc_info=False)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     app_iter = application(self.environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return resp(environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return resp(environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/routes/middleware.py", line 141, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     response = self.app(environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return resp(environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/pecan/middleware/recursive.py", line 56, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return self.application(environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/pecan/core.py", line 840, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return super(Pecan, self).__call__(environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/pecan/core.py", line 736, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     state
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/pecan/core.py", line 865, in handle_hooks
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return super(Pecan, self).handle_hooks(hooks, *args, **kw)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/pecan/core.py", line 342, in handle_hooks
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     result = getattr(hook, hook_type)(*args)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 185, in after
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     for item in to_process
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 189, in <listcomp>
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     pluralized=collection))]
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 207, in _get_filtered_item
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     neutron_context, controller, resource, collection, data)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 226, in _exclude_attributes_by_policy
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     for attr_name in data.keys():
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors RuntimeError: dictionary changed size during iteration
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors 
}

Version-Release number of selected component (if applicable):
Compose: RHOS_TRUNK-15.0-RHEL-8-20190509.n.1
rpm -qa | grep neutron
puppet-neutron-14.4.1-0.20190420042323.400fd54.el8ost.noarch
python3-neutronclient-6.12.0-0.20190312100012.680b417.el8ost.noarch


How reproducible:
Always


Steps to Reproduce:
1. Deploy Overcloud with modified Neutron APIs
2. Create non admin user/tenant
3. Attempt to list ports

Actual results:
Fail to retrieve ports and receive python exceptions

Expected results:
List of ports is returned

Additional info:

** Affects: neutron
     Importance: Undecided
     Assignee: Nate Johnston (nate-johnston)
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1829304

Title:
  Neutron returns HttpException: 500 on certain operations with modified
  list of policies for non-admin users

Status in neutron:
  New

Bug description:
  Description of problem:

  When deploying with a modified list of Neutron API policies, post
  deployment, policies which worked on previous versions will result in
  Neutron API Server returning 'HttpException: 500' when using the API
  with non admin users.

  Additional API policies which were passed during deployment:
  http://paste.openstack.org/show/751347/

  Example:

  1. Source credentials with non admin user
  [stack@undercloud-0 ~]$ source /home/stack/overcloudrc_user_tenant
  2. Query port list with as non admin user
  (overcloud) [stack@undercloud-0 ~]$ openstack port list

  At this point, neutron will return:
  HttpException: 500: Server Error for url: http://10.35.141.150:9696/v2.0/ports, Internal Server Error

  And the following exception will be generated inside server.log on controller nodes:
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors [req-98bfd77f-1fc1-4d13-a0fd-02e82f6caa53 2236f6cc04c04964a0b435599ffb7acb ef4de28cbec04ea785b855010e7f46a1 - default default] An error occurred during processing the request: POST /v2.0/ports HTTP/1.0
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors Traceback (most recent call last):
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/oslo_middleware/catch_errors.py", line 40, in __call__
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     response = req.get_response(self.application)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     application, catch_exc_info=False)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     app_iter = application(self.environ, start_response)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     resp = self.call_func(req, *args, **kw)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return self.func(req, *args, **kwargs)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/osprofiler/web.py", line 112, in __call__
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return request.get_response(self.application)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     application, catch_exc_info=False)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     app_iter = application(self.environ, start_response)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     resp = self.call_func(req, *args, **kw)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return self.func(req, *args, **kwargs)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 333, in __call__
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     response = req.get_response(self._app)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     application, catch_exc_info=False)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     app_iter = application(self.environ, start_response)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return resp(environ, start_response)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return resp(environ, start_response)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/routes/middleware.py", line 141, in __call__
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     response = self.app(environ, start_response)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return resp(environ, start_response)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/pecan/middleware/recursive.py", line 56, in __call__
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return self.application(environ, start_response)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/pecan/core.py", line 840, in __call__
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return super(Pecan, self).__call__(environ, start_response)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/pecan/core.py", line 736, in __call__
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     state
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/pecan/core.py", line 865, in handle_hooks
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return super(Pecan, self).handle_hooks(hooks, *args, **kw)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/pecan/core.py", line 342, in handle_hooks
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     result = getattr(hook, hook_type)(*args)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 185, in after
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     for item in to_process
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 189, in <listcomp>
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     pluralized=collection))]
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 207, in _get_filtered_item
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     neutron_context, controller, resource, collection, data)
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 226, in _exclude_attributes_by_policy
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     for attr_name in data.keys():
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors RuntimeError: dictionary changed size during iteration
  server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors 
  }

  Version-Release number of selected component (if applicable):
  Compose: RHOS_TRUNK-15.0-RHEL-8-20190509.n.1
  rpm -qa | grep neutron
  puppet-neutron-14.4.1-0.20190420042323.400fd54.el8ost.noarch
  python3-neutronclient-6.12.0-0.20190312100012.680b417.el8ost.noarch

  
  How reproducible:
  Always

  
  Steps to Reproduce:
  1. Deploy Overcloud with modified Neutron APIs
  2. Create non admin user/tenant
  3. Attempt to list ports

  Actual results:
  Fail to retrieve ports and receive python exceptions

  Expected results:
  List of ports is returned

  Additional info:

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1829304/+subscriptions
Follow ups

[Bug 1829304] Re: Neutron returns HttpException: 500 on certain operations with modified list of policies for non-admin users
From: OpenStack Infra, 2019-05-16