yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1808951] Re: python3 + Fedora + SSL + wsgi nova deployment, nova api returns RecursionError: maximum recursion depth exceeded while calling a Python object

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Alex Schultz <1808951@xxxxxxxxxxxxxxxxxx>
Date: Fri, 25 Jan 2019 21:00:06 -0000
Reply-to: Bug 1808951 <1808951@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Adding tripleo because this is affecting our fedora28 containers when
deployed via an undercloud with ssl enabled.

** Also affects: tripleo
   Importance: Undecided
       Status: New

** Changed in: tripleo
       Status: New => Incomplete

** Changed in: tripleo
       Status: Incomplete => Triaged

** Changed in: tripleo
   Importance: Undecided => High

** Changed in: tripleo
    Milestone: None => stein-3

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1808951

Title:
  python3 + Fedora + SSL + wsgi nova deployment, nova api returns
  RecursionError: maximum recursion depth exceeded while calling a
  Python object

Status in OpenStack Compute (nova):
  New
Status in tripleo:
  Triaged

Bug description:
  Description:-

  So while testing python3 with Fedora in [1], Found an issue while
  running nova-api behind wsgi. It fails with below Traceback:-

  2018-12-18 07:41:55.364 26870 INFO nova.api.openstack.requestlog [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] 127.0.0.1 "GET /v2.1/servers/detail?all_tenants=True&deleted=True" status: 500 len: 0 microversion: - time: 0.007297
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] Caught error: maximum recursion depth exceeded while calling a Python object: RecursionError: maximum recursion depth exceeded while calling a Python object
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack Traceback (most recent call last):
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/nova/api/openstack/__init__.py", line 94, in __call__
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     return req.get_response(self.application)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     application, catch_exc_info=False)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     app_iter = application(self.environ, start_response)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     resp = self.call_func(req, *args, **kw)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     return self.func(req, *args, **kwargs)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/nova/api/openstack/requestlog.py", line 92, in __call__
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     self._log_req(req, res, start)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     self.force_reraise()
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     six.reraise(self.type_, self.value, self.tb)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     raise value
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/nova/api/openstack/requestlog.py", line 87, in __call__
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     res = req.get_response(self.application)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     application, catch_exc_info=False)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     app_iter = application(self.environ, start_response)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     return resp(environ, start_response)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     resp = self.call_func(req, *args, **kw)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     return self.func(req, *args, **kwargs)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/osprofiler/web.py", line 112, in __call__
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     return request.get_response(self.application)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     application, catch_exc_info=False)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     app_iter = application(self.environ, start_response)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     resp = self.call_func(req, *args, **kw)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     return self.func(req, *args, **kwargs)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 332, in __call__
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     response = self.process_request(req)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 622, in process_request
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     resp = super(AuthProtocol, self).process_request(request)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 405, in process_request
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     allow_expired=allow_expired)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 437, in _do_fetch_token
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     data = self.fetch_token(token, **kwargs)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 761, in fetch_token
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     allow_expired=allow_expired)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 219, in verify_token
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     auth_ref = self._request_strategy.verify_token(
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 170, in _request_strategy
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     strategy_class = self._get_strategy_class()
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 192, in _get_strategy_class
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     if self._adapter.get_endpoint(version=klass.AUTH_VERSION):
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 247, in get_endpoint
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     return self.session.get_endpoint(auth or self.auth, **kwargs)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1113, in get_endpoint
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     return auth.get_endpoint(self, **kwargs)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     allow_version_hack=allow_version_hack, **kwargs)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     service_catalog = self.get_access(session).service_catalog
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     self.auth_ref = self.get_auth_ref(session)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     self._plugin = self._do_create_plugin(session)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     authenticated=False)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     authenticated=authenticated)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1442, in get_discovery
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     disc = Discover(session, url, authenticated=authenticated)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 526, in __init__
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     authenticated=authenticated)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     resp = session.get(url, headers=headers, authenticated=authenticated)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1011, in get
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     return self.request(url, 'GET', **kwargs)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 814, in request
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     resp = send(**kwargs)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 903, in _send_request
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     resp = self.session.request(method, url, **kwargs)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 524, in request
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     resp = self.send(prep, **send_kwargs)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 637, in send
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     r = adapter.send(request, **kwargs)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     timeout=timeout
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     chunked=chunked)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     self._validate_conn(conn)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     conn.connect()
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 332, in connect
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     cert_reqs=resolve_cert_reqs(self.cert_reqs),
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 281, in create_urllib3_context
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     context.options |= options
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib64/python3.6/ssl.py", line 423, in options
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     super(SSLContext, SSLContext).options.__set__(self, value)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib64/python3.6/ssl.py", line 423, in options
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     super(SSLContext, SSLContext).options.__set__(self, value)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib64/python3.6/ssl.py", line 423, in options
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     super(SSLContext, SSLContext).options.__set__(self, value)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   [Previous line repeated 280 more times]
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack RecursionError: maximum recursion depth exceeded while calling a Python object
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack 
  2018-12-18 07:41:55.371 26870 INFO nova.api.openstack [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] https://127.0.0.1:8774/v2.1/servers/detail?all_tenants=True&deleted=True returned with HTTP 500

  
  Environment Details:-
  OS: Fedora 28
  python: 3.6.7
  eventlet: 0.24.1

  
  The issue is around using of eventlet.monkey_patch even if running under wsgi[2]. Eventlet Best practices suggest to not run(skip) monkey_patch when running behind wsgi[3] but with nova it's not the case.

  Expected result
  ===============
  Nova api should response successfully.

  Actual result
  =============
  Nova api returns Recursion Error.

  
  [1] https://review.openstack.org/#/c/620561/
  [2] https://github.com/openstack/nova/blob/a674241d0f026b546b38ec77ab472e916805c679/nova/api/openstack/wsgi_app.py#L32
  [3] https://review.openstack.org/#/c/154642/2/specs/eventlet-best-practices.rst@54

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1808951/+subscriptions
References

[Bug 1808951] [NEW] python3 + Fedora + SSL + wsgi nova deployment, nova api returns RecursionError: maximum recursion depth exceeded while calling a Python object
From: yatin, 2018-12-18