yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1808951] Re: python3 + Fedora + SSL + wsgi nova deployment, nova api returns RecursionError: maximum recursion depth exceeded while calling a Python object

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1808951@xxxxxxxxxxxxxxxxxx>
Date: Fri, 22 Mar 2019 22:31:23 -0000
Reply-to: Bug 1808951 <1808951@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/626952
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3c5e2b0e9fac985294a949852bb8c83d4ed77e04
Submitter: Zuul
Branch:    master

commit 3c5e2b0e9fac985294a949852bb8c83d4ed77e04
Author: Matthew Booth <mbooth@xxxxxxxxxx>
Date:   Wed Jan 30 15:10:25 2019 +0000

    Eventlet monkey patching should be as early as possible
    
    We were seeing infinite recursion opening an ssl socket when running
    various combinations of python3, eventlet, and urllib3. It is not
    clear exactly what combination of versions are affected, but for
    background there is an example of this issue documented here:
    
    https://github.com/eventlet/eventlet/issues/371
    
    The immediate cause in nova's case was that we were calling
    eventlet.monkey_patch() after importing urllib3. Specifically, change
    Ie7bf5d012e2ccbcd63c262ddaf739782afcdaf56 introduced the
    nova.utils.monkey_patch() method to make monkey patching common
    between WSGI and non-WSGI services. Unfortunately, before executing
    this method you must first import nova.utils, which imports a large
    number of modules itself. Anything imported (transitively) by
    nova.utils would therefore be imported before monkey patching, which
    included urllib3. This triggers the infinite recursion problem
    described above if you have an affected combination of library
    versions.
    
    While this specific issue may eventually be worked around or fixed in
    eventlet or urllib3, it remains true that eventlet best practises are
    to monkey patch as early as possible, which we were not doing. To
    avoid this and hopefully future similar issues, this change ensures
    that monkey patching happens as early as possible, and only a minimum
    number of modules are imported first.
    
    This change fixes monkey patching for both non-wsgi and wsgi callers:
    
    * Non-WSGI services (nova/cmd)
    
      This is fixed by using the new monkey_patch module, which has minimal
      dependencies.
    
    * WSGI services (nova/api/openstack)
    
      This is fixed both by using the new monkey_patch module, and by moving
      the patching point up one level so that it is done before importing
      anything in nova/api/openstack/__init__.py.
    
      This move causes issues for some external tools which load this path
      from nova and now monkey patch where they previously did not. However,
      it is unfortunately unavoidable to enable monkey patching for the wsgi
      entry point without major restructuring. This change includes a
      workaround for sphinx to avoid this issue.
    
    This change has been through several iterations. I started with what
    seemed like the simplest and most obvious change, and moved on as I
    discovered more interactions which broke. It is clear that eventlet
    monkey patching is extremely fragile, especially when done implicitly at
    module load time as we do. I would advocate a code restructure to
    improve this situation, but I think the time would be better spent
    removing the eventlet dependency entirely.
    
    Co-authored-by: Lee Yarwood <lyarwood@xxxxxxxxxx>
    
    Closes-Bug: #1808975
    Closes-Bug: #1808951
    Change-Id: Id46e76666b553a10ec4654d4418a9884975b5b95


** Changed in: nova
       Status: In Progress => Fix Released

** Bug watch added: github.com/eventlet/eventlet/issues #371
   https://github.com/eventlet/eventlet/issues/371

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1808951

Title:
  python3 + Fedora + SSL + wsgi nova deployment, nova api returns
  RecursionError: maximum recursion depth exceeded while calling a
  Python object

Status in OpenStack Compute (nova):
  Fix Released
Status in tripleo:
  Triaged

Bug description:
  Description:-

  So while testing python3 with Fedora in [1], Found an issue while
  running nova-api behind wsgi. It fails with below Traceback:-

  2018-12-18 07:41:55.364 26870 INFO nova.api.openstack.requestlog [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] 127.0.0.1 "GET /v2.1/servers/detail?all_tenants=True&deleted=True" status: 500 len: 0 microversion: - time: 0.007297
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] Caught error: maximum recursion depth exceeded while calling a Python object: RecursionError: maximum recursion depth exceeded while calling a Python object
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack Traceback (most recent call last):
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/nova/api/openstack/__init__.py", line 94, in __call__
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     return req.get_response(self.application)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     application, catch_exc_info=False)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     app_iter = application(self.environ, start_response)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     resp = self.call_func(req, *args, **kw)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     return self.func(req, *args, **kwargs)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/nova/api/openstack/requestlog.py", line 92, in __call__
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     self._log_req(req, res, start)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     self.force_reraise()
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     six.reraise(self.type_, self.value, self.tb)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     raise value
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/nova/api/openstack/requestlog.py", line 87, in __call__
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     res = req.get_response(self.application)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     application, catch_exc_info=False)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     app_iter = application(self.environ, start_response)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     return resp(environ, start_response)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     resp = self.call_func(req, *args, **kw)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     return self.func(req, *args, **kwargs)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/osprofiler/web.py", line 112, in __call__
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     return request.get_response(self.application)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     application, catch_exc_info=False)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     app_iter = application(self.environ, start_response)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     resp = self.call_func(req, *args, **kw)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     return self.func(req, *args, **kwargs)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 332, in __call__
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     response = self.process_request(req)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 622, in process_request
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     resp = super(AuthProtocol, self).process_request(request)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 405, in process_request
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     allow_expired=allow_expired)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 437, in _do_fetch_token
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     data = self.fetch_token(token, **kwargs)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 761, in fetch_token
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     allow_expired=allow_expired)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 219, in verify_token
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     auth_ref = self._request_strategy.verify_token(
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 170, in _request_strategy
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     strategy_class = self._get_strategy_class()
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 192, in _get_strategy_class
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     if self._adapter.get_endpoint(version=klass.AUTH_VERSION):
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 247, in get_endpoint
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     return self.session.get_endpoint(auth or self.auth, **kwargs)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1113, in get_endpoint
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     return auth.get_endpoint(self, **kwargs)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     allow_version_hack=allow_version_hack, **kwargs)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     service_catalog = self.get_access(session).service_catalog
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     self.auth_ref = self.get_auth_ref(session)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     self._plugin = self._do_create_plugin(session)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     authenticated=False)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     authenticated=authenticated)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1442, in get_discovery
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     disc = Discover(session, url, authenticated=authenticated)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 526, in __init__
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     authenticated=authenticated)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     resp = session.get(url, headers=headers, authenticated=authenticated)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1011, in get
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     return self.request(url, 'GET', **kwargs)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 814, in request
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     resp = send(**kwargs)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 903, in _send_request
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     resp = self.session.request(method, url, **kwargs)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 524, in request
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     resp = self.send(prep, **send_kwargs)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 637, in send
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     r = adapter.send(request, **kwargs)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     timeout=timeout
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     chunked=chunked)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     self._validate_conn(conn)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     conn.connect()
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 332, in connect
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     cert_reqs=resolve_cert_reqs(self.cert_reqs),
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 281, in create_urllib3_context
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     context.options |= options
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib64/python3.6/ssl.py", line 423, in options
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     super(SSLContext, SSLContext).options.__set__(self, value)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib64/python3.6/ssl.py", line 423, in options
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     super(SSLContext, SSLContext).options.__set__(self, value)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   File "/usr/lib64/python3.6/ssl.py", line 423, in options
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack     super(SSLContext, SSLContext).options.__set__(self, value)
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack   [Previous line repeated 280 more times]
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack RecursionError: maximum recursion depth exceeded while calling a Python object
  2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack 
  2018-12-18 07:41:55.371 26870 INFO nova.api.openstack [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] https://127.0.0.1:8774/v2.1/servers/detail?all_tenants=True&deleted=True returned with HTTP 500

  
  Environment Details:-
  OS: Fedora 28
  python: 3.6.7
  eventlet: 0.24.1

  
  The issue is around using of eventlet.monkey_patch even if running under wsgi[2]. Eventlet Best practices suggest to not run(skip) monkey_patch when running behind wsgi[3] but with nova it's not the case.

  Expected result
  ===============
  Nova api should response successfully.

  Actual result
  =============
  Nova api returns Recursion Error.

  
  [1] https://review.openstack.org/#/c/620561/
  [2] https://github.com/openstack/nova/blob/a674241d0f026b546b38ec77ab472e916805c679/nova/api/openstack/wsgi_app.py#L32
  [3] https://review.openstack.org/#/c/154642/2/specs/eventlet-best-practices.rst@54

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1808951/+subscriptions
References

[Bug 1808951] [NEW] python3 + Fedora + SSL + wsgi nova deployment, nova api returns RecursionError: maximum recursion depth exceeded while calling a Python object
From: yatin, 2018-12-18