yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1808975] [NEW] python3 + Fedora + SSL + nova compute RecursionError: maximum recursion depth exceeded while calling a Python object

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: yatin <ykarel@xxxxxxxxxx>
Date: Tue, 18 Dec 2018 13:04:07 -0000
Reply-to: Bug 1808975 <1808975@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description:- While Testing python3 Fedora deployment for nova in [1]
got below Recursion Error in nova-compute:-

2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager [req-f908a9e0-e77a-4d35-9266-fc5e8d79dfde - - - - -] Error updating resources for node rdo-fedora-stable-rdo-cloud-0000358855.: RecursionError: maximum recursion depth exceeded while calling a Python object
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager Traceback (most recent call last):
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 7690, in _update_available_resource_for_node
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     rt.update_available_resource(context, nodename, startup=startup)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 738, in update_available_resource
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     self._update_available_resource(context, resources, startup=startup)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     return f(*args, **kwargs)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 794, in _update_available_resource
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     instance_by_uuid)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1256, in _remove_deleted_instances_allocations
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     context, cn.uuid)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 2165, in get_allocations_for_resource_provider
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     resp = self.get(url, global_request_id=context.global_id)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 297, in get
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     return self._client.get(url, microversion=version, headers=headers)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 351, in get
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     return self.request(url, 'GET', **kwargs)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 213, in request
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     return self.session.request(url, method, **kwargs)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 684, in request
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     auth_headers = self.get_auth_headers(auth)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     return auth.get_headers(self, **kwargs)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/plugin.py", line 95, in get_headers
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     token = self.get_token(session)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 88, in get_token
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     return self.get_access(session).auth_token
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     self.auth_ref = self.get_auth_ref(session)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     self._plugin = self._do_create_plugin(session)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     authenticated=False)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     authenticated=authenticated)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1436, in get_discovery
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     disc = Discover(session, url, authenticated=authenticated)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 520, in __init__
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     authenticated=authenticated)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     resp = session.get(url, headers=headers, authenticated=authenticated)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1011, in get
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     return self.request(url, 'GET', **kwargs)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 814, in request
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     resp = send(**kwargs)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 903, in _send_request
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     resp = self.session.request(method, url, **kwargs)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 524, in request
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     resp = self.send(prep, **send_kwargs)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 637, in send
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     r = adapter.send(request, **kwargs)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     timeout=timeout
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     chunked=chunked)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     self._validate_conn(conn)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     conn.connect()
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 332, in connect
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     cert_reqs=resolve_cert_reqs(self.cert_reqs),
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 281, in create_urllib3_context
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     context.options |= options
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib64/python3.6/ssl.py", line 423, in options
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     super(SSLContext, SSLContext).options.__set__(self, value)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib64/python3.6/ssl.py", line 423, in options
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     super(SSLContext, SSLContext).options.__set__(self, value)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib64/python3.6/ssl.py", line 423, in options
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     super(SSLContext, SSLContext).options.__set__(self, value)
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   [Previous line repeated 294 more times]
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager RecursionError: maximum recursion depth exceeded while calling a Python object
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager 


Don't know what's causing it but it somehow related to the eventlet monkey patching done in nova. I filed seperated bug(also filed https://bugs.launchpad.net/nova/+bug/1808951 for nova-api running under wsgi) for nova-compute as issue here seems different(as nova-compute is not running under wsgi).


Logs from job: https://logs.rdoproject.org/61/620561/12/openstack-check-rdo/puppet-openstack-integration-5-scenario-py3-tempest-fedora-28/74a9000/logs/nova/nova-compute.txt.gz

Expected Result:- 
nova compute should work correctly under python3

Actual Result:-
nova-compute fails

[1] https://review.openstack.org/#/c/620561

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1808975

Title:
  python3 + Fedora + SSL + nova compute RecursionError: maximum
  recursion depth exceeded while calling a Python object

Status in OpenStack Compute (nova):
  New

Bug description:
  Description:- While Testing python3 Fedora deployment for nova in [1]
  got below Recursion Error in nova-compute:-

  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager [req-f908a9e0-e77a-4d35-9266-fc5e8d79dfde - - - - -] Error updating resources for node rdo-fedora-stable-rdo-cloud-0000358855.: RecursionError: maximum recursion depth exceeded while calling a Python object
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager Traceback (most recent call last):
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 7690, in _update_available_resource_for_node
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     rt.update_available_resource(context, nodename, startup=startup)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 738, in update_available_resource
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     self._update_available_resource(context, resources, startup=startup)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     return f(*args, **kwargs)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 794, in _update_available_resource
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     instance_by_uuid)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1256, in _remove_deleted_instances_allocations
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     context, cn.uuid)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 2165, in get_allocations_for_resource_provider
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     resp = self.get(url, global_request_id=context.global_id)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 297, in get
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     return self._client.get(url, microversion=version, headers=headers)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 351, in get
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     return self.request(url, 'GET', **kwargs)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 213, in request
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     return self.session.request(url, method, **kwargs)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 684, in request
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     auth_headers = self.get_auth_headers(auth)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     return auth.get_headers(self, **kwargs)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/plugin.py", line 95, in get_headers
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     token = self.get_token(session)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 88, in get_token
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     return self.get_access(session).auth_token
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     self.auth_ref = self.get_auth_ref(session)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     self._plugin = self._do_create_plugin(session)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     authenticated=False)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     authenticated=authenticated)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1436, in get_discovery
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     disc = Discover(session, url, authenticated=authenticated)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 520, in __init__
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     authenticated=authenticated)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     resp = session.get(url, headers=headers, authenticated=authenticated)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1011, in get
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     return self.request(url, 'GET', **kwargs)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 814, in request
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     resp = send(**kwargs)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 903, in _send_request
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     resp = self.session.request(method, url, **kwargs)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 524, in request
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     resp = self.send(prep, **send_kwargs)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 637, in send
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     r = adapter.send(request, **kwargs)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     timeout=timeout
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     chunked=chunked)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     self._validate_conn(conn)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     conn.connect()
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 332, in connect
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     cert_reqs=resolve_cert_reqs(self.cert_reqs),
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 281, in create_urllib3_context
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     context.options |= options
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib64/python3.6/ssl.py", line 423, in options
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     super(SSLContext, SSLContext).options.__set__(self, value)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib64/python3.6/ssl.py", line 423, in options
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     super(SSLContext, SSLContext).options.__set__(self, value)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   File "/usr/lib64/python3.6/ssl.py", line 423, in options
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager     super(SSLContext, SSLContext).options.__set__(self, value)
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager   [Previous line repeated 294 more times]
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager RecursionError: maximum recursion depth exceeded while calling a Python object
  2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager 

  
  Don't know what's causing it but it somehow related to the eventlet monkey patching done in nova. I filed seperated bug(also filed https://bugs.launchpad.net/nova/+bug/1808951 for nova-api running under wsgi) for nova-compute as issue here seems different(as nova-compute is not running under wsgi).

  
  Logs from job: https://logs.rdoproject.org/61/620561/12/openstack-check-rdo/puppet-openstack-integration-5-scenario-py3-tempest-fedora-28/74a9000/logs/nova/nova-compute.txt.gz

  Expected Result:- 
  nova compute should work correctly under python3

  Actual Result:-
  nova-compute fails

  [1] https://review.openstack.org/#/c/620561

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1808975/+subscriptions
Follow ups

[Bug 1808975] Re: python3 + Fedora + SSL + nova compute RecursionError: maximum recursion depth exceeded while calling a Python object
From: OpenStack Infra, 2019-03-22