yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1884808] [NEW] [SSL] neutron-rpc-server: RecursionError: maximum recursion depth exceeded while calling a Python object

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Andrii Ostapenko <1884808@xxxxxxxxxxxxxxxxxx>
Date: Tue, 23 Jun 2020 16:32:02 -0000
Reply-to: Bug 1884808 <1884808@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

py36 stable/rocky installation with SSL fails on nova notifier with:

2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova [-] Failed to notify nova on events: [{'server_uuid': 'd212f288-e1a6-4615-ae77-6db03cfc5460', 'name': 'network-vif-plugged', 'status': 'completed', 'tag': '3cc3ed7d-2717-441b-bcb2-bc2b31405eae'}]: RecursionError: maximum recursion depth exceeded while calling a Python object
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova Traceback (most recent call last):
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/neutron/notifiers/nova.py", line 245, in send_events
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     batched_events)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/novaclient/v2/server_external_events.py", line 39, in create
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     return_raw=True)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/novaclient/base.py", line 366, in _create
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     resp, body = self.api.client.post(url, body=body)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 334, in post
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     return self.request(url, 'POST', **kwargs)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/novaclient/client.py", line 77, in request
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     **kwargs)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 487, in request
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 213, in request
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     return self.session.request(url, method, **kwargs)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/session.py", line 684, in request
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     auth_headers = self.get_auth_headers(auth)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     return auth.get_headers(self, **kwargs)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/plugin.py", line 95, in get_headers
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     token = self.get_token(session)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 88, in get_token
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     return self.get_access(session).auth_token
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     self.auth_ref = self.get_auth_ref(session)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     self._plugin = self._do_create_plugin(session)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     authenticated=False)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 601, in get_discovery
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     authenticated=authenticated)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1399, in get_discovery
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     disc = Discover(session, url, authenticated=authenticated)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/discover.py", line 498, in __init__
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     authenticated=authenticated)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     resp = session.get(url, headers=headers, authenticated=authenticated)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/session.py", line 1011, in get
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     return self.request(url, 'GET', **kwargs)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/session.py", line 814, in request
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     resp = send(**kwargs)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/session.py", line 903, in _send_request
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     resp = self.session.request(method, url, **kwargs)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     resp = self.send(prep, **send_kwargs)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     r = adapter.send(request, **kwargs)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     timeout=timeout
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     chunked=chunked)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     self._validate_conn(conn)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/urllib3/connectionpool.py", line 849, in _validate_conn
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     conn.connect()
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/urllib3/connection.py", line 344, in connect
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     cert_reqs=resolve_cert_reqs(self.cert_reqs),
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 292, in create_urllib3_context
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     context.options |= options
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/usr/lib/python3.6/ssl.py", line 465, in options
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     super(SSLContext, SSLContext).options.__set__(self, value)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/usr/lib/python3.6/ssl.py", line 465, in options
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     super(SSLContext, SSLContext).options.__set__(self, value)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/usr/lib/python3.6/ssl.py", line 465, in options
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     super(SSLContext, SSLContext).options.__set__(self, value)
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   [Previous line repeated 303 more times]
2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova RecursionError: maximum recursion depth exceeded while calling a Python object


This seems to be a known eventlet issue: https://github.com/eventlet/eventlet/issues/371
Current eventlet version: v0.20.0
Fixed released in: v0.21.0

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1884808

Title:
  [SSL] neutron-rpc-server: RecursionError: maximum recursion depth
  exceeded while calling a Python object

Status in neutron:
  New

Bug description:
  py36 stable/rocky installation with SSL fails on nova notifier with:

  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova [-] Failed to notify nova on events: [{'server_uuid': 'd212f288-e1a6-4615-ae77-6db03cfc5460', 'name': 'network-vif-plugged', 'status': 'completed', 'tag': '3cc3ed7d-2717-441b-bcb2-bc2b31405eae'}]: RecursionError: maximum recursion depth exceeded while calling a Python object
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova Traceback (most recent call last):
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/neutron/notifiers/nova.py", line 245, in send_events
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     batched_events)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/novaclient/v2/server_external_events.py", line 39, in create
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     return_raw=True)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/novaclient/base.py", line 366, in _create
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     resp, body = self.api.client.post(url, body=body)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 334, in post
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     return self.request(url, 'POST', **kwargs)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/novaclient/client.py", line 77, in request
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     **kwargs)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 487, in request
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 213, in request
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     return self.session.request(url, method, **kwargs)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/session.py", line 684, in request
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     auth_headers = self.get_auth_headers(auth)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     return auth.get_headers(self, **kwargs)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/plugin.py", line 95, in get_headers
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     token = self.get_token(session)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 88, in get_token
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     return self.get_access(session).auth_token
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     self.auth_ref = self.get_auth_ref(session)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     self._plugin = self._do_create_plugin(session)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     authenticated=False)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 601, in get_discovery
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     authenticated=authenticated)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1399, in get_discovery
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     disc = Discover(session, url, authenticated=authenticated)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/discover.py", line 498, in __init__
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     authenticated=authenticated)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     resp = session.get(url, headers=headers, authenticated=authenticated)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/session.py", line 1011, in get
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     return self.request(url, 'GET', **kwargs)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/session.py", line 814, in request
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     resp = send(**kwargs)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/keystoneauth1/session.py", line 903, in _send_request
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     resp = self.session.request(method, url, **kwargs)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     resp = self.send(prep, **send_kwargs)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     r = adapter.send(request, **kwargs)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     timeout=timeout
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     chunked=chunked)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     self._validate_conn(conn)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/urllib3/connectionpool.py", line 849, in _validate_conn
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     conn.connect()
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/urllib3/connection.py", line 344, in connect
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     cert_reqs=resolve_cert_reqs(self.cert_reqs),
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/var/lib/openstack/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 292, in create_urllib3_context
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     context.options |= options
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/usr/lib/python3.6/ssl.py", line 465, in options
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     super(SSLContext, SSLContext).options.__set__(self, value)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/usr/lib/python3.6/ssl.py", line 465, in options
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     super(SSLContext, SSLContext).options.__set__(self, value)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   File "/usr/lib/python3.6/ssl.py", line 465, in options
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova     super(SSLContext, SSLContext).options.__set__(self, value)
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova   [Previous line repeated 303 more times]
  2020-06-23 06:14:35.960 11 ERROR neutron.notifiers.nova RecursionError: maximum recursion depth exceeded while calling a Python object

  
  This seems to be a known eventlet issue: https://github.com/eventlet/eventlet/issues/371
  Current eventlet version: v0.20.0
  Fixed released in: v0.21.0

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1884808/+subscriptions
Follow ups

[Bug 1884808] Re: [SSL] neutron-rpc-server: RecursionError: maximum recursion depth exceeded while calling a Python object
From: Brian Haley, 2022-12-06