yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1902696] [NEW] nova-compute fails with Unhandled error: TypeError: _wrap_socket_sni() got an unexpected keyword argument 'ca_certs'

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Michal Nasiadka <1902696@xxxxxxxxxxxxxxxxxx>
Date: Tue, 03 Nov 2020 12:20:31 -0000
Reply-to: Bug 1902696 <1902696@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

In Kolla-Ansible CI we are observing those errors from 2nd of November:

Link to Zuul job list: https://zuul.openstack.org/builds?job_name=kolla-
ansible-ubuntu-source&job_name=kolla-ansible-
centos8-source&branch=master

Link to log:
https://zuul.openstack.org/build/28d615b8c06640afae885c3ef5a2f316/log/primary/logs/kolla/nova
/nova-compute.txt


2020-11-03 10:32:33.812 6 CRITICAL nova [req-d101564c-d4a7-400d-b5bc-b061c5087b1b - - - - -] Unhandled error: TypeError: _wrap_socket_sni() got an unexpected keyword argument 'ca_certs'
8	2020-11-03 10:32:33.812 6 ERROR nova Traceback (most recent call last):
9	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/bin/nova-compute", line 8, in <module>
10	2020-11-03 10:32:33.812 6 ERROR nova     sys.exit(main())
11	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/cmd/compute.py", line 59, in main
12	2020-11-03 10:32:33.812 6 ERROR nova     topic=compute_rpcapi.RPC_TOPIC)
13	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/service.py", line 256, in create
14	2020-11-03 10:32:33.812 6 ERROR nova     periodic_interval_max=periodic_interval_max)
15	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/service.py", line 115, in __init__
16	2020-11-03 10:32:33.812 6 ERROR nova     conductor_api.wait_until_ready(context.get_admin_context())
17	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/conductor/api.py", line 68, in wait_until_ready
18	2020-11-03 10:32:33.812 6 ERROR nova     timeout=timeout)
19	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/baserpc.py", line 58, in ping
20	2020-11-03 10:32:33.812 6 ERROR nova     return cctxt.call(context, 'ping', arg=arg_p)
21	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 179, in call
22	2020-11-03 10:32:33.812 6 ERROR nova     transport_options=self.transport_options)
23	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/transport.py", line 128, in _send
24	2020-11-03 10:32:33.812 6 ERROR nova     transport_options=transport_options)
25	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 654, in send
26	2020-11-03 10:32:33.812 6 ERROR nova     transport_options=transport_options)
27	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 598, in _send
28	2020-11-03 10:32:33.812 6 ERROR nova     msg.update({'_reply_q': self._get_reply_q()})
29	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 579, in _get_reply_q
30	2020-11-03 10:32:33.812 6 ERROR nova     conn = self._get_connection(rpc_common.PURPOSE_LISTEN)
31	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 570, in _get_connection
32	2020-11-03 10:32:33.812 6 ERROR nova     purpose=purpose)
33	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/common.py", line 425, in __init__
34	2020-11-03 10:32:33.812 6 ERROR nova     self.connection = connection_pool.create(purpose)
35	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/pool.py", line 146, in create
36	2020-11-03 10:32:33.812 6 ERROR nova     return self.connection_cls(self.conf, self.url, purpose)
37	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 618, in __init__
38	2020-11-03 10:32:33.812 6 ERROR nova     self.ensure_connection()
39	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 735, in ensure_connection
40	2020-11-03 10:32:33.812 6 ERROR nova     self.connection.ensure_connection(errback=on_error)
41	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/kombu/connection.py", line 383, in ensure_connection
42	2020-11-03 10:32:33.812 6 ERROR nova     self._ensure_connection(*args, **kwargs)
43	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/kombu/connection.py", line 439, in _ensure_connection
44	2020-11-03 10:32:33.812 6 ERROR nova     callback, timeout=timeout
45	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/kombu/utils/functional.py", line 325, in retry_over_time
46	2020-11-03 10:32:33.812 6 ERROR nova     return fun(*args, **kwargs)
47	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/kombu/connection.py", line 866, in _connection_factory
48	2020-11-03 10:32:33.812 6 ERROR nova     self._connection = self._establish_connection()
49	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/kombu/connection.py", line 801, in _establish_connection
50	2020-11-03 10:32:33.812 6 ERROR nova     conn = self.transport.establish_connection()
51	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/kombu/transport/pyamqp.py", line 128, in establish_connection
52	2020-11-03 10:32:33.812 6 ERROR nova     conn.connect()
53	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/amqp/connection.py", line 314, in connect
54	2020-11-03 10:32:33.812 6 ERROR nova     self.transport.connect()
55	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/amqp/transport.py", line 78, in connect
56	2020-11-03 10:32:33.812 6 ERROR nova     self.socket_settings, self.read_timeout, self.write_timeout,
57	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/amqp/transport.py", line 188, in _init_socket
58	2020-11-03 10:32:33.812 6 ERROR nova     self._setup_transport()
59	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/amqp/transport.py", line 323, in _setup_transport
60	2020-11-03 10:32:33.812 6 ERROR nova     self.sock = self._wrap_socket(self.sock, **self.sslopts)
61	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/amqp/transport.py", line 330, in _wrap_socket
62	2020-11-03 10:32:33.812 6 ERROR nova     return self._wrap_socket_sni(sock, **sslopts)
63	2020-11-03 10:32:33.812 6 ERROR nova TypeError: _wrap_socket_sni() got an unexpected keyword argument 'ca_certs'
64	2020-11-03 10:32:33.812 6 ERROR nova 
65

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1902696

Title:
  nova-compute fails with Unhandled error: TypeError: _wrap_socket_sni()
  got an unexpected keyword argument 'ca_certs'

Status in OpenStack Compute (nova):
  New

Bug description:
  In Kolla-Ansible CI we are observing those errors from 2nd of
  November:

  Link to Zuul job list: https://zuul.openstack.org/builds?job_name
  =kolla-ansible-ubuntu-source&job_name=kolla-ansible-
  centos8-source&branch=master

  Link to log:
  https://zuul.openstack.org/build/28d615b8c06640afae885c3ef5a2f316/log/primary/logs/kolla/nova
  /nova-compute.txt

  
  2020-11-03 10:32:33.812 6 CRITICAL nova [req-d101564c-d4a7-400d-b5bc-b061c5087b1b - - - - -] Unhandled error: TypeError: _wrap_socket_sni() got an unexpected keyword argument 'ca_certs'
  8	2020-11-03 10:32:33.812 6 ERROR nova Traceback (most recent call last):
  9	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/bin/nova-compute", line 8, in <module>
  10	2020-11-03 10:32:33.812 6 ERROR nova     sys.exit(main())
  11	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/cmd/compute.py", line 59, in main
  12	2020-11-03 10:32:33.812 6 ERROR nova     topic=compute_rpcapi.RPC_TOPIC)
  13	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/service.py", line 256, in create
  14	2020-11-03 10:32:33.812 6 ERROR nova     periodic_interval_max=periodic_interval_max)
  15	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/service.py", line 115, in __init__
  16	2020-11-03 10:32:33.812 6 ERROR nova     conductor_api.wait_until_ready(context.get_admin_context())
  17	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/conductor/api.py", line 68, in wait_until_ready
  18	2020-11-03 10:32:33.812 6 ERROR nova     timeout=timeout)
  19	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/baserpc.py", line 58, in ping
  20	2020-11-03 10:32:33.812 6 ERROR nova     return cctxt.call(context, 'ping', arg=arg_p)
  21	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 179, in call
  22	2020-11-03 10:32:33.812 6 ERROR nova     transport_options=self.transport_options)
  23	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/transport.py", line 128, in _send
  24	2020-11-03 10:32:33.812 6 ERROR nova     transport_options=transport_options)
  25	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 654, in send
  26	2020-11-03 10:32:33.812 6 ERROR nova     transport_options=transport_options)
  27	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 598, in _send
  28	2020-11-03 10:32:33.812 6 ERROR nova     msg.update({'_reply_q': self._get_reply_q()})
  29	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 579, in _get_reply_q
  30	2020-11-03 10:32:33.812 6 ERROR nova     conn = self._get_connection(rpc_common.PURPOSE_LISTEN)
  31	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 570, in _get_connection
  32	2020-11-03 10:32:33.812 6 ERROR nova     purpose=purpose)
  33	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/common.py", line 425, in __init__
  34	2020-11-03 10:32:33.812 6 ERROR nova     self.connection = connection_pool.create(purpose)
  35	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/pool.py", line 146, in create
  36	2020-11-03 10:32:33.812 6 ERROR nova     return self.connection_cls(self.conf, self.url, purpose)
  37	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 618, in __init__
  38	2020-11-03 10:32:33.812 6 ERROR nova     self.ensure_connection()
  39	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 735, in ensure_connection
  40	2020-11-03 10:32:33.812 6 ERROR nova     self.connection.ensure_connection(errback=on_error)
  41	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/kombu/connection.py", line 383, in ensure_connection
  42	2020-11-03 10:32:33.812 6 ERROR nova     self._ensure_connection(*args, **kwargs)
  43	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/kombu/connection.py", line 439, in _ensure_connection
  44	2020-11-03 10:32:33.812 6 ERROR nova     callback, timeout=timeout
  45	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/kombu/utils/functional.py", line 325, in retry_over_time
  46	2020-11-03 10:32:33.812 6 ERROR nova     return fun(*args, **kwargs)
  47	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/kombu/connection.py", line 866, in _connection_factory
  48	2020-11-03 10:32:33.812 6 ERROR nova     self._connection = self._establish_connection()
  49	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/kombu/connection.py", line 801, in _establish_connection
  50	2020-11-03 10:32:33.812 6 ERROR nova     conn = self.transport.establish_connection()
  51	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/kombu/transport/pyamqp.py", line 128, in establish_connection
  52	2020-11-03 10:32:33.812 6 ERROR nova     conn.connect()
  53	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/amqp/connection.py", line 314, in connect
  54	2020-11-03 10:32:33.812 6 ERROR nova     self.transport.connect()
  55	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/amqp/transport.py", line 78, in connect
  56	2020-11-03 10:32:33.812 6 ERROR nova     self.socket_settings, self.read_timeout, self.write_timeout,
  57	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/amqp/transport.py", line 188, in _init_socket
  58	2020-11-03 10:32:33.812 6 ERROR nova     self._setup_transport()
  59	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/amqp/transport.py", line 323, in _setup_transport
  60	2020-11-03 10:32:33.812 6 ERROR nova     self.sock = self._wrap_socket(self.sock, **self.sslopts)
  61	2020-11-03 10:32:33.812 6 ERROR nova   File "/var/lib/kolla/venv/lib/python3.6/site-packages/amqp/transport.py", line 330, in _wrap_socket
  62	2020-11-03 10:32:33.812 6 ERROR nova     return self._wrap_socket_sni(sock, **sslopts)
  63	2020-11-03 10:32:33.812 6 ERROR nova TypeError: _wrap_socket_sni() got an unexpected keyword argument 'ca_certs'
  64	2020-11-03 10:32:33.812 6 ERROR nova 
  65

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1902696/+subscriptions
Follow ups

[Bug 1902696] Re: nova-compute fails with Unhandled error: TypeError: _wrap_socket_sni() got an unexpected keyword argument 'ca_certs'
From: Balazs Gibizer, 2020-11-03