yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1624791] Re: Horizon randomly fails to connect to the service APIs

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: "Dr. Jens Rosenboom" <j.rosenboom@xxxxxxxx>
Date: Tue, 22 Nov 2016 16:20:23 -0000
Reply-to: Bug 1624791 <1624791@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
We get the same issue when deploying Newton via Ubuntu UCA. Affected
package versions are:

python-openssl=16.1.0-1~cloud0
python-cryptography=1.5-2~cloud0

The error goes away if similar to the Ansible workaround we downgrade to
the packages from Xenial:

python-openssl=0.15.1-2build1
python-cryptography=1.2.3-1

Still not sure whether this issue could/should have a workaround in
Horizon or whether it is only a packaging issue.

** Also affects: horizon
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Dashboard (Horizon).
https://bugs.launchpad.net/bugs/1624791

Title:
  Horizon randomly fails to connect to the service APIs

Status in OpenStack Dashboard (Horizon):
  New
Status in openstack-ansible:
  Won't Fix

Bug description:
  This started occurring after upgrading to RC1. Before that, I was
  using Horizon's e7b4bdfe5d576766b34bf00cea3dcbcb42436420 plus cherry-
  picked changesets I2db4218e7351e0017a7a74114be6ac7af803476c and
  Idb58cebefab747f204e54ea6350db0852aec60f5.

  Running nova, cinder, etc. from the CLI seems to work perfectly.

  However Horizon randomly fails with "bad handshake: SysCallError(0,
  None)" when connecting to other services, including Keystone.

  The symptoms are:

  * Login is intermittent, it randomly succeeds or fails.

  * After being lucky enough to login, dashboards would randomly fail to
  display information.

  I tried to isolate and see if some of my 3 infra nodes was the
  culprit, without success.

  I also tried to destroy and reconstruct the horizon and keystone
  containers multiple times without success.

  I didn't try yet to downgrade Horizon to the
  e7b4bdfe5d576766b34bf00cea3dcbcb42436420 commit, I will try it ASAP
  and report the results here.

  
  [Sun Sep 18 01:42:52.285253 2016] [wsgi:error] [pid 6470:tid 139771996858112] Traceback (most recent call last):
  [Sun Sep 18 01:42:52.285256 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/instances/tables.py", line 389, in allowed
  [Sun Sep 18 01:42:52.285259 2016] [wsgi:error] [pid 6470:tid 139771996858112]     limits = api.nova.tenant_absolute_limits(request, reserved=True)
  [Sun Sep 18 01:42:52.285261 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 949, in tenant_absolute_limits
  [Sun Sep 18 01:42:52.285264 2016] [wsgi:error] [pid 6470:tid 139771996858112]     limits = novaclient(request).limits.get(reserved=reserved).absolute
  [Sun Sep 18 01:42:52.285266 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../novaclient/v2/limits.py", line 100, in get
  [Sun Sep 18 01:42:52.285269 2016] [wsgi:error] [pid 6470:tid 139771996858112]     return self._get("/limits%s" % query_string, "limits")
  [Sun Sep 18 01:42:52.285271 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../novaclient/base.py", line 346, in _get
  [Sun Sep 18 01:42:52.285273 2016] [wsgi:error] [pid 6470:tid 139771996858112]     resp, body = self.api.client.get(url)
  [Sun Sep 18 01:42:52.285276 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../novaclient/client.py", line 480, in get
  [Sun Sep 18 01:42:52.285278 2016] [wsgi:error] [pid 6470:tid 139771996858112]     return self._cs_request(url, 'GET', **kwargs)
  [Sun Sep 18 01:42:52.285280 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../novaclient/client.py", line 458, in _cs_request
  [Sun Sep 18 01:42:52.285282 2016] [wsgi:error] [pid 6470:tid 139771996858112]     resp, body = self._time_request(url, method, **kwargs)
  [Sun Sep 18 01:42:52.285285 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../novaclient/client.py", line 431, in _time_request
  [Sun Sep 18 01:42:52.285287 2016] [wsgi:error] [pid 6470:tid 139771996858112]     resp, body = self.request(url, method, **kwargs)
  [Sun Sep 18 01:42:52.285289 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../novaclient/client.py", line 396, in request
  [Sun Sep 18 01:42:52.285299 2016] [wsgi:error] [pid 6470:tid 139771996858112]     **kwargs)
  [Sun Sep 18 01:42:52.285303 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../requests/api.py", line 56, in request
  [Sun Sep 18 01:42:52.285307 2016] [wsgi:error] [pid 6470:tid 139771996858112]     return session.request(method=method, url=url, **kwargs)
  [Sun Sep 18 01:42:52.285310 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../requests/sessions.py", line 475, in request
  [Sun Sep 18 01:42:52.285313 2016] [wsgi:error] [pid 6470:tid 139771996858112]     resp = self.send(prep, **send_kwargs)
  [Sun Sep 18 01:42:52.285317 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../requests/sessions.py", line 596, in send
  [Sun Sep 18 01:42:52.285320 2016] [wsgi:error] [pid 6470:tid 139771996858112]     r = adapter.send(request, **kwargs)
  [Sun Sep 18 01:42:52.285324 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../requests/adapters.py", line 497, in send
  [Sun Sep 18 01:42:52.285328 2016] [wsgi:error] [pid 6470:tid 139771996858112]     raise SSLError(e, request=request)
  [Sun Sep 18 01:42:52.285330 2016] [wsgi:error] [pid 6470:tid 139771996858112] SSLError: ('bad handshake: SysCallError(0, None)',)
  [Sun Sep 18 01:42:52.308704 2016] [wsgi:error] [pid 6470:tid 139771996858112] Unable to retrieve project list.
  [Sun Sep 18 01:42:52.308729 2016] [wsgi:error] [pid 6470:tid 139771996858112] Traceback (most recent call last):
  [Sun Sep 18 01:42:52.308734 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../openstack_auth/user.py", line 318, in authorized_tenants
  [Sun Sep 18 01:42:52.308738 2016] [wsgi:error] [pid 6470:tid 139771996858112]     is_federated=self.is_federated)
  [Sun Sep 18 01:42:52.308742 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../openstack_auth/utils.py", line 365, in get_project_list
  [Sun Sep 18 01:42:52.308747 2016] [wsgi:error] [pid 6470:tid 139771996858112]     projects = client.projects.list(user=kwargs.get('user_id'))
  [Sun Sep 18 01:42:52.308751 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../positional/__init__.py", line 101, in inner
  [Sun Sep 18 01:42:52.308755 2016] [wsgi:error] [pid 6470:tid 139771996858112]     return wrapped(*args, **kwargs)
  [Sun Sep 18 01:42:52.308759 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneclient/v3/projects.py", line 119, in list
  [Sun Sep 18 01:42:52.308763 2016] [wsgi:error] [pid 6470:tid 139771996858112]     **kwargs)
  [Sun Sep 18 01:42:52.308767 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneclient/base.py", line 74, in func
  [Sun Sep 18 01:42:52.308771 2016] [wsgi:error] [pid 6470:tid 139771996858112]     return f(*args, **new_kwargs)
  [Sun Sep 18 01:42:52.308774 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneclient/base.py", line 386, in list
  [Sun Sep 18 01:42:52.308778 2016] [wsgi:error] [pid 6470:tid 139771996858112]     self.collection_key)
  [Sun Sep 18 01:42:52.308782 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneclient/base.py", line 124, in _list
  [Sun Sep 18 01:42:52.308797 2016] [wsgi:error] [pid 6470:tid 139771996858112]     resp, body = self.client.get(url, **kwargs)
  [Sun Sep 18 01:42:52.308802 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneauth1/adapter.py", line 187, in get
  [Sun Sep 18 01:42:52.308806 2016] [wsgi:error] [pid 6470:tid 139771996858112]     return self.request(url, 'GET', **kwargs)
  [Sun Sep 18 01:42:52.308810 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneauth1/adapter.py", line 344, in request
  [Sun Sep 18 01:42:52.308814 2016] [wsgi:error] [pid 6470:tid 139771996858112]     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  [Sun Sep 18 01:42:52.308817 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneauth1/adapter.py", line 112, in request
  [Sun Sep 18 01:42:52.308822 2016] [wsgi:error] [pid 6470:tid 139771996858112]     return self.session.request(url, method, **kwargs)
  [Sun Sep 18 01:42:52.308825 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../positional/__init__.py", line 101, in inner
  [Sun Sep 18 01:42:52.308829 2016] [wsgi:error] [pid 6470:tid 139771996858112]     return wrapped(*args, **kwargs)
  [Sun Sep 18 01:42:52.308833 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneauth1/session.py", line 555, in request
  [Sun Sep 18 01:42:52.308837 2016] [wsgi:error] [pid 6470:tid 139771996858112]     resp = send(**kwargs)
  [Sun Sep 18 01:42:52.308840 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneauth1/session.py", line 593, in _send_request
  [Sun Sep 18 01:42:52.308844 2016] [wsgi:error] [pid 6470:tid 139771996858112]     raise exceptions.SSLError(msg)
  [Sun Sep 18 01:42:52.308848 2016] [wsgi:error] [pid 6470:tid 139771996858112] SSLError: SSL exception connecting to https://cloud.ufscar.br:5000/v3/users/f1a04d6876eb4252b78c01059ca557bd/projects: ('bad handshake: SysCallError(0, None)',)

To manage notifications about this bug go to:
https://bugs.launchpad.net/horizon/+bug/1624791/+subscriptions