← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #61380

[Bug 1624791] Re: Horizon randomly fails to connect to the service APIs

  Thread Previous • Date Previous • • Thread Next 
Reviewed:  https://review.openstack.org/430208
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-pip_install/commit/?id=fe4fa1882229c22478357c82a71a4227d868b8c5
Submitter: Jenkins
Branch:    master

commit fe4fa1882229c22478357c82a71a4227d868b8c5
Author: Jean-Philippe Evrard <jean-philippe.evrard@xxxxxxxxxxxxxxx>
Date:   Tue Feb 7 11:26:39 2017 +0000

    Use RDO/UCA everywhere
    
    The pip_install role installs binary dependencies, like python-openssl
    (on Ubuntu), or pyOpenSSL (on Centos).
    
    By default, we didn't configure extra sources (like UCA or RDO)
    in the pip_install role. Because the repo is built using UCA or RDO
    (current default), we could have inconsistency issues:
    containers that haven't UCA or RDO applied during their role execution
    would run with a venv packaged for a different python-openssl
    (or python-cryptography).
    
    This commit brings the consistency by installing UCA/RDO everywhere.
    
    Closes-Bug: 1624791
    
    Change-Id: I9b5cd40b2972c93af348d4ddfde21a038cf9becc
    Signed-off-by: Jean-Philippe Evrard <jean-philippe.evrard@xxxxxxxxxxxxxxx>


** Changed in: openstack-ansible
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Dashboard (Horizon).
https://bugs.launchpad.net/bugs/1624791

Title:
  Horizon randomly fails to connect to the service APIs

Status in OpenStack Dashboard (Horizon):
  New
Status in openstack-ansible:
  Fix Released

Bug description:
  This started occurring after upgrading to RC1. Before that, I was
  using Horizon's e7b4bdfe5d576766b34bf00cea3dcbcb42436420 plus cherry-
  picked changesets I2db4218e7351e0017a7a74114be6ac7af803476c and
  Idb58cebefab747f204e54ea6350db0852aec60f5.

  Running nova, cinder, etc. from the CLI seems to work perfectly.

  However Horizon randomly fails with "bad handshake: SysCallError(0,
  None)" when connecting to other services, including Keystone.

  The symptoms are:

  * Login is intermittent, it randomly succeeds or fails.

  * After being lucky enough to login, dashboards would randomly fail to
  display information.

  I tried to isolate and see if some of my 3 infra nodes was the
  culprit, without success.

  I also tried to destroy and reconstruct the horizon and keystone
  containers multiple times without success.

  I didn't try yet to downgrade Horizon to the
  e7b4bdfe5d576766b34bf00cea3dcbcb42436420 commit, I will try it ASAP
  and report the results here.

  
  [Sun Sep 18 01:42:52.285253 2016] [wsgi:error] [pid 6470:tid 139771996858112] Traceback (most recent call last):
  [Sun Sep 18 01:42:52.285256 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/instances/tables.py", line 389, in allowed
  [Sun Sep 18 01:42:52.285259 2016] [wsgi:error] [pid 6470:tid 139771996858112]     limits = api.nova.tenant_absolute_limits(request, reserved=True)
  [Sun Sep 18 01:42:52.285261 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 949, in tenant_absolute_limits
  [Sun Sep 18 01:42:52.285264 2016] [wsgi:error] [pid 6470:tid 139771996858112]     limits = novaclient(request).limits.get(reserved=reserved).absolute
  [Sun Sep 18 01:42:52.285266 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../novaclient/v2/limits.py", line 100, in get
  [Sun Sep 18 01:42:52.285269 2016] [wsgi:error] [pid 6470:tid 139771996858112]     return self._get("/limits%s" % query_string, "limits")
  [Sun Sep 18 01:42:52.285271 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../novaclient/base.py", line 346, in _get
  [Sun Sep 18 01:42:52.285273 2016] [wsgi:error] [pid 6470:tid 139771996858112]     resp, body = self.api.client.get(url)
  [Sun Sep 18 01:42:52.285276 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../novaclient/client.py", line 480, in get
  [Sun Sep 18 01:42:52.285278 2016] [wsgi:error] [pid 6470:tid 139771996858112]     return self._cs_request(url, 'GET', **kwargs)
  [Sun Sep 18 01:42:52.285280 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../novaclient/client.py", line 458, in _cs_request
  [Sun Sep 18 01:42:52.285282 2016] [wsgi:error] [pid 6470:tid 139771996858112]     resp, body = self._time_request(url, method, **kwargs)
  [Sun Sep 18 01:42:52.285285 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../novaclient/client.py", line 431, in _time_request
  [Sun Sep 18 01:42:52.285287 2016] [wsgi:error] [pid 6470:tid 139771996858112]     resp, body = self.request(url, method, **kwargs)
  [Sun Sep 18 01:42:52.285289 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../novaclient/client.py", line 396, in request
  [Sun Sep 18 01:42:52.285299 2016] [wsgi:error] [pid 6470:tid 139771996858112]     **kwargs)
  [Sun Sep 18 01:42:52.285303 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../requests/api.py", line 56, in request
  [Sun Sep 18 01:42:52.285307 2016] [wsgi:error] [pid 6470:tid 139771996858112]     return session.request(method=method, url=url, **kwargs)
  [Sun Sep 18 01:42:52.285310 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../requests/sessions.py", line 475, in request
  [Sun Sep 18 01:42:52.285313 2016] [wsgi:error] [pid 6470:tid 139771996858112]     resp = self.send(prep, **send_kwargs)
  [Sun Sep 18 01:42:52.285317 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../requests/sessions.py", line 596, in send
  [Sun Sep 18 01:42:52.285320 2016] [wsgi:error] [pid 6470:tid 139771996858112]     r = adapter.send(request, **kwargs)
  [Sun Sep 18 01:42:52.285324 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../requests/adapters.py", line 497, in send
  [Sun Sep 18 01:42:52.285328 2016] [wsgi:error] [pid 6470:tid 139771996858112]     raise SSLError(e, request=request)
  [Sun Sep 18 01:42:52.285330 2016] [wsgi:error] [pid 6470:tid 139771996858112] SSLError: ('bad handshake: SysCallError(0, None)',)
  [Sun Sep 18 01:42:52.308704 2016] [wsgi:error] [pid 6470:tid 139771996858112] Unable to retrieve project list.
  [Sun Sep 18 01:42:52.308729 2016] [wsgi:error] [pid 6470:tid 139771996858112] Traceback (most recent call last):
  [Sun Sep 18 01:42:52.308734 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../openstack_auth/user.py", line 318, in authorized_tenants
  [Sun Sep 18 01:42:52.308738 2016] [wsgi:error] [pid 6470:tid 139771996858112]     is_federated=self.is_federated)
  [Sun Sep 18 01:42:52.308742 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../openstack_auth/utils.py", line 365, in get_project_list
  [Sun Sep 18 01:42:52.308747 2016] [wsgi:error] [pid 6470:tid 139771996858112]     projects = client.projects.list(user=kwargs.get('user_id'))
  [Sun Sep 18 01:42:52.308751 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../positional/__init__.py", line 101, in inner
  [Sun Sep 18 01:42:52.308755 2016] [wsgi:error] [pid 6470:tid 139771996858112]     return wrapped(*args, **kwargs)
  [Sun Sep 18 01:42:52.308759 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneclient/v3/projects.py", line 119, in list
  [Sun Sep 18 01:42:52.308763 2016] [wsgi:error] [pid 6470:tid 139771996858112]     **kwargs)
  [Sun Sep 18 01:42:52.308767 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneclient/base.py", line 74, in func
  [Sun Sep 18 01:42:52.308771 2016] [wsgi:error] [pid 6470:tid 139771996858112]     return f(*args, **new_kwargs)
  [Sun Sep 18 01:42:52.308774 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneclient/base.py", line 386, in list
  [Sun Sep 18 01:42:52.308778 2016] [wsgi:error] [pid 6470:tid 139771996858112]     self.collection_key)
  [Sun Sep 18 01:42:52.308782 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneclient/base.py", line 124, in _list
  [Sun Sep 18 01:42:52.308797 2016] [wsgi:error] [pid 6470:tid 139771996858112]     resp, body = self.client.get(url, **kwargs)
  [Sun Sep 18 01:42:52.308802 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneauth1/adapter.py", line 187, in get
  [Sun Sep 18 01:42:52.308806 2016] [wsgi:error] [pid 6470:tid 139771996858112]     return self.request(url, 'GET', **kwargs)
  [Sun Sep 18 01:42:52.308810 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneauth1/adapter.py", line 344, in request
  [Sun Sep 18 01:42:52.308814 2016] [wsgi:error] [pid 6470:tid 139771996858112]     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  [Sun Sep 18 01:42:52.308817 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneauth1/adapter.py", line 112, in request
  [Sun Sep 18 01:42:52.308822 2016] [wsgi:error] [pid 6470:tid 139771996858112]     return self.session.request(url, method, **kwargs)
  [Sun Sep 18 01:42:52.308825 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../positional/__init__.py", line 101, in inner
  [Sun Sep 18 01:42:52.308829 2016] [wsgi:error] [pid 6470:tid 139771996858112]     return wrapped(*args, **kwargs)
  [Sun Sep 18 01:42:52.308833 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneauth1/session.py", line 555, in request
  [Sun Sep 18 01:42:52.308837 2016] [wsgi:error] [pid 6470:tid 139771996858112]     resp = send(**kwargs)
  [Sun Sep 18 01:42:52.308840 2016] [wsgi:error] [pid 6470:tid 139771996858112]   File "/openstack/venvs/horizon-14.0.0/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneauth1/session.py", line 593, in _send_request
  [Sun Sep 18 01:42:52.308844 2016] [wsgi:error] [pid 6470:tid 139771996858112]     raise exceptions.SSLError(msg)
  [Sun Sep 18 01:42:52.308848 2016] [wsgi:error] [pid 6470:tid 139771996858112] SSLError: SSL exception connecting to https://cloud.ufscar.br:5000/v3/users/f1a04d6876eb4252b78c01059ca557bd/projects: ('bad handshake: SysCallError(0, None)',)

To manage notifications about this bug go to:
https://bugs.launchpad.net/horizon/+bug/1624791/+subscriptions
  Thread Previous • Date Previous • • Thread Next