yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1630664] Re: Intermittent failure in n-api connecting to neutron to list ports after TLS was enabled in CI

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Ihar Hrachyshka <1630664@xxxxxxxxxxxxxxxxxx>
Date: Fri, 28 Apr 2017 15:25:31 -0000
Reply-to: Bug 1630664 <1630664@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
I am seeing that still happening with keystone token fetch. It just hit
this Ocata patch: https://review.openstack.org/#/c/460909/

In http://logs.openstack.org/09/460909/2/check/gate-tempest-dsvm-
neutron-linuxbridge-ubuntu-xenial/67904c9/logs/apache/tls-
proxy_error.txt.gz we see:

[Fri Apr 28 12:46:47.763965 2017] [proxy_http:error] [pid 30068:tid 140271090042624] (20014)Internal error (specific information not available): [client 104.130.119.120:50002] [frontend 104.130.119.120:443] AH01102: error reading status line from remote server 104.130.119.120:80
[Fri Apr 28 12:46:47.764003 2017] [proxy:error] [pid 30068:tid 140271090042624] [client 104.130.119.120:50002] [frontend 104.130.119.120:443] AH00898: Error reading from remote server returned by /identity_admin/v3/auth/tokens

The request that triggered the failure doesn't seem to show up in
keystone log.

** Project changed: nova => devstack

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1630664

Title:
  Intermittent failure in n-api connecting to neutron to list ports
  after TLS was enabled in CI

Status in devstack:
  Confirmed

Bug description:
  Seen here:

  http://logs.openstack.org/00/382000/2/check/gate-tempest-dsvm-neutron-
  full-ubuntu-
  xenial/07e5243/logs/screen-n-api.txt.gz?level=TRACE#_2016-10-05_14_35_04_333

  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack [req-c1bbc78f-89e4-4de2-956d-9b71f8ad1a87 tempest-TestNetworkAdvancedServerOps-960076899 tempest-TestNetworkAdvancedServerOps-960076899] Caught error: Unable to establish connection to https://127.0.0.1:9696/v2.0/ports.json?device_id=bf9a5908-ebdd-4f67-aae4-a0a3e0cf0d09
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack Traceback (most recent call last):
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/__init__.py", line 89, in __call__
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     return req.get_response(self.application)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     application, catch_exc_info=False)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     app_iter = application(self.environ, start_response)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     return resp(environ, start_response)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     return self.func(req, *args, **kwargs)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 323, in __call__
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     response = req.get_response(self._app)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     application, catch_exc_info=False)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     app_iter = application(self.environ, start_response)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     return resp(environ, start_response)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     return resp(environ, start_response)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 141, in __call__
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     response = self.app(environ, start_response)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     return resp(environ, start_response)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     return self.func(req, *args, **kwargs)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 607, in __call__
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     content_type, body, accept)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 687, in _process_stack
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     request, action_args)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 553, in process_extensions
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     **action_args)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/compute/security_groups.py", line 489, in show
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     return self._show(req, resp_obj)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/compute/security_groups.py", line 485, in _show
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     self._extend_servers(req, [resp_obj.obj['server']])
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/compute/security_groups.py", line 467, in _extend_servers
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     servers))
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/opt/stack/new/nova/nova/network/security_group/neutron_driver.py", line 380, in get_instances_security_groups_bindings
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     ports = self._get_ports_from_server_list(servers, neutron)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/opt/stack/new/nova/nova/network/security_group/neutron_driver.py", line 339, in _get_ports_from_server_list
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     ports.extend(neutron.list_ports(**search_opts).get('ports'))
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/opt/stack/new/nova/nova/network/neutronv2/api.py", line 96, in wrapper
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     ret = obj(*args, **kwargs)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 742, in list_ports
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     **_params)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/opt/stack/new/nova/nova/network/neutronv2/api.py", line 96, in wrapper
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     ret = obj(*args, **kwargs)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 375, in list
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     for r in self._pagination(collection, path, **params):
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 390, in _pagination
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     res = self.get(path, params=params)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/opt/stack/new/nova/nova/network/neutronv2/api.py", line 96, in wrapper
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     ret = obj(*args, **kwargs)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 360, in get
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     headers=headers, params=params)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/opt/stack/new/nova/nova/network/neutronv2/api.py", line 96, in wrapper
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     ret = obj(*args, **kwargs)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 337, in retry_request
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     headers=headers, params=params)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/opt/stack/new/nova/nova/network/neutronv2/api.py", line 96, in wrapper
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     ret = obj(*args, **kwargs)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 288, in do_request
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     resp, replybody = self.httpclient.do_request(action, method, body=body)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 311, in do_request
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     return self.request(url, method, **kwargs)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 299, in request
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     resp = super(SessionClient, self).request(*args, **kwargs)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 112, in request
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     return self.session.request(url, method, **kwargs)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     return wrapped(*args, **kwargs)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 555, in request
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     resp = send(**kwargs)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 599, in _send_request
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack     raise exceptions.ConnectFailure(msg)
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack ConnectFailure: Unable to establish connection to https://127.0.0.1:9696/v2.0/ports.json?device_id=bf9a5908-ebdd-4f67-aae4-a0a3e0cf0d09
  2016-10-05 14:35:04.333 18048 ERROR nova.api.openstack 

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ConnectFailure%3A%20Unable%20to%20establish%20connection%20to%20https%3A%2F%2F127.0.0.1%3A9696%2Fv2.0%2F%5C%22%20AND%20tags%3A%5C%22screen-n-api.txt%5C%22%20AND%20voting%3A1&from=10d

  According to logstash this started happening around 9/27, which lines
  up with when TLS was enabled in ocata (master) branch jobs:

  https://review.openstack.org/#/c/375776/

  So I assume those are related.

To manage notifications about this bug go to:
https://bugs.launchpad.net/devstack/+bug/1630664/+subscriptions
References

[Bug 1630664] [NEW] Intermittent failure in n-api connecting to neutron to list ports after TLS was enabled in CI
From: Matt Riedemann, 2016-10-05