yahoo-eng-team team mailing list archive

[Ken'ichi Ohmichi <ken1ohmichi@xxxxxxxxx>]

I faced this problem on the gate job like
http://logs.openstack.org/59/426459/5/check/gate-tempest-dsvm-neutron-
dvr-ubuntu-xenial/c3d2fac/logs

nova-api could not connect to neutron-server via keystoneauth1 like

http://logs.openstack.org/59/426459/5/check/gate-tempest-dsvm-neutron-
dvr-ubuntu-
xenial/c3d2fac/logs/screen-n-api.txt.gz#_2017-02-10_17_26_20_560

 Traceback (most recent call last):
   File "/opt/stack/new/nova/nova/api/openstack/extensions.py", line 338, in wrapped
     return f(*args, **kwargs)
   File "/opt/stack/new/nova/nova/api/openstack/compute/attach_interfaces.py", line 74, in show
     port_info = self.network_api.show_port(context, port_id)
   File "/opt/stack/new/nova/nova/network/neutronv2/api.py", line 1243, in show_port
     return dict(port=self._show_port(context, port_id))
   File "/opt/stack/new/nova/nova/network/neutronv2/api.py", line 1261, in _show_port
     result = neutron_client.show_port(port_id)
   File "/opt/stack/new/nova/nova/network/neutronv2/api.py", line 99, in wrapper
     ret = obj(*args, **kwargs)
   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 760, in show_port
     return self.get(self.port_path % (port), params=_params)
   File "/opt/stack/new/nova/nova/network/neutronv2/api.py", line 99, in wrapper
     ret = obj(*args, **kwargs)
   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 361, in get
     headers=headers, params=params)
   File "/opt/stack/new/nova/nova/network/neutronv2/api.py", line 99, in wrapper
     ret = obj(*args, **kwargs)
   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 338, in retry_request
     headers=headers, params=params)
   File "/opt/stack/new/nova/nova/network/neutronv2/api.py", line 99, in wrapper
     ret = obj(*args, **kwargs)
   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 289, in do_request
     resp, replybody = self.httpclient.do_request(action, method, body=body)
   File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 314, in do_request
     return self.request(url, method, **kwargs)
   File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 302, in request
     resp = super(SessionClient, self).request(*args, **kwargs)
   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 142, in request
     return self.session.request(url, method, **kwargs)
   File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
     return wrapped(*args, **kwargs)
   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 616, in request
     resp = send(**kwargs)
   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 690, in _send_request
     raise exceptions.ConnectFailure(msg)
 ConnectFailure: Unable to establish connection to https://10.25.148.68:9696/v2.0/ports/d5a00be6-bb0e-4911-b1a9-ffbd25544947.json: ('Connection aborted.', BadStatusLine("''",))

and neutron-server's log doesn't contain any access log about this.

** Changed in: keystone
       Status: Opinion => Confirmed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1535640

Title:
  Sometimes getting unable to establish connection error in Keystone
  client

Status in OpenStack Identity (keystone):
  Opinion

Bug description:
  In Openstack HA environment, sometimes  Kesystone client is throwing
  the error "unable to establish connection error". This is happening
  once in fifteen times. The error and stack trace of the error log is
  given below.

  Error: 1) keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://Controller-VIP:8004/v1/ca1d10786fdc41698e276a8f22d98741/stacks
  2) ConnectionRefused: Unable to establish connection to http://Controller-VIP:9696/v2.0/ports.json?tenant_id=ca1d10786fdc41698e276a8f22d98741&device_id=183bea96-b0da-42a5-a731-713882598624

  Stack trace of the error:

  NFO (connectionpool) Starting new HTTP connection (1): Controller-VIP
  Traceback (most recent call last):
    File "/usr/bin/heat", line 10, in <module>
      sys.exit(main())
    File "/usr/lib/python2.7/dist-packages/heatclient/shell.py", line 706, in main
      HeatShell().main(args)
    File "/usr/lib/python2.7/dist-packages/heatclient/shell.py", line 656, in main
      args.func(client, args)
    File "/usr/lib/python2.7/dist-packages/heatclient/v1/shell.py", line 137, in do_stack_create
      hc.stacks.create(**fields)
    File "/usr/lib/python2.7/dist-packages/heatclient/v1/stacks.py", line 134, in create
      data=kwargs, headers=headers)
    File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 266, in json_request
      resp = self._http_request(url, method, **kwargs)
    File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 358, in _http_request
      raise_exc=False, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/keystoneclient/utils.py", line 337, in inner
      return func(*args, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 390, in request
      resp = send(**kwargs)
    File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 434, in _send_request
      raise exceptions.ConnectionRefused(msg)
  keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://Controller-VIP:8004/v1/ca1d10786fdc41698e276a8f22d98741/stacks

  I think we should retry the API connection from keystone client
  whenever there is a failure of connection to API. When I added code to
  retry it is working fine. In the existing code Keystoneclient is
  trying only one time and returning an error if it is not able to
  connect to any other service API.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1535640/+subscriptions