yahoo-eng-team team mailing list archive

[Satyanarayana Patibandla <satya.patibandla@xxxxxxxxx>]

Public bug reported:

In Openstack HA environment, sometimes  Kesystone client is throwing the
error "unable to establish connection error". This is happening once in
fifteen times. The error and stack trace of the error log is given
below.

Error: 1) keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://Controller-VIP:8004/v1/ca1d10786fdc41698e276a8f22d98741/stacks
2) ConnectionRefused: Unable to establish connection to http://Controller-VIP:9696/v2.0/ports.json?tenant_id=ca1d10786fdc41698e276a8f22d98741&device_id=183bea96-b0da-42a5-a731-713882598624

Stack trace of the error:

NFO (connectionpool) Starting new HTTP connection (1): Controller-VIP
Traceback (most recent call last):
  File "/usr/bin/heat", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/dist-packages/heatclient/shell.py", line 706, in main
    HeatShell().main(args)
  File "/usr/lib/python2.7/dist-packages/heatclient/shell.py", line 656, in main
    args.func(client, args)
  File "/usr/lib/python2.7/dist-packages/heatclient/v1/shell.py", line 137, in do_stack_create
    hc.stacks.create(**fields)
  File "/usr/lib/python2.7/dist-packages/heatclient/v1/stacks.py", line 134, in create
    data=kwargs, headers=headers)
  File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 266, in json_request
    resp = self._http_request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 358, in _http_request
    raise_exc=False, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/utils.py", line 337, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 390, in request
    resp = send(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 434, in _send_request
    raise exceptions.ConnectionRefused(msg)
keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://Controller-VIP:8004/v1/ca1d10786fdc41698e276a8f22d98741/stacks

I think we should retry the API connection from keystone client whenever
there is a failure of connection to API. When I added code to retry it
is working fine. In the existing code Keystoneclient is trying only one
time and returning an error if it is not able to connect to any other
service API.

** Affects: keystone
     Importance: Undecided
     Assignee: Satyanarayana Patibandla (satya-patibandla)
         Status: Opinion


** Tags: identity openstack

** Changed in: keystone
     Assignee: (unassigned) => Satyanarayana Patibandla (satya-patibandla)

** Changed in: keystone
       Status: New => Opinion

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1535640

Title:
  Sometimes getting unable to establish connection error in Keystone
  client

Status in OpenStack Identity (keystone):
  Opinion

Bug description:
  In Openstack HA environment, sometimes  Kesystone client is throwing
  the error "unable to establish connection error". This is happening
  once in fifteen times. The error and stack trace of the error log is
  given below.

  Error: 1) keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://Controller-VIP:8004/v1/ca1d10786fdc41698e276a8f22d98741/stacks
  2) ConnectionRefused: Unable to establish connection to http://Controller-VIP:9696/v2.0/ports.json?tenant_id=ca1d10786fdc41698e276a8f22d98741&device_id=183bea96-b0da-42a5-a731-713882598624

  Stack trace of the error:

  NFO (connectionpool) Starting new HTTP connection (1): Controller-VIP
  Traceback (most recent call last):
    File "/usr/bin/heat", line 10, in <module>
      sys.exit(main())
    File "/usr/lib/python2.7/dist-packages/heatclient/shell.py", line 706, in main
      HeatShell().main(args)
    File "/usr/lib/python2.7/dist-packages/heatclient/shell.py", line 656, in main
      args.func(client, args)
    File "/usr/lib/python2.7/dist-packages/heatclient/v1/shell.py", line 137, in do_stack_create
      hc.stacks.create(**fields)
    File "/usr/lib/python2.7/dist-packages/heatclient/v1/stacks.py", line 134, in create
      data=kwargs, headers=headers)
    File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 266, in json_request
      resp = self._http_request(url, method, **kwargs)
    File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 358, in _http_request
      raise_exc=False, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/keystoneclient/utils.py", line 337, in inner
      return func(*args, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 390, in request
      resp = send(**kwargs)
    File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 434, in _send_request
      raise exceptions.ConnectionRefused(msg)
  keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://Controller-VIP:8004/v1/ca1d10786fdc41698e276a8f22d98741/stacks

  I think we should retry the API connection from keystone client
  whenever there is a failure of connection to API. When I added code to
  retry it is working fine. In the existing code Keystoneclient is
  trying only one time and returning an error if it is not able to
  connect to any other service API.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1535640/+subscriptions