yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2055295] Re: Unexpected API Error

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Guytion <2055295@xxxxxxxxxxxxxxxxxx>
Date: Tue, 05 Mar 2024 15:19:45 -0000
Reply-to: Bug 2055295 <2055295@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2055295

Title:
  Unexpected API Error

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  024-02-28 20:28:16.731 2534 WARNING oslo_config.cfg [None req-edd6752a-8653-4f5e-bd96-11fbc924f60a 3e74f7ef625942f39484fd1b9bc7ca62 6861e0fb7b274ca397f6737d026e9a70 - - default default] Deprecated: Option "api_servers" from group "glance" is deprecated for removal (
  Support for image service configuration via standard keystoneauth1 Adapter
  options was added in the 17.0.0 Queens release. The api_servers option was
  retained temporarily to allow consumers time to cut over to a real load
  balancing solution.
  ).  Its value may be silently ignored in the future.
  2024-02-28 20:28:17.235 2534 WARNING keystoneauth.identity.generic.base [None req-edd6752a-8653-4f5e-bd96-11fbc924f60a 3e74f7ef625942f39484fd1b9bc7ca62 6861e0fb7b274ca397f6737d026e9a70 - - default default] Failed to discover available identity versions when contacting https://controller/identity. Attempting to parse version from URL.: keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to https://controller/identity: HTTPSConnectionPool(host='controller', port=443): Max retries exceeded with url: /identity (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f220ebfc040>: Failed to establish a new connection: [Errno 111] ECONNREFUSED'))
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi [None req-edd6752a-8653-4f5e-bd96-11fbc924f60a 3e74f7ef625942f39484fd1b9bc7ca62 6861e0fb7b274ca397f6737d026e9a70 - - default default] Unexpected exception in API method: keystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to https://controller/identity: HTTPSConnectionPool(host='controller', port=443): Max retries exceeded with url: /identity (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f220ebfc040>: Failed to establish a new connection: [Errno 111] ECONNREFUSED'))
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 169, in _new_conn
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     conn = connection.create_connection(
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 96, in create_connection
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     raise err
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 86, in create_connection
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     sock.connect(sa)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 256, in connect
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     socket_checkerr(fd)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 54, in socket_checkerr
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     raise socket.error(err, errno.errorcode[err])
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi ConnectionRefusedError: [Errno 111] ECONNREFUSED
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi During handling of the above exception, another exception occurred:
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 700, in urlopen
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     httplib_response = self._make_request(
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 383, in _make_request
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     self._validate_conn(conn)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1017, in _validate_conn
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     conn.connect()
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 353, in connect
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     conn = self._new_conn()
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 181, in _new_conn
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     raise NewConnectionError(
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f220ebfc040>: Failed to establish a new connection: [Errno 111] ECONNREFUSED
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi During handling of the above exception, another exception occurred:
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     resp = conn.urlopen(
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 756, in urlopen
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     retries = retries.increment(
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 574, in increment
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     raise MaxRetryError(_pool, url, error or ResponseError(cause))
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='controller', port=443): Max retries exceeded with url: /identity (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f220ebfc040>: Failed to establish a new connection: [Errno 111] ECONNREFUSED'))
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi During handling of the above exception, another exception occurred:
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1014, in _send_request
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     resp = self.session.request(method, url, **kwargs)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 544, in request
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     resp = self.send(prep, **send_kwargs)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 657, in send
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     r = adapter.send(request, **kwargs)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     raise ConnectionError(e, request=request)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi requests.exceptions.ConnectionError: HTTPSConnectionPool(host='controller', port=443): Max retries exceeded with url: /identity (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f220ebfc040>: Failed to establish a new connection: [Errno 111] ECONNREFUSED'))
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi During handling of the above exception, another exception occurred:
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/keystoneauth1/identity/generic/base.py", line 133, in _do_create_plugin
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     disc = self.get_discovery(session,
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 605, in get_discovery
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     return discover.get_discovery(session=session, url=url,
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/keystoneauth1/discover.py", line 1459, in get_discovery
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     disc = Discover(session, url, authenticated=authenticated)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/keystoneauth1/discover.py", line 539, in __init__
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     self._data = get_version_data(session, url,
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/keystoneauth1/discover.py", line 106, in get_version_data
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     resp = session.get(url, headers=headers, authenticated=authenticated)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1133, in get
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     return self.request(url, 'GET', **kwargs)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 923, in request
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     resp = send(**kwargs)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1030, in _send_request
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     raise exceptions.ConnectFailure(msg)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to https://controller/identity: HTTPSConnectionPool(host='controller', port=443): Max retries exceeded with url: /identity (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f220ebfc040>: Failed to establish a new connection: [Errno 111] ECONNREFUSED'))
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi During handling of the above exception, another exception occurred:
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/nova/api/openstack/wsgi.py", line 658, in wrapped
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/nova/api/validation/__init__.py", line 110, in wrapper
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/nova/api/validation/__init__.py", line 110, in wrapper
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/nova/api/validation/__init__.py", line 110, in wrapper
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   [Previous line repeated 11 more times]
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/nova/api/openstack/compute/servers.py", line 786, in create
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     instances, resv_id = self.compute_api.create(
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/nova/compute/api.py", line 2207, in create
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     return self._create_instance(
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/nova/compute/api.py", line 1662, in _create_instance
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     image_id, boot_meta = self._get_image(context, image_href)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/nova/compute/api.py", line 1010, in _get_image
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     image = self.image_api.get(context, image_href)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/nova/image/glance.py", line 1205, in get
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     return session.show(context, image_id,
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/nova/image/glance.py", line 287, in show
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     _reraise_translated_image_exception(image_id)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/nova/image/glance.py", line 1031, in _reraise_translated_image_exception
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     raise new_exc.with_traceback(exc_trace)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/nova/image/glance.py", line 285, in show
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     image = self._client.call(context, 2, 'get', args=(image_id,))
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/nova/image/glance.py", line 191, in call
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     result = getattr(controller, method)(*args, **kwargs)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 197, in get
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     return self._get(image_id)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/glanceclient/common/utils.py", line 649, in inner
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     return RequestIdProxy(wrapped(*args, **kwargs))
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 190, in _get
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     resp, body = self.http_client.get(url, headers=header)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 395, in get
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     return self.request(url, 'GET', **kwargs)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/glanceclient/common/http.py", line 361, in request
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     resp = super(SessionClient,
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 257, in request
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     return self.session.request(url, method, **kwargs)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 782, in request
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     auth_headers = self.get_auth_headers(auth)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1193, in get_auth_headers
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     return auth.get_headers(self, **kwargs)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/keystoneauth1/service_token.py", line 30, in get_headers
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     token = self.service_auth.get_token(session, **kwargs)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 85, in get_token
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     return self.get_access(session).auth_token
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 131, in get_access
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     self.auth_ref = self.get_auth_ref(session)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/keystoneauth1/identity/generic/base.py", line 203, in get_auth_ref
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     self._plugin = self._do_create_plugin(session)
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/keystoneauth1/identity/generic/base.py", line 155, in _do_create_plugin
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi     raise exceptions.DiscoveryFailure(
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi keystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to https://controller/identity: HTTPSConnectionPool(host='controller', port=443): Max retries exceeded with url: /identity (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f220ebfc040>: Failed to establish a new connection: [Errno 111] ECONNREFUSED'))
  2024-02-28 20:28:17.244 2534 ERROR nova.api.openstack.wsgi

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2055295/+subscriptions
References

[Bug 2055295] [NEW] Unexpected API Error
From: Guytion, 2024-02-28