yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1891973] Re: Getting HTTP 500 exception frequently in Openstack Ussuri

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Launchpad Bug Tracker <1891973@xxxxxxxxxxxxxxxxxx>
Date: Tue, 27 Oct 2020 04:17:21 -0000
Reply-to: Bug 1891973 <1891973@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
[Expired for OpenStack Compute (nova) because there has been no activity
for 60 days.]

** Changed in: nova
       Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1891973

Title:
  Getting HTTP 500 exception frequently in Openstack Ussuri

Status in OpenStack Compute (nova):
  Expired

Bug description:
  I am using openstack ussuri with Centos 8, installed using RDO packstack.
  I frequently getting http 500 EXCEPTION ERROR when ever am doing some operation, i need to restart httpd service inorder to overcome this issue.

  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-8868aed7-265d-4e8e-aa37-d427b1b561cb)

  : nova.exception.ResourceProviderAllocationRetrievalFailed: Failed to retrieve allocations for resource provider 3a80ea71-91a6-4da7-a05b-682b501fe6c9: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  2020-08-17 22:53:46.574 63567 ERROR nova.scheduler.client.report [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  <html><head>
  <title>500 Internal Server Error</title>
  </head><body>
  <h1>Internal Server Error</h1>
  <p>The server encountered an internal error or
  misconfiguration and was unable to complete
  your request.</p>
  <p>Please contact the server administrator at
   [no address given] to inform them of the time this error occurred,
   and the actions you performed just before this error.</p>
  <p>More information about this error may be available
  in the server error log.</p>
  </body></html>
  .
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Error updating resources for node openstack-Karty.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager Traceback (most recent call last):
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     startup=startup)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     self._update_available_resource(context, resources, startup=startup)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     return f(*args, **kwargs)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     self._update(context, cn, startup=startup)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     self._update_to_placement(context, compute_node, startup)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     return Retrying(*dargs, **dkw).call(f, *args, **kw)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     return attempt.get(self._wrap_exception)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     six.reraise(self.value[0], self.value[1], self.value[2])
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     raise value
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     context, compute_node.uuid, name=compute_node.hypervisor_hostname)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     rps_to_refresh = self.get_providers_in_tree(context, uuid)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 503, in get_providers_in_tree
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     raise exception.ResourceProviderRetrievalFailed(uuid=uuid)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager
  2020-08-17 22:54:46.284 63567 ERROR nova.scheduler.client.report [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  <html><head>
  <title>500 Internal Server Error</title>
  </head><body>
  <h1>Internal Server Error</h1>
  <p>The server encountered an internal error or
  misconfiguration and was unable to complete
  your request.</p>
  <p>Please contact the server administrator at
   [no address given] to inform them of the time this error occurred,
   and the actions you performed just before this error.</p>
  <p>More information about this error may be available
  in the server error log.</p>
  </body></html>
  .
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Error updating resources for node openstack-Karty.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager Traceback (most recent call last):
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     startup=startup)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     self._update_available_resource(context, resources, startup=startup)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     return f(*args, **kwargs)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     self._update(context, cn, startup=startup)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     self._update_to_placement(context, compute_node, startup)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     return Retrying(*dargs, **dkw).call(f, *args, **kw)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     return attempt.get(self._wrap_exception)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     six.reraise(self.value[0], self.value[1], self.value[2])
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     raise value
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     context, compute_node.uuid, name=compute_node.hypervisor_hostname)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     rps_to_refresh = self.get_providers_in_tree(context, uuid)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 503, in get_providers_in_tree
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     raise exception.ResourceProviderRetrievalFailed(uuid=uuid)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager

  
  root@openstack-Karty nova(keystone_admin)]# openstack server list
  Unknown Error (HTTP 500)
  [root@openstack-Karty nova(keystone_admin)]# openstack server list
  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'TypeError'> (HTTP 500) (Request-ID: req-7d3a09b6-2e6a-43b5-8b4b-03c1ac3d37dd)
  [root@openstack-Karty nova(keystone_admin)]# tail -f -n 100 nova-compute.log 
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     self, "Failed to establish a new connection: %s" % e
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager During handling of the above exception, another exception occurred:
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager Traceback (most recent call last):
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     timeout=timeout
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     raise MaxRetryError(_pool, url, error or ResponseError(cause))
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='10.24.0.117', port=8778): Max retries exceeded with url: /placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager During handling of the above exception, another exception occurred:
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager Traceback (most recent call last):
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1004, in _send_request
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     resp = self.session.request(method, url, **kwargs)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     resp = self.send(prep, **send_kwargs)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     r = adapter.send(request, **kwargs)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     raise ConnectionError(e, request=request)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager requests.exceptions.ConnectionError: HTTPConnectionPool(host='10.24.0.117', port=8778): Max retries exceeded with url: /placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager During handling of the above exception, another exception occurred:
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager Traceback (most recent call last):
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     startup=startup)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     self._update_available_resource(context, resources, startup=startup)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     return f(*args, **kwargs)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     self._update(context, cn, startup=startup)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     self._update_to_placement(context, compute_node, startup)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     return Retrying(*dargs, **dkw).call(f, *args, **kw)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     return attempt.get(self._wrap_exception)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     six.reraise(self.value[0], self.value[1], self.value[2])
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     raise value
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     context, compute_node.uuid, name=compute_node.hypervisor_hostname)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     rps_to_refresh = self.get_providers_in_tree(context, uuid)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 486, in get_providers_in_tree
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     global_request_id=context.global_id)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 237, in get
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     global_request_id=global_request_id)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 386, in get
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     return self.request(url, 'GET', **kwargs)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/openstack/proxy.py", line 98, in request
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     **kwargs)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 248, in request
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     return self.session.request(url, method, **kwargs)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 913, in request
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     resp = send(**kwargs)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1043, in _send_request
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     **kwargs)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1020, in _send_request
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     raise exceptions.ConnectFailure(msg)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://10.24.0.117:8778/placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9: HTTPConnectionPool(host='10.24.0.117', port=8778): Max retries exceeded with url: /placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
  2020-08-17 22:59:56.498 63567 WARNING nova.virt.libvirt.imagecache [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Unknown base file: /var/lib/nova/instances/_base/5a8a3c88277cdf03ea71a4448af35b8e684f7943
  2020-08-17 22:59:56.498 63567 INFO nova.virt.libvirt.imagecache [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Removable base files: /var/lib/nova/instances/_base/5a8a3c88277cdf03ea71a4448af35b8e684f7943
  2020-08-17 22:59:56.499 63567 INFO nova.virt.libvirt.imagecache [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Base or swap file too young to remove: /var/lib/nova/instances/_base/5a8a3c88277cdf03ea71a4448af35b8e684f7943
  2020-08-17 23:00:05.503 63567 INFO nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Running instance usage audit for host openstack-Karty from 2020-08-18 05:00:00 to 2020-08-18 06:00:00. 0 instances.
  2020-08-17 23:02:10.221 63567 INFO nova.compute.claims [req-1de4c903-89cf-4494-99d5-3e6237cd67bf 108551f604a24c23989a699b058e584b 9d3e3eadbc9c4d5fa6a3b27bfee40734 - default default] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] Claim successful on node openstack-Karty
  2020-08-17 23:02:10.621 63567 INFO nova.virt.libvirt.driver [req-1de4c903-89cf-4494-99d5-3e6237cd67bf 108551f604a24c23989a699b058e584b 9d3e3eadbc9c4d5fa6a3b27bfee40734 - default default] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] Creating image
  2020-08-17 23:02:14.710 63567 INFO os_vif [req-1de4c903-89cf-4494-99d5-3e6237cd67bf 108551f604a24c23989a699b058e584b 9d3e3eadbc9c4d5fa6a3b27bfee40734 - default default] Successfully plugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:15:cb:de,bridge_name='br-int',has_traffic_filtering=True,id=bb31ebbf-b1f4-48bd-9412-5b1a8e60fecd,network=Network(ca332b6f-a3c3-4c96-bc27-5a2243929fb9),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapbb31ebbf-b1')
  2020-08-17 23:02:15.093 63567 INFO nova.compute.manager [req-b9522c87-a4bb-42ab-a9a9-a60d987e60a8 - - - - -] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] VM Started (Lifecycle Event)
  2020-08-17 23:02:15.131 63567 INFO nova.compute.manager [req-b9522c87-a4bb-42ab-a9a9-a60d987e60a8 - - - - -] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] VM Paused (Lifecycle Event)
  2020-08-17 23:02:15.199 63567 INFO nova.compute.manager [req-b9522c87-a4bb-42ab-a9a9-a60d987e60a8 - - - - -] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] During sync_power_state the instance has a pending task (spawning). Skip.
  2020-08-17 23:02:51.316 63567 ERROR nova.compute.resource_tracker [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Skipping removal of allocations for deleted instances: Failed to retrieve allocations for resource provider 3a80ea71-91a6-4da7-a05b-682b501fe6c9: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  <html><head>
  <title>500 Internal Server Error</title>
  </head><body>
  <h1>Internal Server Error</h1>
  <p>The server encountered an internal error or
  misconfiguration and was unable to complete
  your request.</p>
  <p>Please contact the server administrator at 
   [no address given] to inform them of the time this error occurred,
   and the actions you performed just before this error.</p>
  <p>More information about this error may be available
  in the server error log.</p>
  </body></html>
  : nova.exception.ResourceProviderAllocationRetrievalFailed: Failed to retrieve allocations for resource provider 3a80ea71-91a6-4da7-a05b-682b501fe6c9: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  2020-08-17 23:03:53.500 63567 ERROR nova.scheduler.client.report [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  <html><head>
  <title>500 Internal Server Error</title>
  </head><body>
  <h1>Internal Server Error</h1>
  <p>The server encountered an internal error or
  misconfiguration and was unable to complete
  your request.</p>
  <p>Please contact the server administrator at 
   [no address given] to inform them of the time this error occurred,
   and the actions you performed just before this error.</p>
  <p>More information about this error may be available
  in the server error log.</p>
  </body></html>
  .
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Error updating resources for node openstack-Karty.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager Traceback (most recent call last):
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     startup=startup)
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     self._update_available_resource(context, resources, startup=startup)
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     return f(*args, **kwargs)
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     self._update(context, cn, startup=startup)
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     self._update_to_placement(context, compute_node, startup)
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     return Retrying(*dargs, **dkw).call(f, *args, **kw)
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     return attempt.get(self._wrap_exception)
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     six.reraise(self.value[0], self.value[1], self.value[2])
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     raise value
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     context, compute_node.uuid, name=compute_node.hypervisor_hostname)
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     rps_to_refresh = self.get_providers_in_tree(context, uuid)
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 503, in get_providers_in_tree
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     raise exception.ResourceProviderRetrievalFailed(uuid=uuid)
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager 


  Gateway Timeout
  The gateway did not receive a timely response from the upstream server or application.

  I need to keep restarting service httpd restart inorder to overcome
  this issue. Please let us know a permanent solution. Thanks.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1891973/+subscriptions
References

[Bug 1891973] [NEW] Getting HTTP 500 exception frequently in Openstack Ussuri
From: karthik, 2020-08-18