yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1891973] [NEW] Getting HTTP 500 exception frequently in Openstack Ussuri

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: karthik <1891973@xxxxxxxxxxxxxxxxxx>
Date: Tue, 18 Aug 2020 05:59:12 -0000
Reply-to: Bug 1891973 <1891973@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I am using openstack ussuri with Centos 8, installed using RDO packstack.
I frequently getting http 500 EXCEPTION ERROR when ever am doing some operation, i need to restart httpd service inorder to overcome this issue.

Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-8868aed7-265d-4e8e-aa37-d427b1b561cb)

: nova.exception.ResourceProviderAllocationRetrievalFailed: Failed to retrieve allocations for resource provider 3a80ea71-91a6-4da7-a05b-682b501fe6c9: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
2020-08-17 22:53:46.574 63567 ERROR nova.scheduler.client.report [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
 [no address given] to inform them of the time this error occurred,
 and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>
.
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Error updating resources for node openstack-Karty.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager Traceback (most recent call last):
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     startup=startup)
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     self._update_available_resource(context, resources, startup=startup)
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     return f(*args, **kwargs)
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     self._update(context, cn, startup=startup)
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     self._update_to_placement(context, compute_node, startup)
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     return Retrying(*dargs, **dkw).call(f, *args, **kw)
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     return attempt.get(self._wrap_exception)
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     six.reraise(self.value[0], self.value[1], self.value[2])
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     raise value
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     context, compute_node.uuid, name=compute_node.hypervisor_hostname)
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     rps_to_refresh = self.get_providers_in_tree(context, uuid)
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 503, in get_providers_in_tree
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     raise exception.ResourceProviderRetrievalFailed(uuid=uuid)
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager
2020-08-17 22:54:46.284 63567 ERROR nova.scheduler.client.report [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
 [no address given] to inform them of the time this error occurred,
 and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>
.
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Error updating resources for node openstack-Karty.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager Traceback (most recent call last):
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     startup=startup)
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     self._update_available_resource(context, resources, startup=startup)
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     return f(*args, **kwargs)
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     self._update(context, cn, startup=startup)
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     self._update_to_placement(context, compute_node, startup)
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     return Retrying(*dargs, **dkw).call(f, *args, **kw)
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     return attempt.get(self._wrap_exception)
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     six.reraise(self.value[0], self.value[1], self.value[2])
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     raise value
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     context, compute_node.uuid, name=compute_node.hypervisor_hostname)
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     rps_to_refresh = self.get_providers_in_tree(context, uuid)
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 503, in get_providers_in_tree
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     raise exception.ResourceProviderRetrievalFailed(uuid=uuid)
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager


root@openstack-Karty nova(keystone_admin)]# openstack server list
Unknown Error (HTTP 500)
[root@openstack-Karty nova(keystone_admin)]# openstack server list
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'TypeError'> (HTTP 500) (Request-ID: req-7d3a09b6-2e6a-43b5-8b4b-03c1ac3d37dd)
[root@openstack-Karty nova(keystone_admin)]# tail -f -n 100 nova-compute.log 
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     self, "Failed to establish a new connection: %s" % e
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager During handling of the above exception, another exception occurred:
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager Traceback (most recent call last):
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     timeout=timeout
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     raise MaxRetryError(_pool, url, error or ResponseError(cause))
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='10.24.0.117', port=8778): Max retries exceeded with url: /placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager During handling of the above exception, another exception occurred:
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager Traceback (most recent call last):
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1004, in _send_request
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     resp = self.session.request(method, url, **kwargs)
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     resp = self.send(prep, **send_kwargs)
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     r = adapter.send(request, **kwargs)
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     raise ConnectionError(e, request=request)
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager requests.exceptions.ConnectionError: HTTPConnectionPool(host='10.24.0.117', port=8778): Max retries exceeded with url: /placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager During handling of the above exception, another exception occurred:
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager Traceback (most recent call last):
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     startup=startup)
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     self._update_available_resource(context, resources, startup=startup)
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     return f(*args, **kwargs)
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     self._update(context, cn, startup=startup)
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     self._update_to_placement(context, compute_node, startup)
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     return Retrying(*dargs, **dkw).call(f, *args, **kw)
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     return attempt.get(self._wrap_exception)
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     six.reraise(self.value[0], self.value[1], self.value[2])
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     raise value
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     context, compute_node.uuid, name=compute_node.hypervisor_hostname)
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     rps_to_refresh = self.get_providers_in_tree(context, uuid)
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 486, in get_providers_in_tree
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     global_request_id=context.global_id)
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 237, in get
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     global_request_id=global_request_id)
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 386, in get
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     return self.request(url, 'GET', **kwargs)
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/openstack/proxy.py", line 98, in request
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     **kwargs)
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 248, in request
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     return self.session.request(url, method, **kwargs)
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 913, in request
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     resp = send(**kwargs)
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1043, in _send_request
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     **kwargs)
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1020, in _send_request
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     raise exceptions.ConnectFailure(msg)
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://10.24.0.117:8778/placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9: HTTPConnectionPool(host='10.24.0.117', port=8778): Max retries exceeded with url: /placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
2020-08-17 22:59:56.498 63567 WARNING nova.virt.libvirt.imagecache [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Unknown base file: /var/lib/nova/instances/_base/5a8a3c88277cdf03ea71a4448af35b8e684f7943
2020-08-17 22:59:56.498 63567 INFO nova.virt.libvirt.imagecache [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Removable base files: /var/lib/nova/instances/_base/5a8a3c88277cdf03ea71a4448af35b8e684f7943
2020-08-17 22:59:56.499 63567 INFO nova.virt.libvirt.imagecache [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Base or swap file too young to remove: /var/lib/nova/instances/_base/5a8a3c88277cdf03ea71a4448af35b8e684f7943
2020-08-17 23:00:05.503 63567 INFO nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Running instance usage audit for host openstack-Karty from 2020-08-18 05:00:00 to 2020-08-18 06:00:00. 0 instances.
2020-08-17 23:02:10.221 63567 INFO nova.compute.claims [req-1de4c903-89cf-4494-99d5-3e6237cd67bf 108551f604a24c23989a699b058e584b 9d3e3eadbc9c4d5fa6a3b27bfee40734 - default default] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] Claim successful on node openstack-Karty
2020-08-17 23:02:10.621 63567 INFO nova.virt.libvirt.driver [req-1de4c903-89cf-4494-99d5-3e6237cd67bf 108551f604a24c23989a699b058e584b 9d3e3eadbc9c4d5fa6a3b27bfee40734 - default default] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] Creating image
2020-08-17 23:02:14.710 63567 INFO os_vif [req-1de4c903-89cf-4494-99d5-3e6237cd67bf 108551f604a24c23989a699b058e584b 9d3e3eadbc9c4d5fa6a3b27bfee40734 - default default] Successfully plugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:15:cb:de,bridge_name='br-int',has_traffic_filtering=True,id=bb31ebbf-b1f4-48bd-9412-5b1a8e60fecd,network=Network(ca332b6f-a3c3-4c96-bc27-5a2243929fb9),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapbb31ebbf-b1')
2020-08-17 23:02:15.093 63567 INFO nova.compute.manager [req-b9522c87-a4bb-42ab-a9a9-a60d987e60a8 - - - - -] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] VM Started (Lifecycle Event)
2020-08-17 23:02:15.131 63567 INFO nova.compute.manager [req-b9522c87-a4bb-42ab-a9a9-a60d987e60a8 - - - - -] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] VM Paused (Lifecycle Event)
2020-08-17 23:02:15.199 63567 INFO nova.compute.manager [req-b9522c87-a4bb-42ab-a9a9-a60d987e60a8 - - - - -] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] During sync_power_state the instance has a pending task (spawning). Skip.
2020-08-17 23:02:51.316 63567 ERROR nova.compute.resource_tracker [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Skipping removal of allocations for deleted instances: Failed to retrieve allocations for resource provider 3a80ea71-91a6-4da7-a05b-682b501fe6c9: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at 
 [no address given] to inform them of the time this error occurred,
 and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>
: nova.exception.ResourceProviderAllocationRetrievalFailed: Failed to retrieve allocations for resource provider 3a80ea71-91a6-4da7-a05b-682b501fe6c9: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
2020-08-17 23:03:53.500 63567 ERROR nova.scheduler.client.report [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at 
 [no address given] to inform them of the time this error occurred,
 and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>
.
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Error updating resources for node openstack-Karty.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager Traceback (most recent call last):
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     startup=startup)
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     self._update_available_resource(context, resources, startup=startup)
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     return f(*args, **kwargs)
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     self._update(context, cn, startup=startup)
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     self._update_to_placement(context, compute_node, startup)
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     return Retrying(*dargs, **dkw).call(f, *args, **kw)
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     return attempt.get(self._wrap_exception)
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     six.reraise(self.value[0], self.value[1], self.value[2])
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     raise value
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     context, compute_node.uuid, name=compute_node.hypervisor_hostname)
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     rps_to_refresh = self.get_providers_in_tree(context, uuid)
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 503, in get_providers_in_tree
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     raise exception.ResourceProviderRetrievalFailed(uuid=uuid)
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager 


Gateway Timeout
The gateway did not receive a timely response from the upstream server or application.

I need to keep restarting service httpd restart inorder to overcome this
issue. Please let us know a permanent solution. Thanks.

** Affects: nova
     Importance: Undecided
         Status: New

** Summary changed:

- Getting HTTP 500 frequently in Openstack Ussuri
+ Getting HTTP 500 exception frequently in Openstack Ussuri

** Description changed:

- I am using openstack ussuri with Centos 8, installed using RDO
- packstack.
+ I am using openstack ussuri with Centos 8, installed using RDO packstack.
+ I frequently getting http 500 EXCEPTION ERROR when ever am doing some operation, i need to restart httpd service inorder to overcome this issue. 
  
  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-8868aed7-265d-4e8e-aa37-d427b1b561cb)
- 
  
  : nova.exception.ResourceProviderAllocationRetrievalFailed: Failed to retrieve allocations for resource provider 3a80ea71-91a6-4da7-a05b-682b501fe6c9: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  2020-08-17 22:53:46.574 63567 ERROR nova.scheduler.client.report [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  <html><head>
  <title>500 Internal Server Error</title>
  </head><body>
  <h1>Internal Server Error</h1>
  <p>The server encountered an internal error or
  misconfiguration and was unable to complete
  your request.</p>
- <p>Please contact the server administrator at 
-  [no address given] to inform them of the time this error occurred,
-  and the actions you performed just before this error.</p>
+ <p>Please contact the server administrator at
+  [no address given] to inform them of the time this error occurred,
+  and the actions you performed just before this error.</p>
  <p>More information about this error may be available
  in the server error log.</p>
  </body></html>
  .
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Error updating resources for node openstack-Karty.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager Traceback (most recent call last):
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     startup=startup)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     self._update_available_resource(context, resources, startup=startup)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     return f(*args, **kwargs)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     self._update(context, cn, startup=startup)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     self._update_to_placement(context, compute_node, startup)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     return Retrying(*dargs, **dkw).call(f, *args, **kw)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     return attempt.get(self._wrap_exception)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     six.reraise(self.value[0], self.value[1], self.value[2])
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     raise value
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     context, compute_node.uuid, name=compute_node.hypervisor_hostname)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     rps_to_refresh = self.get_providers_in_tree(context, uuid)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 503, in get_providers_in_tree
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     raise exception.ResourceProviderRetrievalFailed(uuid=uuid)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
- 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager 
+ 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager
  2020-08-17 22:54:46.284 63567 ERROR nova.scheduler.client.report [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  <html><head>
  <title>500 Internal Server Error</title>
  </head><body>
  <h1>Internal Server Error</h1>
  <p>The server encountered an internal error or
  misconfiguration and was unable to complete
  your request.</p>
- <p>Please contact the server administrator at 
-  [no address given] to inform them of the time this error occurred,
-  and the actions you performed just before this error.</p>
+ <p>Please contact the server administrator at
+  [no address given] to inform them of the time this error occurred,
+  and the actions you performed just before this error.</p>
  <p>More information about this error may be available
  in the server error log.</p>
  </body></html>
  .
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Error updating resources for node openstack-Karty.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager Traceback (most recent call last):
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     startup=startup)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     self._update_available_resource(context, resources, startup=startup)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     return f(*args, **kwargs)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     self._update(context, cn, startup=startup)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     self._update_to_placement(context, compute_node, startup)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     return Retrying(*dargs, **dkw).call(f, *args, **kw)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     return attempt.get(self._wrap_exception)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     six.reraise(self.value[0], self.value[1], self.value[2])
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     raise value
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     context, compute_node.uuid, name=compute_node.hypervisor_hostname)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     rps_to_refresh = self.get_providers_in_tree(context, uuid)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 503, in get_providers_in_tree
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     raise exception.ResourceProviderRetrievalFailed(uuid=uuid)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
- 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager 
+ 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager
  
  Gateway Timeout
  The gateway did not receive a timely response from the upstream server or application.
  
- 
- I need to keep restarting service httpd restart inorder to overcome this issue. Please let us know a permanent solution. Thanks.
+ I need to keep restarting service httpd restart inorder to overcome this
+ issue. Please let us know a permanent solution. Thanks.

** Description changed:

  I am using openstack ussuri with Centos 8, installed using RDO packstack.
- I frequently getting http 500 EXCEPTION ERROR when ever am doing some operation, i need to restart httpd service inorder to overcome this issue. 
+ I frequently getting http 500 EXCEPTION ERROR when ever am doing some operation, i need to restart httpd service inorder to overcome this issue.
  
  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-8868aed7-265d-4e8e-aa37-d427b1b561cb)
  
  : nova.exception.ResourceProviderAllocationRetrievalFailed: Failed to retrieve allocations for resource provider 3a80ea71-91a6-4da7-a05b-682b501fe6c9: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  2020-08-17 22:53:46.574 63567 ERROR nova.scheduler.client.report [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  <html><head>
  <title>500 Internal Server Error</title>
  </head><body>
  <h1>Internal Server Error</h1>
  <p>The server encountered an internal error or
  misconfiguration and was unable to complete
  your request.</p>
  <p>Please contact the server administrator at
   [no address given] to inform them of the time this error occurred,
   and the actions you performed just before this error.</p>
  <p>More information about this error may be available
  in the server error log.</p>
  </body></html>
  .
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Error updating resources for node openstack-Karty.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager Traceback (most recent call last):
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     startup=startup)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     self._update_available_resource(context, resources, startup=startup)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     return f(*args, **kwargs)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     self._update(context, cn, startup=startup)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     self._update_to_placement(context, compute_node, startup)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     return Retrying(*dargs, **dkw).call(f, *args, **kw)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     return attempt.get(self._wrap_exception)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     six.reraise(self.value[0], self.value[1], self.value[2])
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     raise value
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     context, compute_node.uuid, name=compute_node.hypervisor_hostname)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     rps_to_refresh = self.get_providers_in_tree(context, uuid)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 503, in get_providers_in_tree
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     raise exception.ResourceProviderRetrievalFailed(uuid=uuid)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager
  2020-08-17 22:54:46.284 63567 ERROR nova.scheduler.client.report [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  <html><head>
  <title>500 Internal Server Error</title>
  </head><body>
  <h1>Internal Server Error</h1>
  <p>The server encountered an internal error or
  misconfiguration and was unable to complete
  your request.</p>
  <p>Please contact the server administrator at
   [no address given] to inform them of the time this error occurred,
   and the actions you performed just before this error.</p>
  <p>More information about this error may be available
  in the server error log.</p>
  </body></html>
  .
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Error updating resources for node openstack-Karty.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager Traceback (most recent call last):
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     startup=startup)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     self._update_available_resource(context, resources, startup=startup)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     return f(*args, **kwargs)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     self._update(context, cn, startup=startup)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     self._update_to_placement(context, compute_node, startup)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     return Retrying(*dargs, **dkw).call(f, *args, **kw)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     return attempt.get(self._wrap_exception)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     six.reraise(self.value[0], self.value[1], self.value[2])
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     raise value
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     context, compute_node.uuid, name=compute_node.hypervisor_hostname)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     rps_to_refresh = self.get_providers_in_tree(context, uuid)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 503, in get_providers_in_tree
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     raise exception.ResourceProviderRetrievalFailed(uuid=uuid)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager
  
+ 
+ root@openstack-Karty nova(keystone_admin)]# openstack server list
+ Unknown Error (HTTP 500)
+ [root@openstack-Karty nova(keystone_admin)]# openstack server list
+ Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
+ <class 'TypeError'> (HTTP 500) (Request-ID: req-7d3a09b6-2e6a-43b5-8b4b-03c1ac3d37dd)
+ [root@openstack-Karty nova(keystone_admin)]# tail -f -n 100 nova-compute.log 
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     self, "Failed to establish a new connection: %s" % e
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager During handling of the above exception, another exception occurred:
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager Traceback (most recent call last):
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     timeout=timeout
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     raise MaxRetryError(_pool, url, error or ResponseError(cause))
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='10.24.0.117', port=8778): Max retries exceeded with url: /placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager During handling of the above exception, another exception occurred:
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager Traceback (most recent call last):
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1004, in _send_request
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     resp = self.session.request(method, url, **kwargs)
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     resp = self.send(prep, **send_kwargs)
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     r = adapter.send(request, **kwargs)
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     raise ConnectionError(e, request=request)
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager requests.exceptions.ConnectionError: HTTPConnectionPool(host='10.24.0.117', port=8778): Max retries exceeded with url: /placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager During handling of the above exception, another exception occurred:
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager Traceback (most recent call last):
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     startup=startup)
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     self._update_available_resource(context, resources, startup=startup)
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     return f(*args, **kwargs)
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     self._update(context, cn, startup=startup)
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     self._update_to_placement(context, compute_node, startup)
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     return Retrying(*dargs, **dkw).call(f, *args, **kw)
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     return attempt.get(self._wrap_exception)
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     six.reraise(self.value[0], self.value[1], self.value[2])
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     raise value
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     context, compute_node.uuid, name=compute_node.hypervisor_hostname)
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     rps_to_refresh = self.get_providers_in_tree(context, uuid)
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 486, in get_providers_in_tree
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     global_request_id=context.global_id)
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 237, in get
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     global_request_id=global_request_id)
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 386, in get
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     return self.request(url, 'GET', **kwargs)
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/openstack/proxy.py", line 98, in request
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     **kwargs)
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 248, in request
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     return self.session.request(url, method, **kwargs)
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 913, in request
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     resp = send(**kwargs)
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1043, in _send_request
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     **kwargs)
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1020, in _send_request
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     raise exceptions.ConnectFailure(msg)
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://10.24.0.117:8778/placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9: HTTPConnectionPool(host='10.24.0.117', port=8778): Max retries exceeded with url: /placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
+ 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
+ 2020-08-17 22:59:56.498 63567 WARNING nova.virt.libvirt.imagecache [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Unknown base file: /var/lib/nova/instances/_base/5a8a3c88277cdf03ea71a4448af35b8e684f7943
+ 2020-08-17 22:59:56.498 63567 INFO nova.virt.libvirt.imagecache [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Removable base files: /var/lib/nova/instances/_base/5a8a3c88277cdf03ea71a4448af35b8e684f7943
+ 2020-08-17 22:59:56.499 63567 INFO nova.virt.libvirt.imagecache [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Base or swap file too young to remove: /var/lib/nova/instances/_base/5a8a3c88277cdf03ea71a4448af35b8e684f7943
+ 2020-08-17 23:00:05.503 63567 INFO nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Running instance usage audit for host openstack-Karty from 2020-08-18 05:00:00 to 2020-08-18 06:00:00. 0 instances.
+ 2020-08-17 23:02:10.221 63567 INFO nova.compute.claims [req-1de4c903-89cf-4494-99d5-3e6237cd67bf 108551f604a24c23989a699b058e584b 9d3e3eadbc9c4d5fa6a3b27bfee40734 - default default] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] Claim successful on node openstack-Karty
+ 2020-08-17 23:02:10.621 63567 INFO nova.virt.libvirt.driver [req-1de4c903-89cf-4494-99d5-3e6237cd67bf 108551f604a24c23989a699b058e584b 9d3e3eadbc9c4d5fa6a3b27bfee40734 - default default] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] Creating image
+ 2020-08-17 23:02:14.710 63567 INFO os_vif [req-1de4c903-89cf-4494-99d5-3e6237cd67bf 108551f604a24c23989a699b058e584b 9d3e3eadbc9c4d5fa6a3b27bfee40734 - default default] Successfully plugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:15:cb:de,bridge_name='br-int',has_traffic_filtering=True,id=bb31ebbf-b1f4-48bd-9412-5b1a8e60fecd,network=Network(ca332b6f-a3c3-4c96-bc27-5a2243929fb9),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapbb31ebbf-b1')
+ 2020-08-17 23:02:15.093 63567 INFO nova.compute.manager [req-b9522c87-a4bb-42ab-a9a9-a60d987e60a8 - - - - -] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] VM Started (Lifecycle Event)
+ 2020-08-17 23:02:15.131 63567 INFO nova.compute.manager [req-b9522c87-a4bb-42ab-a9a9-a60d987e60a8 - - - - -] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] VM Paused (Lifecycle Event)
+ 2020-08-17 23:02:15.199 63567 INFO nova.compute.manager [req-b9522c87-a4bb-42ab-a9a9-a60d987e60a8 - - - - -] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] During sync_power_state the instance has a pending task (spawning). Skip.
+ 2020-08-17 23:02:51.316 63567 ERROR nova.compute.resource_tracker [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Skipping removal of allocations for deleted instances: Failed to retrieve allocations for resource provider 3a80ea71-91a6-4da7-a05b-682b501fe6c9: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
+ <html><head>
+ <title>500 Internal Server Error</title>
+ </head><body>
+ <h1>Internal Server Error</h1>
+ <p>The server encountered an internal error or
+ misconfiguration and was unable to complete
+ your request.</p>
+ <p>Please contact the server administrator at 
+  [no address given] to inform them of the time this error occurred,
+  and the actions you performed just before this error.</p>
+ <p>More information about this error may be available
+ in the server error log.</p>
+ </body></html>
+ : nova.exception.ResourceProviderAllocationRetrievalFailed: Failed to retrieve allocations for resource provider 3a80ea71-91a6-4da7-a05b-682b501fe6c9: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
+ 2020-08-17 23:03:53.500 63567 ERROR nova.scheduler.client.report [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
+ <html><head>
+ <title>500 Internal Server Error</title>
+ </head><body>
+ <h1>Internal Server Error</h1>
+ <p>The server encountered an internal error or
+ misconfiguration and was unable to complete
+ your request.</p>
+ <p>Please contact the server administrator at 
+  [no address given] to inform them of the time this error occurred,
+  and the actions you performed just before this error.</p>
+ <p>More information about this error may be available
+ in the server error log.</p>
+ </body></html>
+ .
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Error updating resources for node openstack-Karty.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager Traceback (most recent call last):
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     startup=startup)
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     self._update_available_resource(context, resources, startup=startup)
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     return f(*args, **kwargs)
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     self._update(context, cn, startup=startup)
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     self._update_to_placement(context, compute_node, startup)
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     return Retrying(*dargs, **dkw).call(f, *args, **kw)
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     return attempt.get(self._wrap_exception)
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     six.reraise(self.value[0], self.value[1], self.value[2])
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     raise value
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     context, compute_node.uuid, name=compute_node.hypervisor_hostname)
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     rps_to_refresh = self.get_providers_in_tree(context, uuid)
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 503, in get_providers_in_tree
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     raise exception.ResourceProviderRetrievalFailed(uuid=uuid)
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
+ 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager 
+ 
+ 
  Gateway Timeout
  The gateway did not receive a timely response from the upstream server or application.
  
  I need to keep restarting service httpd restart inorder to overcome this
  issue. Please let us know a permanent solution. Thanks.

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1891973

Title:
  Getting HTTP 500 exception frequently in Openstack Ussuri

Status in OpenStack Compute (nova):
  New

Bug description:
  I am using openstack ussuri with Centos 8, installed using RDO packstack.
  I frequently getting http 500 EXCEPTION ERROR when ever am doing some operation, i need to restart httpd service inorder to overcome this issue.

  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-8868aed7-265d-4e8e-aa37-d427b1b561cb)

  : nova.exception.ResourceProviderAllocationRetrievalFailed: Failed to retrieve allocations for resource provider 3a80ea71-91a6-4da7-a05b-682b501fe6c9: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  2020-08-17 22:53:46.574 63567 ERROR nova.scheduler.client.report [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  <html><head>
  <title>500 Internal Server Error</title>
  </head><body>
  <h1>Internal Server Error</h1>
  <p>The server encountered an internal error or
  misconfiguration and was unable to complete
  your request.</p>
  <p>Please contact the server administrator at
   [no address given] to inform them of the time this error occurred,
   and the actions you performed just before this error.</p>
  <p>More information about this error may be available
  in the server error log.</p>
  </body></html>
  .
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Error updating resources for node openstack-Karty.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager Traceback (most recent call last):
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     startup=startup)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     self._update_available_resource(context, resources, startup=startup)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     return f(*args, **kwargs)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     self._update(context, cn, startup=startup)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     self._update_to_placement(context, compute_node, startup)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     return Retrying(*dargs, **dkw).call(f, *args, **kw)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     return attempt.get(self._wrap_exception)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     six.reraise(self.value[0], self.value[1], self.value[2])
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     raise value
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     context, compute_node.uuid, name=compute_node.hypervisor_hostname)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     rps_to_refresh = self.get_providers_in_tree(context, uuid)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 503, in get_providers_in_tree
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager     raise exception.ResourceProviderRetrievalFailed(uuid=uuid)
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
  2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager
  2020-08-17 22:54:46.284 63567 ERROR nova.scheduler.client.report [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  <html><head>
  <title>500 Internal Server Error</title>
  </head><body>
  <h1>Internal Server Error</h1>
  <p>The server encountered an internal error or
  misconfiguration and was unable to complete
  your request.</p>
  <p>Please contact the server administrator at
   [no address given] to inform them of the time this error occurred,
   and the actions you performed just before this error.</p>
  <p>More information about this error may be available
  in the server error log.</p>
  </body></html>
  .
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Error updating resources for node openstack-Karty.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager Traceback (most recent call last):
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     startup=startup)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     self._update_available_resource(context, resources, startup=startup)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     return f(*args, **kwargs)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     self._update(context, cn, startup=startup)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     self._update_to_placement(context, compute_node, startup)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     return Retrying(*dargs, **dkw).call(f, *args, **kw)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     return attempt.get(self._wrap_exception)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     six.reraise(self.value[0], self.value[1], self.value[2])
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     raise value
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     context, compute_node.uuid, name=compute_node.hypervisor_hostname)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     rps_to_refresh = self.get_providers_in_tree(context, uuid)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 503, in get_providers_in_tree
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager     raise exception.ResourceProviderRetrievalFailed(uuid=uuid)
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
  2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager

  
  root@openstack-Karty nova(keystone_admin)]# openstack server list
  Unknown Error (HTTP 500)
  [root@openstack-Karty nova(keystone_admin)]# openstack server list
  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'TypeError'> (HTTP 500) (Request-ID: req-7d3a09b6-2e6a-43b5-8b4b-03c1ac3d37dd)
  [root@openstack-Karty nova(keystone_admin)]# tail -f -n 100 nova-compute.log 
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     self, "Failed to establish a new connection: %s" % e
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager During handling of the above exception, another exception occurred:
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager Traceback (most recent call last):
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     timeout=timeout
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     raise MaxRetryError(_pool, url, error or ResponseError(cause))
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='10.24.0.117', port=8778): Max retries exceeded with url: /placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager During handling of the above exception, another exception occurred:
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager Traceback (most recent call last):
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1004, in _send_request
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     resp = self.session.request(method, url, **kwargs)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     resp = self.send(prep, **send_kwargs)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     r = adapter.send(request, **kwargs)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     raise ConnectionError(e, request=request)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager requests.exceptions.ConnectionError: HTTPConnectionPool(host='10.24.0.117', port=8778): Max retries exceeded with url: /placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager During handling of the above exception, another exception occurred:
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager Traceback (most recent call last):
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     startup=startup)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     self._update_available_resource(context, resources, startup=startup)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     return f(*args, **kwargs)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     self._update(context, cn, startup=startup)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     self._update_to_placement(context, compute_node, startup)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     return Retrying(*dargs, **dkw).call(f, *args, **kw)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     return attempt.get(self._wrap_exception)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     six.reraise(self.value[0], self.value[1], self.value[2])
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     raise value
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     context, compute_node.uuid, name=compute_node.hypervisor_hostname)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     rps_to_refresh = self.get_providers_in_tree(context, uuid)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 486, in get_providers_in_tree
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     global_request_id=context.global_id)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 237, in get
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     global_request_id=global_request_id)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 386, in get
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     return self.request(url, 'GET', **kwargs)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/openstack/proxy.py", line 98, in request
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     **kwargs)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 248, in request
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     return self.session.request(url, method, **kwargs)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 913, in request
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     resp = send(**kwargs)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1043, in _send_request
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     **kwargs)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1020, in _send_request
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager     raise exceptions.ConnectFailure(msg)
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://10.24.0.117:8778/placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9: HTTPConnectionPool(host='10.24.0.117', port=8778): Max retries exceeded with url: /placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
  2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 
  2020-08-17 22:59:56.498 63567 WARNING nova.virt.libvirt.imagecache [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Unknown base file: /var/lib/nova/instances/_base/5a8a3c88277cdf03ea71a4448af35b8e684f7943
  2020-08-17 22:59:56.498 63567 INFO nova.virt.libvirt.imagecache [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Removable base files: /var/lib/nova/instances/_base/5a8a3c88277cdf03ea71a4448af35b8e684f7943
  2020-08-17 22:59:56.499 63567 INFO nova.virt.libvirt.imagecache [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Base or swap file too young to remove: /var/lib/nova/instances/_base/5a8a3c88277cdf03ea71a4448af35b8e684f7943
  2020-08-17 23:00:05.503 63567 INFO nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Running instance usage audit for host openstack-Karty from 2020-08-18 05:00:00 to 2020-08-18 06:00:00. 0 instances.
  2020-08-17 23:02:10.221 63567 INFO nova.compute.claims [req-1de4c903-89cf-4494-99d5-3e6237cd67bf 108551f604a24c23989a699b058e584b 9d3e3eadbc9c4d5fa6a3b27bfee40734 - default default] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] Claim successful on node openstack-Karty
  2020-08-17 23:02:10.621 63567 INFO nova.virt.libvirt.driver [req-1de4c903-89cf-4494-99d5-3e6237cd67bf 108551f604a24c23989a699b058e584b 9d3e3eadbc9c4d5fa6a3b27bfee40734 - default default] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] Creating image
  2020-08-17 23:02:14.710 63567 INFO os_vif [req-1de4c903-89cf-4494-99d5-3e6237cd67bf 108551f604a24c23989a699b058e584b 9d3e3eadbc9c4d5fa6a3b27bfee40734 - default default] Successfully plugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:15:cb:de,bridge_name='br-int',has_traffic_filtering=True,id=bb31ebbf-b1f4-48bd-9412-5b1a8e60fecd,network=Network(ca332b6f-a3c3-4c96-bc27-5a2243929fb9),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapbb31ebbf-b1')
  2020-08-17 23:02:15.093 63567 INFO nova.compute.manager [req-b9522c87-a4bb-42ab-a9a9-a60d987e60a8 - - - - -] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] VM Started (Lifecycle Event)
  2020-08-17 23:02:15.131 63567 INFO nova.compute.manager [req-b9522c87-a4bb-42ab-a9a9-a60d987e60a8 - - - - -] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] VM Paused (Lifecycle Event)
  2020-08-17 23:02:15.199 63567 INFO nova.compute.manager [req-b9522c87-a4bb-42ab-a9a9-a60d987e60a8 - - - - -] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] During sync_power_state the instance has a pending task (spawning). Skip.
  2020-08-17 23:02:51.316 63567 ERROR nova.compute.resource_tracker [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Skipping removal of allocations for deleted instances: Failed to retrieve allocations for resource provider 3a80ea71-91a6-4da7-a05b-682b501fe6c9: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  <html><head>
  <title>500 Internal Server Error</title>
  </head><body>
  <h1>Internal Server Error</h1>
  <p>The server encountered an internal error or
  misconfiguration and was unable to complete
  your request.</p>
  <p>Please contact the server administrator at 
   [no address given] to inform them of the time this error occurred,
   and the actions you performed just before this error.</p>
  <p>More information about this error may be available
  in the server error log.</p>
  </body></html>
  : nova.exception.ResourceProviderAllocationRetrievalFailed: Failed to retrieve allocations for resource provider 3a80ea71-91a6-4da7-a05b-682b501fe6c9: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  2020-08-17 23:03:53.500 63567 ERROR nova.scheduler.client.report [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  <html><head>
  <title>500 Internal Server Error</title>
  </head><body>
  <h1>Internal Server Error</h1>
  <p>The server encountered an internal error or
  misconfiguration and was unable to complete
  your request.</p>
  <p>Please contact the server administrator at 
   [no address given] to inform them of the time this error occurred,
   and the actions you performed just before this error.</p>
  <p>More information about this error may be available
  in the server error log.</p>
  </body></html>
  .
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Error updating resources for node openstack-Karty.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager Traceback (most recent call last):
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     startup=startup)
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     self._update_available_resource(context, resources, startup=startup)
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     return f(*args, **kwargs)
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     self._update(context, cn, startup=startup)
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     self._update_to_placement(context, compute_node, startup)
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     return Retrying(*dargs, **dkw).call(f, *args, **kw)
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     return attempt.get(self._wrap_exception)
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     six.reraise(self.value[0], self.value[1], self.value[2])
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     raise value
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     context, compute_node.uuid, name=compute_node.hypervisor_hostname)
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     rps_to_refresh = self.get_providers_in_tree(context, uuid)
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 503, in get_providers_in_tree
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager     raise exception.ResourceProviderRetrievalFailed(uuid=uuid)
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9
  2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager 


  Gateway Timeout
  The gateway did not receive a timely response from the upstream server or application.

  I need to keep restarting service httpd restart inorder to overcome
  this issue. Please let us know a permanent solution. Thanks.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1891973/+subscriptions
Follow ups

[Bug 1891973] Re: Getting HTTP 500 exception frequently in Openstack Ussuri
From: Launchpad Bug Tracker, 2020-10-27