yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1785771] Re: nova.compute.manager ResourceProviderRetrievalFailed: Failed to get resource provider

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Wed, 08 Aug 2018 19:24:31 -0000
Reply-to: Bug 1785771 <1785771@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
It looks like placement isn't running, you got a 503 response from the
placement API, not a 404. The compute node record in the 'nova' (cell1)
database is auto-generated when the nova-compute service starts up. That
compute node record uuid is used to create the resource provider in
placement.

So:

select uuid from nova.compute_nodes where host='compute02';

And I expect you'll see d08014c5-cf6b-4236-ad15-52b624d1ab4d. Can you
confirm?

But the real error is here:

2018-08-07 16:37:56.804 168722 ERROR nova.scheduler.client.report [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID d08014c5-cf6b-4236-ad15-52b624d1ab4d. Got 503: <html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

Check the placement-api logs for errors - why is a 503 being returned?

** Summary changed:

- nova.compute.manager ResourceProviderRetrievalFailed: Failed to get resource provider
+ nova.compute.manager ResourceProviderRetrievalFailed: Failed to get resource provider (503)

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1785771

Title:
  nova.compute.manager ResourceProviderRetrievalFailed: Failed to get
  resource provider (503)

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  openstack:Q
  centos7.5



  (openstack) compute service list
  +----+------------------+--------------+----------+---------+-------+----------------------------+
  | ID | Binary           | Host         | Zone     | Status  | State | Updated At                 |
  +----+------------------+--------------+----------+---------+-------+----------------------------+
  |  4 | nova-consoleauth | controller01 | internal | enabled | up    | 2018-08-07T08:49:10.000000 |
  |  7 | nova-scheduler   | controller01 | internal | enabled | up    | 2018-08-07T08:49:16.000000 |
  | 13 | nova-conductor   | controller01 | internal | enabled | up    | 2018-08-07T08:49:09.000000 |
  | 27 | nova-compute     | compute02    | nova     | enabled | up    | 2018-08-07T08:49:09.000000 |
  +----+------------------+--------------+----------+---------+-------+----------------------------+

  
  [root@controller01 ~]# su -s /bin/sh -c "nova-manage cell_v2 discover_hosts --verbose" nova
  /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py:332: NotSupportedWarning: Configuration option(s) ['use_tpool'] not supported
    exception.NotSupportedWarning
  Found 2 cell mappings.
  Skipping cell0 since it does not contain hosts.
  Getting computes from cell 'cell1': b781e305-4ec9-4a44-817b-71f0d91a4160
  Found 0 unmapped computes in cell: b781e305-4ec9-4a44-817b-71f0d91a4160
  [root@controller01 ~]# 

  
  D08014c5-cf6b-4236-ad15-52b624d1ab4d Where is this uuid generated?No clue to this question

  /var/log/nova/nova-compute.log: 
  2018-08-07 16:37:56.013 168722 DEBUG oslo_concurrency.lockutils [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] Lock "compute_resources" acquired by "nova.compute.resource_tracker._update_available_resource" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273
  2018-08-07 16:37:56.117 168722 DEBUG oslo_concurrency.lockutils [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] Lock "placement_client" acquired by "nova.scheduler.client.report._create_client" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273
  2018-08-07 16:37:56.120 168722 DEBUG oslo_concurrency.lockutils [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] Lock "placement_client" released by "nova.scheduler.client.report._create_client" :: held 0.004s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285
  2018-08-07 16:37:56.804 168722 ERROR nova.scheduler.client.report [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID d08014c5-cf6b-4236-ad15-52b624d1ab4d. Got 503: <html><body><h1>503 Service Unavailable</h1>
  No server is available to handle this request.
  </body></html>
  .
  2018-08-07 16:37:56.805 168722 DEBUG oslo_concurrency.lockutils [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] Lock "compute_resources" released by "nova.compute.resource_tracker._update_available_resource" :: held 0.792s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] Error updating resources for node compute02.: ResourceProviderRetrievalFailed: Failed to get resource provider with UUID d08014c5-cf6b-4236-ad15-52b624d1ab4d
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager Traceback (most recent call last):
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7284, in update_available_resource_for_node
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     rt.update_available_resource(context, nodename)
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 680, in update_available_resource
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     self._update_available_resource(context, resources)
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     return f(*args, **kwargs)
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 704, in _update_available_resource
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     self._init_compute_node(context, resources)
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 571, in _init_compute_node
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     self._update(context, cn)
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 878, in _update
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     inv_data,
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 68, in set_inventory_for_provider
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid,
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     return getattr(self.instance, __name)(*args, **kwargs)
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 1103, in set_inventory_for_provider
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 660, in _ensure_resource_provider
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     rps_to_refresh = self._get_providers_in_tree(context, uuid)
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 64, in wrapper
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     return f(self, *a, **k)
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 540, in _get_providers_in_tree
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     raise exception.ResourceProviderRetrievalFailed(uuid=uuid)
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager ResourceProviderRetrievalFailed: Failed to get resource provider with UUID d08014c5-cf6b-4236-ad15-52b624d1ab4d
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager 
  2018-08-07 16:37:56.807 168722 DEBUG nova.service [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] Creating RPC server for service compute start /usr/lib/python2.7/site-packages/nova/service.py:184
  2018-08-07 16:37:56.838 168722 DEBUG nova.service [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] Join ServiceGroup membership for this service compute start /usr/lib/python2.7/site-packages/nova/service.py:202
  2018-08-07 16:37:56.839 168722 DEBUG nova.servicegroup.drivers.db [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] DB_Driver: join new ServiceGroup member compute02 to the compute group, service = <Service: host=compute02, binary=nova-compute, manager_class_name=nova.compute.manager.ComputeManager> join /usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py:47

  2018-08-07 16:38:38.841 168722 DEBUG oslo_service.periodic_task [req-
  960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] Running periodic task
  ComputeManager._sync_power_states run_periodic_tasks
  /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1785771/+subscriptions
References

[Bug 1785771] [NEW] nova.compute.manager ResourceProviderRetrievalFailed: Failed to get resource provider
From: men, 2018-08-07