← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #74160

[Bug 1785771] [NEW] nova.compute.manager ResourceProviderRetrievalFailed: Failed to get resource provider

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

openstack:Q
centos7.5


(openstack) compute service list
+----+------------------+--------------+----------+---------+-------+----------------------------+
| ID | Binary           | Host         | Zone     | Status  | State | Updated At                 |
+----+------------------+--------------+----------+---------+-------+----------------------------+
|  4 | nova-consoleauth | controller01 | internal | enabled | up    | 2018-08-07T08:49:10.000000 |
|  7 | nova-scheduler   | controller01 | internal | enabled | up    | 2018-08-07T08:49:16.000000 |
| 13 | nova-conductor   | controller01 | internal | enabled | up    | 2018-08-07T08:49:09.000000 |
| 27 | nova-compute     | compute02    | nova     | enabled | up    | 2018-08-07T08:49:09.000000 |
+----+------------------+--------------+----------+---------+-------+----------------------------+


[root@controller01 ~]# su -s /bin/sh -c "nova-manage cell_v2 discover_hosts --verbose" nova
/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py:332: NotSupportedWarning: Configuration option(s) ['use_tpool'] not supported
  exception.NotSupportedWarning
Found 2 cell mappings.
Skipping cell0 since it does not contain hosts.
Getting computes from cell 'cell1': b781e305-4ec9-4a44-817b-71f0d91a4160
Found 0 unmapped computes in cell: b781e305-4ec9-4a44-817b-71f0d91a4160
[root@controller01 ~]# 


D08014c5-cf6b-4236-ad15-52b624d1ab4d Where is this uuid generated?No clue to this question

/var/log/nova/nova-compute.log: 
2018-08-07 16:37:56.013 168722 DEBUG oslo_concurrency.lockutils [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] Lock "compute_resources" acquired by "nova.compute.resource_tracker._update_available_resource" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273
2018-08-07 16:37:56.117 168722 DEBUG oslo_concurrency.lockutils [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] Lock "placement_client" acquired by "nova.scheduler.client.report._create_client" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273
2018-08-07 16:37:56.120 168722 DEBUG oslo_concurrency.lockutils [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] Lock "placement_client" released by "nova.scheduler.client.report._create_client" :: held 0.004s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285
2018-08-07 16:37:56.804 168722 ERROR nova.scheduler.client.report [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID d08014c5-cf6b-4236-ad15-52b624d1ab4d. Got 503: <html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
.
2018-08-07 16:37:56.805 168722 DEBUG oslo_concurrency.lockutils [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] Lock "compute_resources" released by "nova.compute.resource_tracker._update_available_resource" :: held 0.792s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] Error updating resources for node compute02.: ResourceProviderRetrievalFailed: Failed to get resource provider with UUID d08014c5-cf6b-4236-ad15-52b624d1ab4d
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager Traceback (most recent call last):
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7284, in update_available_resource_for_node
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     rt.update_available_resource(context, nodename)
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 680, in update_available_resource
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     self._update_available_resource(context, resources)
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     return f(*args, **kwargs)
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 704, in _update_available_resource
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     self._init_compute_node(context, resources)
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 571, in _init_compute_node
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     self._update(context, cn)
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 878, in _update
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     inv_data,
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 68, in set_inventory_for_provider
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid,
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     return getattr(self.instance, __name)(*args, **kwargs)
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 1103, in set_inventory_for_provider
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 660, in _ensure_resource_provider
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     rps_to_refresh = self._get_providers_in_tree(context, uuid)
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 64, in wrapper
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     return f(self, *a, **k)
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 540, in _get_providers_in_tree
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     raise exception.ResourceProviderRetrievalFailed(uuid=uuid)
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager ResourceProviderRetrievalFailed: Failed to get resource provider with UUID d08014c5-cf6b-4236-ad15-52b624d1ab4d
2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager 
2018-08-07 16:37:56.807 168722 DEBUG nova.service [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] Creating RPC server for service compute start /usr/lib/python2.7/site-packages/nova/service.py:184
2018-08-07 16:37:56.838 168722 DEBUG nova.service [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] Join ServiceGroup membership for this service compute start /usr/lib/python2.7/site-packages/nova/service.py:202
2018-08-07 16:37:56.839 168722 DEBUG nova.servicegroup.drivers.db [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] DB_Driver: join new ServiceGroup member compute02 to the compute group, service = <Service: host=compute02, binary=nova-compute, manager_class_name=nova.compute.manager.ComputeManager> join /usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py:47

2018-08-07 16:38:38.841 168722 DEBUG oslo_service.periodic_task [req-
960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] Running periodic task
ComputeManager._sync_power_states run_periodic_tasks /usr/lib/python2.7
/site-packages/oslo_service/periodic_task.py:215

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1785771

Title:
  nova.compute.manager ResourceProviderRetrievalFailed: Failed to get
  resource provider

Status in OpenStack Compute (nova):
  New

Bug description:
  openstack:Q
  centos7.5



  (openstack) compute service list
  +----+------------------+--------------+----------+---------+-------+----------------------------+
  | ID | Binary           | Host         | Zone     | Status  | State | Updated At                 |
  +----+------------------+--------------+----------+---------+-------+----------------------------+
  |  4 | nova-consoleauth | controller01 | internal | enabled | up    | 2018-08-07T08:49:10.000000 |
  |  7 | nova-scheduler   | controller01 | internal | enabled | up    | 2018-08-07T08:49:16.000000 |
  | 13 | nova-conductor   | controller01 | internal | enabled | up    | 2018-08-07T08:49:09.000000 |
  | 27 | nova-compute     | compute02    | nova     | enabled | up    | 2018-08-07T08:49:09.000000 |
  +----+------------------+--------------+----------+---------+-------+----------------------------+

  
  [root@controller01 ~]# su -s /bin/sh -c "nova-manage cell_v2 discover_hosts --verbose" nova
  /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py:332: NotSupportedWarning: Configuration option(s) ['use_tpool'] not supported
    exception.NotSupportedWarning
  Found 2 cell mappings.
  Skipping cell0 since it does not contain hosts.
  Getting computes from cell 'cell1': b781e305-4ec9-4a44-817b-71f0d91a4160
  Found 0 unmapped computes in cell: b781e305-4ec9-4a44-817b-71f0d91a4160
  [root@controller01 ~]# 

  
  D08014c5-cf6b-4236-ad15-52b624d1ab4d Where is this uuid generated?No clue to this question

  /var/log/nova/nova-compute.log: 
  2018-08-07 16:37:56.013 168722 DEBUG oslo_concurrency.lockutils [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] Lock "compute_resources" acquired by "nova.compute.resource_tracker._update_available_resource" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273
  2018-08-07 16:37:56.117 168722 DEBUG oslo_concurrency.lockutils [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] Lock "placement_client" acquired by "nova.scheduler.client.report._create_client" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273
  2018-08-07 16:37:56.120 168722 DEBUG oslo_concurrency.lockutils [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] Lock "placement_client" released by "nova.scheduler.client.report._create_client" :: held 0.004s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285
  2018-08-07 16:37:56.804 168722 ERROR nova.scheduler.client.report [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID d08014c5-cf6b-4236-ad15-52b624d1ab4d. Got 503: <html><body><h1>503 Service Unavailable</h1>
  No server is available to handle this request.
  </body></html>
  .
  2018-08-07 16:37:56.805 168722 DEBUG oslo_concurrency.lockutils [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] Lock "compute_resources" released by "nova.compute.resource_tracker._update_available_resource" :: held 0.792s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] Error updating resources for node compute02.: ResourceProviderRetrievalFailed: Failed to get resource provider with UUID d08014c5-cf6b-4236-ad15-52b624d1ab4d
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager Traceback (most recent call last):
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7284, in update_available_resource_for_node
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     rt.update_available_resource(context, nodename)
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 680, in update_available_resource
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     self._update_available_resource(context, resources)
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     return f(*args, **kwargs)
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 704, in _update_available_resource
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     self._init_compute_node(context, resources)
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 571, in _init_compute_node
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     self._update(context, cn)
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 878, in _update
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     inv_data,
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 68, in set_inventory_for_provider
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid,
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     return getattr(self.instance, __name)(*args, **kwargs)
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 1103, in set_inventory_for_provider
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 660, in _ensure_resource_provider
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     rps_to_refresh = self._get_providers_in_tree(context, uuid)
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 64, in wrapper
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     return f(self, *a, **k)
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 540, in _get_providers_in_tree
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager     raise exception.ResourceProviderRetrievalFailed(uuid=uuid)
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager ResourceProviderRetrievalFailed: Failed to get resource provider with UUID d08014c5-cf6b-4236-ad15-52b624d1ab4d
  2018-08-07 16:37:56.805 168722 ERROR nova.compute.manager 
  2018-08-07 16:37:56.807 168722 DEBUG nova.service [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] Creating RPC server for service compute start /usr/lib/python2.7/site-packages/nova/service.py:184
  2018-08-07 16:37:56.838 168722 DEBUG nova.service [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] Join ServiceGroup membership for this service compute start /usr/lib/python2.7/site-packages/nova/service.py:202
  2018-08-07 16:37:56.839 168722 DEBUG nova.servicegroup.drivers.db [req-960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] DB_Driver: join new ServiceGroup member compute02 to the compute group, service = <Service: host=compute02, binary=nova-compute, manager_class_name=nova.compute.manager.ComputeManager> join /usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py:47

  2018-08-07 16:38:38.841 168722 DEBUG oslo_service.periodic_task [req-
  960d6cfa-8ff4-4258-b7b2-eae671d27118 - - - - -] Running periodic task
  ComputeManager._sync_power_states run_periodic_tasks
  /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1785771/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next