yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #95174
[Bug 2093869] [NEW] update conflict: Inventory for 'DISK_GB' on resource provider 'r_p-id' in use
Public bug reported:
Description
===========
nova compute service is down.
caompute Unable to execute any server related requests.
Steps to reproduce
==================
not sure, this has happend 2wice in newly deployed VM on 5th-day-operation, there is enough disk available
stack@devstack-jan:~/nova$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-lv--0 52G 19G 32G 37% /
Expected result
===============
Compute service should stay active or able to restart.
Placement should be able to update resource provider
Actual result
=============
Compute service status stays down
Compute service can't be restarted.
Resource provider inventory does not get updated
stack@devstack-jan:~/nova$ openstack resource provider inventory list 5cdfba85-3122-49ea-b7b1-1fd0af461588
+----------------+------------------+----------+----------+----------+-----------+-------+------+
| resource_class | allocation_ratio | min_unit | max_unit | reserved | step_size | total | used |
+----------------+------------------+----------+----------+----------+-----------+-------+------+
| VCPU | 4.0 | 1 | 6 | 0 | 1 | 6 | 1 |
| MEMORY_MB | 1.0 | 1 | 8812 | 512 | 1 | 8812 | 2048 |
| DISK_GB | 1.0 | 1 | 29 | 0 | 1 | 29 | 20 |
+----------------+------------------+----------+----------+----------+-----------+-------+------+
If we recreate resource provider, then new once does not have DISK_GB
resource-class
Environment
===========
Single node devstack from master branch
Logs & Configs
==============
Compute-logs
-----------------
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service [None req-de66825f-768a-4951-a57b-c92f9f035255 None None] Error starting thread.: nova.exception.PlacementPciException: Failed to gather or report PCI resources to Placement: Th
ere was a conflict when trying to complete your request.
Jan 13 07:33:20 devstack-jan nova-compute[65544]: update conflict: Inventory for 'DISK_GB' on resource provider '5cdfba85-3122-49ea-b7b1-1fd0af461588' in use.
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service Traceback (most recent call last):
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1360, in _update_to_placement
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self.reportclient.update_from_provider_tree(
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/scheduler/client/report.py", line 1502, in update_from_provider_tree
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self.set_inventory_for_provider(
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/scheduler/client/report.py", line 1005, in set_inventory_for_provider
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service raise exception.InventoryInUse(err['detail'])
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service nova.exception.InventoryInUse: There was a conflict when trying to complete your request.
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service update conflict: Inventory for 'DISK_GB' on resource provider '5cdfba85-3122-49ea-b7b1-1fd0af461588' in use.
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service During handling of the above exception, another exception occurred:
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service Traceback (most recent call last):
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_service/service.py", line 810, in run_service
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service service.start()
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/service.py", line 181, in start
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self.manager.pre_start_hook(self.service_ref)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/manager.py", line 1793, in pre_start_hook
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self.update_available_resource(nova.context.get_admin_context(),
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/manager.py", line 11227, in update_available_resource
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self._update_available_resource_for_node(context, nodename,
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/manager.py", line 11102, in _update_available_resource_for_node Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self.rt.update_available_resource(context, nodename,
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/resource_tracker.py", line 935, in update_available_resource Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self._update_available_resource(context, resources, startup=startup)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py", line 412, in inner
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service return f(*args, **kwargs)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1066, in _update_available_resource
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self._update(context, cn, startup=startup)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1375, in _update
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self._update_to_placement(context, compute_node, startup)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line 56, in wrapped_f
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service return Retrying(*dargs, **dkw).call(f, *args, **kw)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line 257, in call
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service return attempt.get(self._wrap_exception)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line 301, in get
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service six.reraise(self.value[0], self.value[1], self.value[2])
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/six.py", line 724, in reraise
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service raise value
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line 251, in call
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1370, in _update_to_placement
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service raise exception.PlacementPciException(error=str(e))
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service nova.exception.PlacementPciException: Failed to gather or report PCI resources to Placement: There was a conflict when trying to complete your request.
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service update conflict: Inventory for 'DISK_GB' on resource provider '5cdfba85-3122-49ea-b7b1-1fd0af461588' in use.
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Placement-logs
---------------
Jan 13 08:35:00 devstack-jan devstack@placement-api.service[2127]: DEBUG placement.requestlog [req-69024d02-03ef-4fb7-9f71-28c34ec4d8e1 req-9262ed87-9e58-49e0-95e5-284f8899b065 None None] Starting request: 192.168.122.183 "PUT /placement/resource_providers/5cdfba85-3122-49ea-b7b1-1fd0af461588/inventories" {{(pid=2127) __call__ /opt/stack/placement/placement/requestlog.py:55}}
Jan 13 08:35:00 devstack-jan devstack@placement-api.service[2127]: DEBUG placement.wsgi_wrapper [req-69024d02-03ef-4fb7-9f71-28c34ec4d8e1 req-9262ed87-9e58-49e0-95e5-284f8899b065 service nova] Placement API returning an error response: update conflict: Inventory for 'DISK_GB' on resource provider '5cdfba85-3122-49ea-b7b1-1fd0af461588' in use. {{(pid=2127) call_func /opt/stack/placement/placement/wsgi_wrapper.py:31}}
Jan 13 08:35:00 devstack-jan devstack@placement-api.service[2127]: INFO placement.requestlog [req-69024d02-03ef-4fb7-9f71-28c34ec4d8e1 req-9262ed87-9e58-49e0-95e5-284f8899b065 service nova] 192.168.122.183 "PUT /placement/resource_providers/5cdfba85-3122-49ea-b7b1-1fd0af461588/inventories" status: 409 len: 332 microversion: 1.26
** Affects: nova
Importance: High
Status: New
** Changed in: nova
Importance: Undecided => High
** Description changed:
-
Description
===========
nova compute service is down.
- Unable to execute any API request.
-
+ caompute Unable to execute any server related requests.
Steps to reproduce
==================
not sure, this has happend 2wice in newly deployed VM on 5th-day-operation, there is enough disk available
stack@devstack-jan:~/nova$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-lv--0 52G 19G 32G 37% /
-
Expected result
===============
Compute service should stay active or able to restart.
Placement should be able to update resource provider
Actual result
=============
Compute service status stays down
Compute service can't be restarted.
Resource provider inventory does not get updated
- stack@devstack-jan:~/nova$ openstack resource provider inventory list 5cdfba85-3122-49ea-b7b1-1fd0af461588
+ stack@devstack-jan:~/nova$ openstack resource provider inventory list 5cdfba85-3122-49ea-b7b1-1fd0af461588
+----------------+------------------+----------+----------+----------+-----------+-------+------+
| resource_class | allocation_ratio | min_unit | max_unit | reserved | step_size | total | used |
+----------------+------------------+----------+----------+----------+-----------+-------+------+
| VCPU | 4.0 | 1 | 6 | 0 | 1 | 6 | 1 |
| MEMORY_MB | 1.0 | 1 | 8812 | 512 | 1 | 8812 | 2048 |
| DISK_GB | 1.0 | 1 | 29 | 0 | 1 | 29 | 20 |
+----------------+------------------+----------+----------+----------+-----------+-------+------+
If we recreate resource provider, then new once does not have DISK_GB
resource-class
-
Environment
===========
Single node devstack from master branch
Logs & Configs
==============
Compute-logs
-----------------
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service [None req-de66825f-768a-4951-a57b-c92f9f035255 None None] Error starting thread.: nova.exception.PlacementPciException: Failed to gather or report PCI resources to Placement: Th
- ere was a conflict when trying to complete your request.
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: update conflict: Inventory for 'DISK_GB' on resource provider '5cdfba85-3122-49ea-b7b1-1fd0af461588' in use.
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service Traceback (most recent call last):
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1360, in _update_to_placement
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self.reportclient.update_from_provider_tree(
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/scheduler/client/report.py", line 1502, in update_from_provider_tree
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self.set_inventory_for_provider(
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/scheduler/client/report.py", line 1005, in set_inventory_for_provider
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service raise exception.InventoryInUse(err['detail'])
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service nova.exception.InventoryInUse: There was a conflict when trying to complete your request.
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service update conflict: Inventory for 'DISK_GB' on resource provider '5cdfba85-3122-49ea-b7b1-1fd0af461588' in use.
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service During handling of the above exception, another exception occurred:
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service Traceback (most recent call last):
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_service/service.py", line 810, in run_service
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service service.start()
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/service.py", line 181, in start
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self.manager.pre_start_hook(self.service_ref)
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/manager.py", line 1793, in pre_start_hook
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self.update_available_resource(nova.context.get_admin_context(),
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/manager.py", line 11227, in update_available_resource
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self._update_available_resource_for_node(context, nodename,
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/manager.py", line 11102, in _update_available_resource_for_node Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self.rt.update_available_resource(context, nodename,
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/resource_tracker.py", line 935, in update_available_resource Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self._update_available_resource(context, resources, startup=startup)
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py", line 412, in inner
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service return f(*args, **kwargs)
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1066, in _update_available_resource
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self._update(context, cn, startup=startup)
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1375, in _update
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self._update_to_placement(context, compute_node, startup)
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line 56, in wrapped_f
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service return Retrying(*dargs, **dkw).call(f, *args, **kw)
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line 257, in call
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service return attempt.get(self._wrap_exception)
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line 301, in get
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service six.reraise(self.value[0], self.value[1], self.value[2])
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/six.py", line 724, in reraise
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service raise value
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line 251, in call
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1370, in _update_to_placement
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service raise exception.PlacementPciException(error=str(e))
+ ere was a conflict when trying to complete your request.
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: update conflict: Inventory for 'DISK_GB' on resource provider '5cdfba85-3122-49ea-b7b1-1fd0af461588' in use.
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service Traceback (most recent call last):
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1360, in _update_to_placement
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self.reportclient.update_from_provider_tree(
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/scheduler/client/report.py", line 1502, in update_from_provider_tree
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self.set_inventory_for_provider(
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/scheduler/client/report.py", line 1005, in set_inventory_for_provider
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service raise exception.InventoryInUse(err['detail'])
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service nova.exception.InventoryInUse: There was a conflict when trying to complete your request.
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service update conflict: Inventory for 'DISK_GB' on resource provider '5cdfba85-3122-49ea-b7b1-1fd0af461588' in use.
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service During handling of the above exception, another exception occurred:
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service Traceback (most recent call last):
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_service/service.py", line 810, in run_service
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service service.start()
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/service.py", line 181, in start
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self.manager.pre_start_hook(self.service_ref)
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/manager.py", line 1793, in pre_start_hook
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self.update_available_resource(nova.context.get_admin_context(),
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/manager.py", line 11227, in update_available_resource
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self._update_available_resource_for_node(context, nodename,
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/manager.py", line 11102, in _update_available_resource_for_node Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self.rt.update_available_resource(context, nodename,
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/resource_tracker.py", line 935, in update_available_resource Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self._update_available_resource(context, resources, startup=startup)
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py", line 412, in inner
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service return f(*args, **kwargs)
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1066, in _update_available_resource
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self._update(context, cn, startup=startup)
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1375, in _update
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self._update_to_placement(context, compute_node, startup)
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line 56, in wrapped_f
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service return Retrying(*dargs, **dkw).call(f, *args, **kw)
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line 257, in call
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service return attempt.get(self._wrap_exception)
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line 301, in get
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service six.reraise(self.value[0], self.value[1], self.value[2])
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/six.py", line 724, in reraise
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service raise value
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line 251, in call
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1370, in _update_to_placement
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service raise exception.PlacementPciException(error=str(e))
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service nova.exception.PlacementPciException: Failed to gather or report PCI resources to Placement: There was a conflict when trying to complete your request.
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service update conflict: Inventory for 'DISK_GB' on resource provider '5cdfba85-3122-49ea-b7b1-1fd0af461588' in use.
- Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
-
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service update conflict: Inventory for 'DISK_GB' on resource provider '5cdfba85-3122-49ea-b7b1-1fd0af461588' in use.
+ Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Placement-logs
---------------
Jan 13 08:35:00 devstack-jan devstack@placement-api.service[2127]: DEBUG placement.requestlog [req-69024d02-03ef-4fb7-9f71-28c34ec4d8e1 req-9262ed87-9e58-49e0-95e5-284f8899b065 None None] Starting request: 192.168.122.183 "PUT /placement/resource_providers/5cdfba85-3122-49ea-b7b1-1fd0af461588/inventories" {{(pid=2127) __call__ /opt/stack/placement/placement/requestlog.py:55}}
Jan 13 08:35:00 devstack-jan devstack@placement-api.service[2127]: DEBUG placement.wsgi_wrapper [req-69024d02-03ef-4fb7-9f71-28c34ec4d8e1 req-9262ed87-9e58-49e0-95e5-284f8899b065 service nova] Placement API returning an error response: update conflict: Inventory for 'DISK_GB' on resource provider '5cdfba85-3122-49ea-b7b1-1fd0af461588' in use. {{(pid=2127) call_func /opt/stack/placement/placement/wsgi_wrapper.py:31}}
Jan 13 08:35:00 devstack-jan devstack@placement-api.service[2127]: INFO placement.requestlog [req-69024d02-03ef-4fb7-9f71-28c34ec4d8e1 req-9262ed87-9e58-49e0-95e5-284f8899b065 service nova] 192.168.122.183 "PUT /placement/resource_providers/5cdfba85-3122-49ea-b7b1-1fd0af461588/inventories" status: 409 len: 332 microversion: 1.26
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2093869
Title:
update conflict: Inventory for 'DISK_GB' on resource provider 'r_p-id'
in use
Status in OpenStack Compute (nova):
New
Bug description:
Description
===========
nova compute service is down.
caompute Unable to execute any server related requests.
Steps to reproduce
==================
not sure, this has happend 2wice in newly deployed VM on 5th-day-operation, there is enough disk available
stack@devstack-jan:~/nova$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-lv--0 52G 19G 32G 37% /
Expected result
===============
Compute service should stay active or able to restart.
Placement should be able to update resource provider
Actual result
=============
Compute service status stays down
Compute service can't be restarted.
Resource provider inventory does not get updated
stack@devstack-jan:~/nova$ openstack resource provider inventory list 5cdfba85-3122-49ea-b7b1-1fd0af461588
+----------------+------------------+----------+----------+----------+-----------+-------+------+
| resource_class | allocation_ratio | min_unit | max_unit | reserved | step_size | total | used |
+----------------+------------------+----------+----------+----------+-----------+-------+------+
| VCPU | 4.0 | 1 | 6 | 0 | 1 | 6 | 1 |
| MEMORY_MB | 1.0 | 1 | 8812 | 512 | 1 | 8812 | 2048 |
| DISK_GB | 1.0 | 1 | 29 | 0 | 1 | 29 | 20 |
+----------------+------------------+----------+----------+----------+-----------+-------+------+
If we recreate resource provider, then new once does not have DISK_GB
resource-class
Environment
===========
Single node devstack from master branch
Logs & Configs
==============
Compute-logs
-----------------
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service [None req-de66825f-768a-4951-a57b-c92f9f035255 None None] Error starting thread.: nova.exception.PlacementPciException: Failed to gather or report PCI resources to Placement: Th
ere was a conflict when trying to complete your request.
Jan 13 07:33:20 devstack-jan nova-compute[65544]: update conflict: Inventory for 'DISK_GB' on resource provider '5cdfba85-3122-49ea-b7b1-1fd0af461588' in use.
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service Traceback (most recent call last):
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1360, in _update_to_placement
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self.reportclient.update_from_provider_tree(
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/scheduler/client/report.py", line 1502, in update_from_provider_tree
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self.set_inventory_for_provider(
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/scheduler/client/report.py", line 1005, in set_inventory_for_provider
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service raise exception.InventoryInUse(err['detail'])
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service nova.exception.InventoryInUse: There was a conflict when trying to complete your request.
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service update conflict: Inventory for 'DISK_GB' on resource provider '5cdfba85-3122-49ea-b7b1-1fd0af461588' in use.
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service During handling of the above exception, another exception occurred:
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service Traceback (most recent call last):
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_service/service.py", line 810, in run_service
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service service.start()
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/service.py", line 181, in start
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self.manager.pre_start_hook(self.service_ref)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/manager.py", line 1793, in pre_start_hook
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self.update_available_resource(nova.context.get_admin_context(),
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/manager.py", line 11227, in update_available_resource
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self._update_available_resource_for_node(context, nodename,
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/manager.py", line 11102, in _update_available_resource_for_node Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self.rt.update_available_resource(context, nodename,
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/resource_tracker.py", line 935, in update_available_resource Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self._update_available_resource(context, resources, startup=startup)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py", line 412, in inner
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service return f(*args, **kwargs)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1066, in _update_available_resource
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self._update(context, cn, startup=startup)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1375, in _update
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service self._update_to_placement(context, compute_node, startup)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line 56, in wrapped_f
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service return Retrying(*dargs, **dkw).call(f, *args, **kw)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line 257, in call
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service return attempt.get(self._wrap_exception)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line 301, in get
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service six.reraise(self.value[0], self.value[1], self.value[2])
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/six.py", line 724, in reraise
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service raise value
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/retrying.py", line 251, in call
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1370, in _update_to_placement
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service raise exception.PlacementPciException(error=str(e))
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service nova.exception.PlacementPciException: Failed to gather or report PCI resources to Placement: There was a conflict when trying to complete your request.
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service update conflict: Inventory for 'DISK_GB' on resource provider '5cdfba85-3122-49ea-b7b1-1fd0af461588' in use.
Jan 13 07:33:20 devstack-jan nova-compute[65544]: ERROR oslo_service.service
Placement-logs
---------------
Jan 13 08:35:00 devstack-jan devstack@placement-api.service[2127]: DEBUG placement.requestlog [req-69024d02-03ef-4fb7-9f71-28c34ec4d8e1 req-9262ed87-9e58-49e0-95e5-284f8899b065 None None] Starting request: 192.168.122.183 "PUT /placement/resource_providers/5cdfba85-3122-49ea-b7b1-1fd0af461588/inventories" {{(pid=2127) __call__ /opt/stack/placement/placement/requestlog.py:55}}
Jan 13 08:35:00 devstack-jan devstack@placement-api.service[2127]: DEBUG placement.wsgi_wrapper [req-69024d02-03ef-4fb7-9f71-28c34ec4d8e1 req-9262ed87-9e58-49e0-95e5-284f8899b065 service nova] Placement API returning an error response: update conflict: Inventory for 'DISK_GB' on resource provider '5cdfba85-3122-49ea-b7b1-1fd0af461588' in use. {{(pid=2127) call_func /opt/stack/placement/placement/wsgi_wrapper.py:31}}
Jan 13 08:35:00 devstack-jan devstack@placement-api.service[2127]: INFO placement.requestlog [req-69024d02-03ef-4fb7-9f71-28c34ec4d8e1 req-9262ed87-9e58-49e0-95e5-284f8899b065 service nova] 192.168.122.183 "PUT /placement/resource_providers/5cdfba85-3122-49ea-b7b1-1fd0af461588/inventories" status: 409 len: 332 microversion: 1.26
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2093869/+subscriptions
