← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 2119542] [NEW] Nova fails to reshape compute node with cpu_dedicated_set after workload migration and zero allocations

 

Public bug reported:

Summary:
After configuring cpu_dedicated_set on a compute node in nova.conf, Nova fails to perform the required resource provider reshape, even when all instances have been migrated away, and Placement shows zero allocations. The error is persistent across service restarts and results in the compute node flapping between old and new inventory layouts (e.g., PCPU appears and disappears).

Steps to Reproduce:
1) Deploy OpenStack (tested on 2024.1 Caracal).

2) Schedule several instances to a compute node (e.g., compute01).

3) Migrate all instances off that node so it has no active workloads.

4) Verify openstack server list --all --host compute01 is empty.

5) Add the following to /etc/nova/nova.conf on compute01:

[compute]
cpu_shared_set = 0-17,36-53
cpu_dedicated_set = 18-35,54-71


6) Restart nova-compute service.

Expected Result:
Nova should detect the change, perform a reshape, and update Placement to report PCPU inventory.

Actual Result:

Nova logs show:

"2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task Traceback (most recent call last):
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     task(self, context)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 10731, in update_available_resource
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     self._update_available_resource_for_node(context, nodename,
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 10633, in _update_available_resource_for_node
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     with excutils.save_and_reraise_exception():
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     self.force_reraise()
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     raise self.value
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 10606, in _update_available_resource_for_node
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     self.rt.update_available_resource(context, nodename,
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 935, in update_available_resource
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     self._update_available_resource(context, resources, startup=startup)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py", line 412, in inner
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     return f(*args, **kwargs)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 1066, in _update_available_resource
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     self._update(context, cn, startup=startup)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 1375, in _update
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     self._update_to_placement(context, compute_node, startup)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/retrying.py", line 56, in wrapped_f
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     return Retrying(*dargs, **dkw).call(f, *args, **kw)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/retrying.py", line 257, in call
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     return attempt.get(self._wrap_exception)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/retrying.py", line 301, in get
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     six.reraise(self.value[0], self.value[1], self.value[2])
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/six.py", line 719, in reraise
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     raise value
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/retrying.py", line 251, in call
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 1288, in _update_to_placement
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     self.driver.update_provider_tree(prov_tree, nodename)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 9284, in update_provider_tree
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     self._update_provider_tree_for_pcpu(
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 9801, in _update_provider_tree_for_pcpu
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     raise exception.ReshapeNeeded()
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task nova.exception.ReshapeNeeded: Virt driver indicates that provider inventories need to be moved.
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task
2025-08-04 19:15:52.787 7 ERROR nova.compute.manager [None req-caeacd66-7732-4826-89ea-f2231fed87aa - - - - - -] ReshapeNeeded exception is unexpected here!: nova.exception.ReshapeNeeded: Virt driver indicates that provider inventories need to be moved."

No reshape is performed.

Placement inventory flips between VCPU-only and VCPU+PCPU models.

No resource allocations exist on the host (confirmed via openstack
allocation list).

Additional Info:
The issue persists even after multiple restarts of nova-compute

Environment:
Nova version: 2024.1 (Caracal)

Placement version: 11.0.0

Deployment: Kolla-Ansible

Host: drained compute node with no allocations

CPU pinning being enabled for the first time (cpu_dedicated_set)

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: nova

** Description changed:

  Summary:
  After configuring cpu_dedicated_set on a compute node in nova.conf, Nova fails to perform the required resource provider reshape, even when all instances have been migrated away, and Placement shows zero allocations. The error is persistent across service restarts and results in the compute node flapping between old and new inventory layouts (e.g., PCPU appears and disappears).
- 
  
  Steps to Reproduce:
  1) Deploy OpenStack (tested on 2024.1 Caracal).
  
  2) Schedule several instances to a compute node (e.g., compute01).
  
  3) Migrate all instances off that node so it has no active workloads.
  
  4) Verify openstack server list --all --host compute01 is empty.
  
  5) Add the following to /etc/nova/nova.conf on compute01:
  
  [compute]
- cpu_dedicated_set = 18-35
- cpu_shared_set = 0-17,36-71
+ cpu_shared_set = 0-17,36-53
+ cpu_dedicated_set = 18-35,54-71
+ 
  
  6) Restart nova-compute service.
- 
  
  Expected Result:
  Nova should detect the change, perform a reshape, and update Placement to report PCPU inventory.
  
  Actual Result:
  
  Nova logs show:
  
  "2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task Traceback (most recent call last):
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     task(self, context)
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 10731, in update_available_resource
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     self._update_available_resource_for_node(context, nodename,
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 10633, in _update_available_resource_for_node
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     with excutils.save_and_reraise_exception():
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     self.force_reraise()
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     raise self.value
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 10606, in _update_available_resource_for_node
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     self.rt.update_available_resource(context, nodename,
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 935, in update_available_resource
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     self._update_available_resource(context, resources, startup=startup)
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py", line 412, in inner
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     return f(*args, **kwargs)
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 1066, in _update_available_resource
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     self._update(context, cn, startup=startup)
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 1375, in _update
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     self._update_to_placement(context, compute_node, startup)
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/retrying.py", line 56, in wrapped_f
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     return Retrying(*dargs, **dkw).call(f, *args, **kw)
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/retrying.py", line 257, in call
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     return attempt.get(self._wrap_exception)
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/retrying.py", line 301, in get
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     six.reraise(self.value[0], self.value[1], self.value[2])
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/six.py", line 719, in reraise
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     raise value
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/retrying.py", line 251, in call
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 1288, in _update_to_placement
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     self.driver.update_provider_tree(prov_tree, nodename)
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 9284, in update_provider_tree
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     self._update_provider_tree_for_pcpu(
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 9801, in _update_provider_tree_for_pcpu
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     raise exception.ReshapeNeeded()
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task nova.exception.ReshapeNeeded: Virt driver indicates that provider inventories need to be moved.
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task
  2025-08-04 19:15:52.787 7 ERROR nova.compute.manager [None req-caeacd66-7732-4826-89ea-f2231fed87aa - - - - - -] ReshapeNeeded exception is unexpected here!: nova.exception.ReshapeNeeded: Virt driver indicates that provider inventories need to be moved."
  
- 
  No reshape is performed.
  
  Placement inventory flips between VCPU-only and VCPU+PCPU models.
  
  No resource allocations exist on the host (confirmed via openstack
  allocation list).
  
- 
  Additional Info:
  The issue persists even after multiple restarts of nova-compute
- 
  
  Environment:
  Nova version: 2024.1 (Caracal)
  
  Placement version: 11.0.0
  
  Deployment: Kolla-Ansible
  
  Host: drained compute node with no allocations
  
  CPU pinning being enabled for the first time (cpu_dedicated_set)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2119542

Title:
  Nova fails to reshape compute node with cpu_dedicated_set after
  workload migration and zero allocations

Status in OpenStack Compute (nova):
  New

Bug description:
  Summary:
  After configuring cpu_dedicated_set on a compute node in nova.conf, Nova fails to perform the required resource provider reshape, even when all instances have been migrated away, and Placement shows zero allocations. The error is persistent across service restarts and results in the compute node flapping between old and new inventory layouts (e.g., PCPU appears and disappears).

  Steps to Reproduce:
  1) Deploy OpenStack (tested on 2024.1 Caracal).

  2) Schedule several instances to a compute node (e.g., compute01).

  3) Migrate all instances off that node so it has no active workloads.

  4) Verify openstack server list --all --host compute01 is empty.

  5) Add the following to /etc/nova/nova.conf on compute01:

  [compute]
  cpu_shared_set = 0-17,36-53
  cpu_dedicated_set = 18-35,54-71

  
  6) Restart nova-compute service.

  Expected Result:
  Nova should detect the change, perform a reshape, and update Placement to report PCPU inventory.

  Actual Result:

  Nova logs show:

  "2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task Traceback (most recent call last):
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     task(self, context)
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 10731, in update_available_resource
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     self._update_available_resource_for_node(context, nodename,
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 10633, in _update_available_resource_for_node
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     with excutils.save_and_reraise_exception():
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     self.force_reraise()
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     raise self.value
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 10606, in _update_available_resource_for_node
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     self.rt.update_available_resource(context, nodename,
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 935, in update_available_resource
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     self._update_available_resource(context, resources, startup=startup)
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py", line 412, in inner
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     return f(*args, **kwargs)
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 1066, in _update_available_resource
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     self._update(context, cn, startup=startup)
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 1375, in _update
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     self._update_to_placement(context, compute_node, startup)
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/retrying.py", line 56, in wrapped_f
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     return Retrying(*dargs, **dkw).call(f, *args, **kw)
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/retrying.py", line 257, in call
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     return attempt.get(self._wrap_exception)
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/retrying.py", line 301, in get
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     six.reraise(self.value[0], self.value[1], self.value[2])
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/six.py", line 719, in reraise
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     raise value
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/retrying.py", line 251, in call
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 1288, in _update_to_placement
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     self.driver.update_provider_tree(prov_tree, nodename)
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 9284, in update_provider_tree
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     self._update_provider_tree_for_pcpu(
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 9801, in _update_provider_tree_for_pcpu
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task     raise exception.ReshapeNeeded()
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task nova.exception.ReshapeNeeded: Virt driver indicates that provider inventories need to be moved.
  2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task
  2025-08-04 19:15:52.787 7 ERROR nova.compute.manager [None req-caeacd66-7732-4826-89ea-f2231fed87aa - - - - - -] ReshapeNeeded exception is unexpected here!: nova.exception.ReshapeNeeded: Virt driver indicates that provider inventories need to be moved."

  No reshape is performed.

  Placement inventory flips between VCPU-only and VCPU+PCPU models.

  No resource allocations exist on the host (confirmed via openstack
  allocation list).

  Additional Info:
  The issue persists even after multiple restarts of nova-compute

  Environment:
  Nova version: 2024.1 (Caracal)

  Placement version: 11.0.0

  Deployment: Kolla-Ansible

  Host: drained compute node with no allocations

  CPU pinning being enabled for the first time (cpu_dedicated_set)

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2119542/+subscriptions