← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 2023035] [NEW] 2023.1: Live migration fails with new compare_hypervisor_cpu method

 

Public bug reported:

We have test setup with 3 identical compute nodes. Recently test setup
upgraded from Zed to 2023.1 Openstack Version, with nova 27.0.0, libvirt
8.0.0, qemu 6.2

During live migration we always getting error:

2023-06-05 13:33:06.158 1458525 ERROR nova.virt.libvirt.driver [None
req-6295f150-f0cf-41d1-8dd4-60c0ac32223f
59de9e2e2a8a413384be5ee27e027fc1 185764021e19409dae135a967f032fa4 - -
b589410bd7e14872bf3ac74c45057691 b589410bd7e14872bf3ac74c45057691] CPU
doesn't have compatibility.

0

Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server [None req-6295f150-f0cf-41d1-8dd4-60c0ac32223f 59de9e2e2a8a413384be5ee27e027fc1 185764021e19409dae135a967f032fa4 - - b589410bd7e14872bf3ac74c45057691 b589410bd7e14872bf3ac74c45057691] Exception during message handling: nova.exception.MigrationPreCheckEr
ror: Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility.

0

Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 9637, in check_can_live_migrate_destination
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     self._compare_cpu(None, source_cpu_info, instance)
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 10014, in _compare_cpu
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     raise exception.InvalidCPUInfo(reason=m % {'ret': ret, 'u': u})
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server nova.exception.InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility.
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server 0
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 65, in wrapped
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     self.force_reraise()
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     raise self.value
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 63, in wrapped
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1439, in decorated_function
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 213, in decorated_function
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     self.force_reraise()
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     raise self.value
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 203, in decorated_function
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 8408, in check_can_live_migrate_destination
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     dest_check_data = self.driver.check_can_live_migrate_destination(ctxt,
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 9641, in check_can_live_migrate_destination
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     raise exception.MigrationPreCheckError(reason=e)
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server nova.exception.MigrationPreCheckError: Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility.


Zed setup works ok with this nodes.

More info:

1. https://github.com/openstack/nova/commit/468b03e0ee4a917ae26106f6e57081bcd9e7a65b
If revert libvirt to old cpu_compare instead cpu_hypervisor_compare in libvirt driver
ret = self._host.compare_cpu(cpu_xml)
#ret = self._host.compare_hypervisor_cpu(cpu_xml)
live migration works ok.
ret = self._host.compare_cpu(cpu_xml) - always returns 2 (VIR_CPU_COMPARE_SUPERSET)
ret = self._host.compare_hypervisor_cpu(cpu_xml) - always returns 0 (VIR_CPU_COMPARE_INCOMPATIBLE)

2. Disabling cpu Comparation with changing nova.conf:

[workarounds]
skip_cpu_compare_on_dest = True

makes live migration work ok.

3. Entry xml to cpu_compare looks like:
<cpu>
  <arch>x86_64</arch>
  <model>Skylake-Client-noTSX-IBRS</model>
  <vendor>Intel</vendor>
  <topology sockets="1" cores="4" threads="2"/>
  <feature name="3dnowprefetch"/>
  <feature name="abm"/>
  <feature name="acpi"/>
  <feature name="adx"/>
  <feature name="aes"/>
  <feature name="apic"/>
  <feature name="arat"/>
  <feature name="arch-capabilities"/>
  <feature name="avx"/>
  <feature name="avx2"/>
  <feature name="bmi1"/>
  <feature name="bmi2"/>
  <feature name="clflush"/>
  <feature name="clflushopt"/>
  <feature name="cmov"/>
  <feature name="cx16"/>
  <feature name="cx8"/>
  <feature name="de"/>
  <feature name="ds"/>
  <feature name="ds_cpl"/>
  <feature name="dtes64"/>
  <feature name="erms"/>
  <feature name="est"/>
  <feature name="f16c"/>
  <feature name="fma"/>
  <feature name="fpu"/>
  <feature name="fsgsbase"/>
  <feature name="fxsr"/>
  <feature name="ht"/>
  <feature name="intel-pt"/>
  <feature name="invpcid"/>
  <feature name="invtsc"/>
  <feature name="lahf_lm"/>
  <feature name="lm"/>
  <feature name="mca"/>
  <feature name="mce"/>
  <feature name="md-clear"/>
  <feature name="mmx"/>
  <feature name="monitor"/>
  <feature name="movbe"/>
  <feature name="mpx"/>
  <feature name="msr"/>
  <feature name="mtrr"/>
  <feature name="nx"/>
  <feature name="pae"/>
  <feature name="pat"/>
  <feature name="pbe"/>
  <feature name="pcid"/>
  <feature name="pclmuldq"/>
  <feature name="pdcm"/>
  <feature name="pdpe1gb"/>
  <feature name="pge"/>
  <feature name="pni"/>
  <feature name="popcnt"/>
  <feature name="pse"/>
  <feature name="pse36"/>
  <feature name="rdrand"/>
  <feature name="rdseed"/>
  <feature name="rdtscp"/>
  <feature name="rsba"/>
  <feature name="sep"/>
  <feature name="smap"/>
  <feature name="smep"/>
  <feature name="smx"/>
  <feature name="spec-ctrl"/>
  <feature name="ss"/>
  <feature name="ssbd"/>
  <feature name="sse"/>
  <feature name="sse2"/>
  <feature name="sse4.1"/>
  <feature name="sse4.2"/>
  <feature name="ssse3"/>
  <feature name="stibp"/>
  <feature name="syscall"/>
  <feature name="tm"/>
  <feature name="tm2"/>
  <feature name="tsc"/>
  <feature name="tsc-deadline"/>
  <feature name="tsc_adjust"/>
  <feature name="vme"/>
  <feature name="vmx"/>
  <feature name="x2apic"/>
  <feature name="xgetbv1"/>
  <feature name="xsave"/>
  <feature name="xsavec"/>
  <feature name="xsaveopt"/>
  <feature name="xsaves"/>
  <feature name="xtpr"/>
</cpu>

nova.conf
[libvirt]
cpu_mode = host-model

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2023035

Title:
  2023.1: Live migration fails with new compare_hypervisor_cpu method

Status in OpenStack Compute (nova):
  New

Bug description:
  We have test setup with 3 identical compute nodes. Recently test setup
  upgraded from Zed to 2023.1 Openstack Version, with nova 27.0.0,
  libvirt 8.0.0, qemu 6.2

  During live migration we always getting error:

  2023-06-05 13:33:06.158 1458525 ERROR nova.virt.libvirt.driver [None
  req-6295f150-f0cf-41d1-8dd4-60c0ac32223f
  59de9e2e2a8a413384be5ee27e027fc1 185764021e19409dae135a967f032fa4 - -
  b589410bd7e14872bf3ac74c45057691 b589410bd7e14872bf3ac74c45057691] CPU
  doesn't have compatibility.

  0

  Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server [None req-6295f150-f0cf-41d1-8dd4-60c0ac32223f 59de9e2e2a8a413384be5ee27e027fc1 185764021e19409dae135a967f032fa4 - - b589410bd7e14872bf3ac74c45057691 b589410bd7e14872bf3ac74c45057691] Exception during message handling: nova.exception.MigrationPreCheckEr
  ror: Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility.

  0

  Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 9637, in check_can_live_migrate_destination
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     self._compare_cpu(None, source_cpu_info, instance)
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 10014, in _compare_cpu
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     raise exception.InvalidCPUInfo(reason=m % {'ret': ret, 'u': u})
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server nova.exception.InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility.
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server 0
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 65, in wrapped
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     raise self.value
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 63, in wrapped
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1439, in decorated_function
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 213, in decorated_function
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     raise self.value
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 203, in decorated_function
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 8408, in check_can_live_migrate_destination
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     dest_check_data = self.driver.check_can_live_migrate_destination(ctxt,
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 9641, in check_can_live_migrate_destination
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server     raise exception.MigrationPreCheckError(reason=e)
  2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server nova.exception.MigrationPreCheckError: Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility.

  
  Zed setup works ok with this nodes.

  More info:

  1. https://github.com/openstack/nova/commit/468b03e0ee4a917ae26106f6e57081bcd9e7a65b
  If revert libvirt to old cpu_compare instead cpu_hypervisor_compare in libvirt driver
  ret = self._host.compare_cpu(cpu_xml)
  #ret = self._host.compare_hypervisor_cpu(cpu_xml)
  live migration works ok.
  ret = self._host.compare_cpu(cpu_xml) - always returns 2 (VIR_CPU_COMPARE_SUPERSET)
  ret = self._host.compare_hypervisor_cpu(cpu_xml) - always returns 0 (VIR_CPU_COMPARE_INCOMPATIBLE)

  2. Disabling cpu Comparation with changing nova.conf:

  [workarounds]
  skip_cpu_compare_on_dest = True

  makes live migration work ok.

  3. Entry xml to cpu_compare looks like:
  <cpu>
    <arch>x86_64</arch>
    <model>Skylake-Client-noTSX-IBRS</model>
    <vendor>Intel</vendor>
    <topology sockets="1" cores="4" threads="2"/>
    <feature name="3dnowprefetch"/>
    <feature name="abm"/>
    <feature name="acpi"/>
    <feature name="adx"/>
    <feature name="aes"/>
    <feature name="apic"/>
    <feature name="arat"/>
    <feature name="arch-capabilities"/>
    <feature name="avx"/>
    <feature name="avx2"/>
    <feature name="bmi1"/>
    <feature name="bmi2"/>
    <feature name="clflush"/>
    <feature name="clflushopt"/>
    <feature name="cmov"/>
    <feature name="cx16"/>
    <feature name="cx8"/>
    <feature name="de"/>
    <feature name="ds"/>
    <feature name="ds_cpl"/>
    <feature name="dtes64"/>
    <feature name="erms"/>
    <feature name="est"/>
    <feature name="f16c"/>
    <feature name="fma"/>
    <feature name="fpu"/>
    <feature name="fsgsbase"/>
    <feature name="fxsr"/>
    <feature name="ht"/>
    <feature name="intel-pt"/>
    <feature name="invpcid"/>
    <feature name="invtsc"/>
    <feature name="lahf_lm"/>
    <feature name="lm"/>
    <feature name="mca"/>
    <feature name="mce"/>
    <feature name="md-clear"/>
    <feature name="mmx"/>
    <feature name="monitor"/>
    <feature name="movbe"/>
    <feature name="mpx"/>
    <feature name="msr"/>
    <feature name="mtrr"/>
    <feature name="nx"/>
    <feature name="pae"/>
    <feature name="pat"/>
    <feature name="pbe"/>
    <feature name="pcid"/>
    <feature name="pclmuldq"/>
    <feature name="pdcm"/>
    <feature name="pdpe1gb"/>
    <feature name="pge"/>
    <feature name="pni"/>
    <feature name="popcnt"/>
    <feature name="pse"/>
    <feature name="pse36"/>
    <feature name="rdrand"/>
    <feature name="rdseed"/>
    <feature name="rdtscp"/>
    <feature name="rsba"/>
    <feature name="sep"/>
    <feature name="smap"/>
    <feature name="smep"/>
    <feature name="smx"/>
    <feature name="spec-ctrl"/>
    <feature name="ss"/>
    <feature name="ssbd"/>
    <feature name="sse"/>
    <feature name="sse2"/>
    <feature name="sse4.1"/>
    <feature name="sse4.2"/>
    <feature name="ssse3"/>
    <feature name="stibp"/>
    <feature name="syscall"/>
    <feature name="tm"/>
    <feature name="tm2"/>
    <feature name="tsc"/>
    <feature name="tsc-deadline"/>
    <feature name="tsc_adjust"/>
    <feature name="vme"/>
    <feature name="vmx"/>
    <feature name="x2apic"/>
    <feature name="xgetbv1"/>
    <feature name="xsave"/>
    <feature name="xsavec"/>
    <feature name="xsaveopt"/>
    <feature name="xsaves"/>
    <feature name="xtpr"/>
  </cpu>

  nova.conf
  [libvirt]
  cpu_mode = host-model

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2023035/+subscriptions