yahoo-eng-team team mailing list archive

[Stephen Finucane <1888414@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Attempting to create a snapshot of a shutdown instance fails. It seems
nova assumes the instance exists and is running when attempting to
create the snapshot.

# Steps to reproduce

  $ openstack server create \
      --os-compute-api-version=2.latest --flavor m1.tiny --image cirros-0.5.1-x86_64-disk \
      --nic none --wait test.server
  $ openstack server stop test.server
  $ openstack server image create test.server

# Expected result

A snapshot of the instance root disk should be created.

# Actual result

The snapshot is not created. Attempts to resume the instance fail:

  $ openstack server start test.server
  Cannot 'start' instance aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda while it is in task_state image_pending_upload (HTTP 409) (Request-ID: req-39d4bd58-366b-4b93-8d7d-72a487183088)

# Additional details

I see the following in the logs:

  nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-0b7dfe74-d465-4c2b-90e3-54ed26ea4244 demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Skipping quiescing instance: QEMU guest agent is not enabled.
  nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-0b7dfe74-d465-4c2b-90e3-54ed26ea4244 demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Instance instance-0000000c disappeared while taking snapshot of it: [Error Code 42] Domain not found: no domain with matching uuid 'aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda' (instance-0000000c)
  nova-compute[20898]: DEBUG nova.compute.manager [None req-0b7dfe74-d465-4c2b-90e3-54ed26ea4244 demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Instance disappeared during snapshot {{(pid=20898) _snapshot_instance /opt/stack/nova/nova/compute/manager.py:3874}}

Compare with logs from snapshot of a running guest:

  nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Skipping quiescing instance: QEMU guest agent is not enabled.
  nova-compute[20898]: DEBUG nova.privsep.utils [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] Path '/opt/stack/data/nova/instances' supports direct I/O {{(pid=20898) supports_direct_io /opt/stack/nova/nova/privsep/utils.py:64}}
  nova-compute[20898]: DEBUG oslo_concurrency.processutils [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] Running cmd (subprocess): qemu-img convert -t none -O qcow2 -f qcow2 /opt/stack/data/nova/instances/snapshots/tmpal0gmbcx/d64bd94655da448495d69b274ca14581.delta /opt/stack/data/nova/instances/snapshots/tmpal0gmbcx/d64bd94655da448495d69b274ca14581 {{(pid=20898) execute /usr/local/lib/python3.6/dist-packages/oslo_concurrency/processutils.py:371}}
  nova-compute[20898]: DEBUG oslo_concurrency.processutils [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] CMD "qemu-img convert -t none -O qcow2 -f qcow2 /opt/stack/data/nova/instances/snapshots/tmpal0gmbcx/d64bd94655da448495d69b274ca14581.delta /opt/stack/data/nova/instances/snapshots/tmpal0gmbcx/d64bd94655da448495d69b274ca14581" returned: 0 in 0.403s {{(pid=20898) execute /usr/local/lib/python3.6/dist-packages/oslo_concurrency/processutils.py:408}}
  nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Snapshot extracted, beginning image upload
  nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Snapshot image upload complete
  nova-compute[20898]: INFO nova.compute.manager [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Took 2.44 seconds to snapshot the instance on the hypervisor.

We see the same issue if the issues is suspended ('openstack server
suspend'). There are no issues if the instance is paused, however
('openstack server pause') but

** Affects: nova
     Importance: Medium
         Status: Confirmed


** Tags: libvirt snapshot

** Tags added: libvirt snapshot

** Changed in: nova
       Status: New => Confirmed

** Changed in: nova
   Importance: Undecided => Medium

** Description changed:

  Attempting to create a snapshot of a shutdown instance fails. It seems
  nova assumes the instance exists and is running when attempting to
  create the snapshot.
  
  # Steps to reproduce
  
-   $ openstack server create \
-       --os-compute-api-version=2.latest --flavor m1.tiny --image cirros-0.5.1-x86_64-disk \
-       --nic none --wait test.server
-   $ openstack server stop test.server
-   $ openstack server image create test.server
+   $ openstack server create \
+       --os-compute-api-version=2.latest --flavor m1.tiny --image cirros-0.5.1-x86_64-disk \
+       --nic none --wait test.server
+   $ openstack server stop test.server
+   $ openstack server image create test.server
  
  # Expected result
  
  A snapshot of the instance root disk should be created.
  
  # Actual result
  
  The snapshot is not created. Attempts to resume the instance fail:
  
-   $ openstack server start test.server
-   Cannot 'start' instance aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda while it is in task_state image_pending_upload (HTTP 409) (Request-ID: req-39d4bd58-366b-4b93-8d7d-72a487183088)
+   $ openstack server start test.server
+   Cannot 'start' instance aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda while it is in task_state image_pending_upload (HTTP 409) (Request-ID: req-39d4bd58-366b-4b93-8d7d-72a487183088)
  
  # Additional details
  
  I see the following in the logs:
  
-   nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-0b7dfe74-d465-4c2b-90e3-54ed26ea4244 demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Skipping quiescing instance: QEMU guest agent is not enabled.
-   nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-0b7dfe74-d465-4c2b-90e3-54ed26ea4244 demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Instance instance-0000000c disappeared while taking snapshot of it: [Error Code 42] Domain not found: no domain with matching uuid 'aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda' (instance-0000000c)
-   nova-compute[20898]: DEBUG nova.compute.manager [None req-0b7dfe74-d465-4c2b-90e3-54ed26ea4244 demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Instance disappeared during snapshot {{(pid=20898) _snapshot_instance /opt/stack/nova/nova/compute/manager.py:3874}}
+   nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-0b7dfe74-d465-4c2b-90e3-54ed26ea4244 demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Skipping quiescing instance: QEMU guest agent is not enabled.
+   nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-0b7dfe74-d465-4c2b-90e3-54ed26ea4244 demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Instance instance-0000000c disappeared while taking snapshot of it: [Error Code 42] Domain not found: no domain with matching uuid 'aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda' (instance-0000000c)
+   nova-compute[20898]: DEBUG nova.compute.manager [None req-0b7dfe74-d465-4c2b-90e3-54ed26ea4244 demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Instance disappeared during snapshot {{(pid=20898) _snapshot_instance /opt/stack/nova/nova/compute/manager.py:3874}}
  
  Compare with logs from snapshot of a running guest:
  
-   nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Skipping quiescing instance: QEMU guest agent is not enabled.
-   nova-compute[20898]: DEBUG nova.privsep.utils [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] Path '/opt/stack/data/nova/instances' supports direct I/O {{(pid=20898) supports_direct_io /opt/stack/nova/nova/privsep/utils.py:64}}
-   nova-compute[20898]: DEBUG oslo_concurrency.processutils [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] Running cmd (subprocess): qemu-img convert -t none -O qcow2 -f qcow2 /opt/stack/data/nova/instances/snapshots/tmpal0gmbcx/d64bd94655da448495d69b274ca14581.delta /opt/stack/data/nova/instances/snapshots/tmpal0gmbcx/d64bd94655da448495d69b274ca14581 {{(pid=20898) execute /usr/local/lib/python3.6/dist-packages/oslo_concurrency/processutils.py:371}}
-   nova-compute[20898]: DEBUG oslo_concurrency.processutils [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] CMD "qemu-img convert -t none -O qcow2 -f qcow2 /opt/stack/data/nova/instances/snapshots/tmpal0gmbcx/d64bd94655da448495d69b274ca14581.delta /opt/stack/data/nova/instances/snapshots/tmpal0gmbcx/d64bd94655da448495d69b274ca14581" returned: 0 in 0.403s {{(pid=20898) execute /usr/local/lib/python3.6/dist-packages/oslo_concurrency/processutils.py:408}}
-   nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Snapshot extracted, beginning image upload
-   nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Snapshot image upload complete
-   nova-compute[20898]: INFO nova.compute.manager [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Took 2.44 seconds to snapshot the instance on the hypervisor.
+   nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Skipping quiescing instance: QEMU guest agent is not enabled.
+   nova-compute[20898]: DEBUG nova.privsep.utils [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] Path '/opt/stack/data/nova/instances' supports direct I/O {{(pid=20898) supports_direct_io /opt/stack/nova/nova/privsep/utils.py:64}}
+   nova-compute[20898]: DEBUG oslo_concurrency.processutils [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] Running cmd (subprocess): qemu-img convert -t none -O qcow2 -f qcow2 /opt/stack/data/nova/instances/snapshots/tmpal0gmbcx/d64bd94655da448495d69b274ca14581.delta /opt/stack/data/nova/instances/snapshots/tmpal0gmbcx/d64bd94655da448495d69b274ca14581 {{(pid=20898) execute /usr/local/lib/python3.6/dist-packages/oslo_concurrency/processutils.py:371}}
+   nova-compute[20898]: DEBUG oslo_concurrency.processutils [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] CMD "qemu-img convert -t none -O qcow2 -f qcow2 /opt/stack/data/nova/instances/snapshots/tmpal0gmbcx/d64bd94655da448495d69b274ca14581.delta /opt/stack/data/nova/instances/snapshots/tmpal0gmbcx/d64bd94655da448495d69b274ca14581" returned: 0 in 0.403s {{(pid=20898) execute /usr/local/lib/python3.6/dist-packages/oslo_concurrency/processutils.py:408}}
+   nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Snapshot extracted, beginning image upload
+   nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Snapshot image upload complete
+   nova-compute[20898]: INFO nova.compute.manager [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Took 2.44 seconds to snapshot the instance on the hypervisor.
+ 
+ Things also work if the instance is paused ('openstack server pause')

** Summary changed:

- Snapshot of stopped instance fails
+ Snapshot of stopped, suspended instance fails

** Description changed:

  Attempting to create a snapshot of a shutdown instance fails. It seems
  nova assumes the instance exists and is running when attempting to
  create the snapshot.
  
  # Steps to reproduce
  
    $ openstack server create \
        --os-compute-api-version=2.latest --flavor m1.tiny --image cirros-0.5.1-x86_64-disk \
        --nic none --wait test.server
    $ openstack server stop test.server
    $ openstack server image create test.server
  
  # Expected result
  
  A snapshot of the instance root disk should be created.
  
  # Actual result
  
  The snapshot is not created. Attempts to resume the instance fail:
  
    $ openstack server start test.server
    Cannot 'start' instance aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda while it is in task_state image_pending_upload (HTTP 409) (Request-ID: req-39d4bd58-366b-4b93-8d7d-72a487183088)
  
  # Additional details
  
  I see the following in the logs:
  
    nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-0b7dfe74-d465-4c2b-90e3-54ed26ea4244 demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Skipping quiescing instance: QEMU guest agent is not enabled.
    nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-0b7dfe74-d465-4c2b-90e3-54ed26ea4244 demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Instance instance-0000000c disappeared while taking snapshot of it: [Error Code 42] Domain not found: no domain with matching uuid 'aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda' (instance-0000000c)
    nova-compute[20898]: DEBUG nova.compute.manager [None req-0b7dfe74-d465-4c2b-90e3-54ed26ea4244 demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Instance disappeared during snapshot {{(pid=20898) _snapshot_instance /opt/stack/nova/nova/compute/manager.py:3874}}
  
  Compare with logs from snapshot of a running guest:
  
    nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Skipping quiescing instance: QEMU guest agent is not enabled.
    nova-compute[20898]: DEBUG nova.privsep.utils [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] Path '/opt/stack/data/nova/instances' supports direct I/O {{(pid=20898) supports_direct_io /opt/stack/nova/nova/privsep/utils.py:64}}
    nova-compute[20898]: DEBUG oslo_concurrency.processutils [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] Running cmd (subprocess): qemu-img convert -t none -O qcow2 -f qcow2 /opt/stack/data/nova/instances/snapshots/tmpal0gmbcx/d64bd94655da448495d69b274ca14581.delta /opt/stack/data/nova/instances/snapshots/tmpal0gmbcx/d64bd94655da448495d69b274ca14581 {{(pid=20898) execute /usr/local/lib/python3.6/dist-packages/oslo_concurrency/processutils.py:371}}
    nova-compute[20898]: DEBUG oslo_concurrency.processutils [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] CMD "qemu-img convert -t none -O qcow2 -f qcow2 /opt/stack/data/nova/instances/snapshots/tmpal0gmbcx/d64bd94655da448495d69b274ca14581.delta /opt/stack/data/nova/instances/snapshots/tmpal0gmbcx/d64bd94655da448495d69b274ca14581" returned: 0 in 0.403s {{(pid=20898) execute /usr/local/lib/python3.6/dist-packages/oslo_concurrency/processutils.py:408}}
    nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Snapshot extracted, beginning image upload
    nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Snapshot image upload complete
    nova-compute[20898]: INFO nova.compute.manager [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Took 2.44 seconds to snapshot the instance on the hypervisor.
  
- Things also work if the instance is paused ('openstack server pause')
+ We see the same issue if the issues is suspended ('openstack server
+ suspend'). There are no issues if the instance is paused, however
+ ('openstack server pause') but

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1888414

Title:
  Snapshot of stopped, suspended instance fails

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  Attempting to create a snapshot of a shutdown instance fails. It seems
  nova assumes the instance exists and is running when attempting to
  create the snapshot.

  # Steps to reproduce

    $ openstack server create \
        --os-compute-api-version=2.latest --flavor m1.tiny --image cirros-0.5.1-x86_64-disk \
        --nic none --wait test.server
    $ openstack server stop test.server
    $ openstack server image create test.server

  # Expected result

  A snapshot of the instance root disk should be created.

  # Actual result

  The snapshot is not created. Attempts to resume the instance fail:

    $ openstack server start test.server
    Cannot 'start' instance aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda while it is in task_state image_pending_upload (HTTP 409) (Request-ID: req-39d4bd58-366b-4b93-8d7d-72a487183088)

  # Additional details

  I see the following in the logs:

    nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-0b7dfe74-d465-4c2b-90e3-54ed26ea4244 demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Skipping quiescing instance: QEMU guest agent is not enabled.
    nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-0b7dfe74-d465-4c2b-90e3-54ed26ea4244 demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Instance instance-0000000c disappeared while taking snapshot of it: [Error Code 42] Domain not found: no domain with matching uuid 'aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda' (instance-0000000c)
    nova-compute[20898]: DEBUG nova.compute.manager [None req-0b7dfe74-d465-4c2b-90e3-54ed26ea4244 demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Instance disappeared during snapshot {{(pid=20898) _snapshot_instance /opt/stack/nova/nova/compute/manager.py:3874}}

  Compare with logs from snapshot of a running guest:

    nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Skipping quiescing instance: QEMU guest agent is not enabled.
    nova-compute[20898]: DEBUG nova.privsep.utils [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] Path '/opt/stack/data/nova/instances' supports direct I/O {{(pid=20898) supports_direct_io /opt/stack/nova/nova/privsep/utils.py:64}}
    nova-compute[20898]: DEBUG oslo_concurrency.processutils [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] Running cmd (subprocess): qemu-img convert -t none -O qcow2 -f qcow2 /opt/stack/data/nova/instances/snapshots/tmpal0gmbcx/d64bd94655da448495d69b274ca14581.delta /opt/stack/data/nova/instances/snapshots/tmpal0gmbcx/d64bd94655da448495d69b274ca14581 {{(pid=20898) execute /usr/local/lib/python3.6/dist-packages/oslo_concurrency/processutils.py:371}}
    nova-compute[20898]: DEBUG oslo_concurrency.processutils [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] CMD "qemu-img convert -t none -O qcow2 -f qcow2 /opt/stack/data/nova/instances/snapshots/tmpal0gmbcx/d64bd94655da448495d69b274ca14581.delta /opt/stack/data/nova/instances/snapshots/tmpal0gmbcx/d64bd94655da448495d69b274ca14581" returned: 0 in 0.403s {{(pid=20898) execute /usr/local/lib/python3.6/dist-packages/oslo_concurrency/processutils.py:408}}
    nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Snapshot extracted, beginning image upload
    nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Snapshot image upload complete
    nova-compute[20898]: INFO nova.compute.manager [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Took 2.44 seconds to snapshot the instance on the hypervisor.

  We see the same issue if the issues is suspended ('openstack server
  suspend'). There are no issues if the instance is paused, however
  ('openstack server pause') but

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1888414/+subscriptions