← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1938765] Re: nova-lvm job constantly fails on Glance image upload

 

Reviewed:  https://review.opendev.org/c/openstack/nova/+/803322
Committed: https://opendev.org/openstack/nova/commit/dbf3e276ba5971f1f6483e04b84cb0fc403cd592
Submitter: "Zuul (22348)"
Branch:    master

commit dbf3e276ba5971f1f6483e04b84cb0fc403cd592
Author: Lee Yarwood <lyarwood@xxxxxxxxxx>
Date:   Tue Aug 3 10:05:42 2021 +0100

    zuul: Increase GLANCE_LIMIT_IMAGE_SIZE_TOTAL for nova-lvm
    
    As seen before in bug #1913451 the nova-lvm job creates large RAW
    snapshots during each run due to the nature of using LVM storage for the
    ephemeral instance disks.
    
    Id425aa546f1a5973bae8be9c017782d18f0b4a47 recently landed capping the
    per teneant image size total to 1000 MiB that is far too low for this
    job given the RAW snapshots it's creating. This in turn resulted in
    504 failures when n-cpu attempted to upload snapshots to g-api.
    
    This change simply increases the per tenant limit to 10 GiB that should
    be more than enough for all of the tests covered by the nova-lvm job.
    
    Closes-Bug: #1938765
    Change-Id: I824655387a10ac9b813c1b7b2399e25ed95f7fc3


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1938765

Title:
  nova-lvm job constantly fails on Glance image upload

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  The nova-lvm job seems to fail for every change [1] which prevents us
  to merge any change touching nova/virt/libvirt [2]

  All failures seem to relate to the same Tempest tests (6 of them)
  failing with the same problem : a Glance image upload issue as Glance
  API returns a HTTP502.

  
  Aug 01 03:50:12.036077 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR os_brick.initiator.linuxscsi [None req-8fa95fb0-15e8-4bf7-8314-0d2dac1c9b9c tempest-VolumesAdminNegativeTest-243853426 tempest-VolumesAdminNegativeTest-243853426-project] multipathd is not running: exit code None: oslo_concurrency.processutils.ProcessExecutionError: [Errno 2] No such file or directory

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server [None
  req-76a536ab-394f-4347-8dec-30842c1ec1d2 tempest-
  ListImageFiltersTestJSON-1499681503 tempest-
  ListImageFiltersTestJSON-1499681503-project] Exception during message
  handling: glanceclient.exc.HTTPBadGateway: HTTP 502 Bad Gateway: Bad
  Gateway: The proxy server received an invalid: response from an
  upstream server.: Apache/2.4.41 (Ubuntu) Server at 158.69.72.121 Port
  80

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server Traceback (most
  recent call last):

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2916, in snapshot

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  metadata['location'] = root_disk.direct_snapshot(

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 452, in
  direct_snapshot

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     raise
  NotImplementedError(_('direct_snapshot() is not implemented'))

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server NotImplementedError:
  direct_snapshot() is not implemented

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server During handling of
  the above exception, another exception occurred:

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server Traceback (most
  recent call last):

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py",
  line 165, in _process_incoming

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     res =
  self.dispatcher.dispatch(message)

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-
  packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     return
  self._do_dispatch(endpoint, method, ctxt, args)

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-
  packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     result =
  func(ctxt, **new_args)

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/exception_wrapper.py", line 71, in wrapped

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  _emit_versioned_exception_notification(

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  227, in __exit__

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  self.force_reraise()

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  200, in force_reraise

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     raise self.value

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/exception_wrapper.py", line 63, in wrapped

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     return f(self,
  context, *args, **kw)

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 183, in
  decorated_function

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  LOG.warning("Failed to revert task state for instance. "

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  227, in __exit__

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  self.force_reraise()

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  200, in force_reraise

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     raise self.value

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 154, in
  decorated_function

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     return
  function(self, context, *args, **kwargs)

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/utils.py", line 1433, in
  decorated_function

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     return
  function(self, context, *args, **kwargs)

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 211, in
  decorated_function

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  compute_utils.add_instance_fault_from_exc(context,

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  227, in __exit__

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  self.force_reraise()

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  200, in force_reraise

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     raise self.value

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 200, in
  decorated_function

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     return
  function(self, context, *args, **kwargs)

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 231, in
  decorated_function

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  compute_utils.delete_image(

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  227, in __exit__

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  self.force_reraise()

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  200, in force_reraise

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     raise self.value

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 227, in
  decorated_function

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     return
  function(self, context, image_id, instance,

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 3917, in
  snapshot_instance

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  self._snapshot_instance(context, image_id, instance,

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 3950, in
  _snapshot_instance

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  self.driver.snapshot(context, instance, image_id,

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2987, in snapshot

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  self._image_api.update(context,

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/image/glance.py", line 1240, in update

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     return
  session.update(context, image_id, image_info, data=data,

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/image/glance.py", line 690, in update

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  _reraise_translated_image_exception(image_id)

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/image/glance.py", line 1028, in
  _reraise_translated_image_exception

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     raise
  new_exc.with_traceback(exc_trace)

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/image/glance.py", line 688, in update

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     image =
  self._update_v2(context, sent_service_image_meta, data)

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/image/glance.py", line 707, in _update_v2

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     image =
  self._upload_data(context, image_id, data)

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/image/glance.py", line 583, in _upload_data

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  utils.tpool_execute(self._client.call,

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/utils.py", line 687, in tpool_execute

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  eventlet.tpool.execute(func, *args, **kwargs)

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 132,
  in execute

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     six.reraise(c, e,
  tb)

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/six.py", line 719, in reraise

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     raise value

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 86,
  in tworker

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     rv = meth(*args,
  **kwargs)

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/image/glance.py", line 190, in call

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     result =
  getattr(controller, method)(*args, **kwargs)

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/glanceclient/common/utils.py",
  line 628, in inner

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     return
  RequestIdProxy(wrapped(*args, **kwargs))

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/glanceclient/v2/images.py",
  line 308, in upload

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     resp, body =
  self.http_client.put(url, headers=hdrs, data=body)

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/keystoneauth1/adapter.py",
  line 404, in put

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     return
  self.request(url, 'PUT', **kwargs)

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/glanceclient/common/http.py",
  line 380, in request

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     return
  self._handle_response(resp)

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/glanceclient/common/http.py",
  line 120, in _handle_response

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     raise
  exc.from_response(resp, resp.content)

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  glanceclient.exc.HTTPBadGateway: HTTP 502 Bad Gateway: Bad Gateway:
  The proxy server received an invalid: response from an upstream
  server.: Apache/2.4.41 (Ubuntu) Server at 158.69.72.121 Port 80

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server


  
  [1] https://zuul.openstack.org/builds?job_name=nova-lvm&branch=master
  [2] https://zuul.openstack.org/job/nova-lvm

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1938765/+subscriptions



References