yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1938765] [NEW] nova-lvm job constantly fails on Glance image upload

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sylvain Bauza <1938765@xxxxxxxxxxxxxxxxxx>
Date: Tue, 03 Aug 2021 08:46:07 -0000
Reply-to: Bug 1938765 <1938765@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

The nova-lvm job seems to fail for every change [1] which prevents us to
merge any change touching nova/virt/libvirt [2]

All failures seem to relate to the same Tempest tests (6 of them)
failing with the same problem : a Glance image upload issue as Glance
API returns a HTTP502.


Aug 01 03:50:12.036077 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR os_brick.initiator.linuxscsi [None req-8fa95fb0-15e8-4bf7-8314-0d2dac1c9b9c tempest-VolumesAdminNegativeTest-243853426 tempest-VolumesAdminNegativeTest-243853426-project] multipathd is not running: exit code None: oslo_concurrency.processutils.ProcessExecutionError: [Errno 2] No such file or directory

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server [None
req-76a536ab-394f-4347-8dec-30842c1ec1d2 tempest-
ListImageFiltersTestJSON-1499681503 tempest-
ListImageFiltersTestJSON-1499681503-project] Exception during message
handling: glanceclient.exc.HTTPBadGateway: HTTP 502 Bad Gateway: Bad
Gateway: The proxy server received an invalid: response from an upstream
server.: Apache/2.4.41 (Ubuntu) Server at 158.69.72.121 Port 80

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server Traceback (most recent
call last):

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 2916, in snapshot

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
metadata['location'] = root_disk.direct_snapshot(

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 452, in
direct_snapshot

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     raise
NotImplementedError(_('direct_snapshot() is not implemented'))

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server NotImplementedError:
direct_snapshot() is not implemented

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server During handling of the
above exception, another exception occurred:

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server Traceback (most recent
call last):

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py",
line 165, in _process_incoming

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     res =
self.dispatcher.dispatch(message)

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-
packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     return
self._do_dispatch(endpoint, method, ctxt, args)

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-
packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     result = func(ctxt,
**new_args)

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/exception_wrapper.py", line 71, in wrapped

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
_emit_versioned_exception_notification(

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
227, in __exit__

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self.force_reraise()

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
200, in force_reraise

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     raise self.value

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/exception_wrapper.py", line 63, in wrapped

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     return f(self,
context, *args, **kw)

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/compute/manager.py", line 183, in
decorated_function

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     LOG.warning("Failed
to revert task state for instance. "

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
227, in __exit__

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self.force_reraise()

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
200, in force_reraise

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     raise self.value

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/compute/manager.py", line 154, in
decorated_function

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     return
function(self, context, *args, **kwargs)

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/compute/utils.py", line 1433, in
decorated_function

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     return
function(self, context, *args, **kwargs)

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/compute/manager.py", line 211, in
decorated_function

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
compute_utils.add_instance_fault_from_exc(context,

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
227, in __exit__

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self.force_reraise()

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
200, in force_reraise

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     raise self.value

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/compute/manager.py", line 200, in
decorated_function

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     return
function(self, context, *args, **kwargs)

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/compute/manager.py", line 231, in
decorated_function

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
compute_utils.delete_image(

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
227, in __exit__

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self.force_reraise()

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
200, in force_reraise

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     raise self.value

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/compute/manager.py", line 227, in
decorated_function

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     return
function(self, context, image_id, instance,

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/compute/manager.py", line 3917, in
snapshot_instance

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self._snapshot_instance(context, image_id, instance,

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/compute/manager.py", line 3950, in
_snapshot_instance

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self.driver.snapshot(context, instance, image_id,

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 2987, in snapshot

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self._image_api.update(context,

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/image/glance.py", line 1240, in update

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     return
session.update(context, image_id, image_info, data=data,

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/image/glance.py", line 690, in update

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
_reraise_translated_image_exception(image_id)

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/image/glance.py", line 1028, in
_reraise_translated_image_exception

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     raise
new_exc.with_traceback(exc_trace)

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/image/glance.py", line 688, in update

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     image =
self._update_v2(context, sent_service_image_meta, data)

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/image/glance.py", line 707, in _update_v2

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     image =
self._upload_data(context, image_id, data)

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/image/glance.py", line 583, in _upload_data

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
utils.tpool_execute(self._client.call,

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/utils.py", line 687, in tpool_execute

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
eventlet.tpool.execute(func, *args, **kwargs)

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 132, in
execute

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     six.reraise(c, e,
tb)

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/six.py", line 719, in reraise

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     raise value

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 86, in
tworker

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     rv = meth(*args,
**kwargs)

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/image/glance.py", line 190, in call

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     result =
getattr(controller, method)(*args, **kwargs)

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/glanceclient/common/utils.py",
line 628, in inner

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     return
RequestIdProxy(wrapped(*args, **kwargs))

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/glanceclient/v2/images.py", line
308, in upload

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     resp, body =
self.http_client.put(url, headers=hdrs, data=body)

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/keystoneauth1/adapter.py", line
404, in put

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     return
self.request(url, 'PUT', **kwargs)

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/glanceclient/common/http.py",
line 380, in request

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     return
self._handle_response(resp)

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/glanceclient/common/http.py",
line 120, in _handle_response

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server     raise
exc.from_response(resp, resp.content)

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
glanceclient.exc.HTTPBadGateway: HTTP 502 Bad Gateway: Bad Gateway: The
proxy server received an invalid: response from an upstream server.:
Apache/2.4.41 (Ubuntu) Server at 158.69.72.121 Port 80

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server


[1] https://zuul.openstack.org/builds?job_name=nova-lvm&branch=master
[2] https://zuul.openstack.org/job/nova-lvm

** Affects: nova
     Importance: Critical
         Status: Confirmed


** Tags: gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1938765

Title:
  nova-lvm job constantly fails on Glance image upload

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  The nova-lvm job seems to fail for every change [1] which prevents us
  to merge any change touching nova/virt/libvirt [2]

  All failures seem to relate to the same Tempest tests (6 of them)
  failing with the same problem : a Glance image upload issue as Glance
  API returns a HTTP502.

  
  Aug 01 03:50:12.036077 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR os_brick.initiator.linuxscsi [None req-8fa95fb0-15e8-4bf7-8314-0d2dac1c9b9c tempest-VolumesAdminNegativeTest-243853426 tempest-VolumesAdminNegativeTest-243853426-project] multipathd is not running: exit code None: oslo_concurrency.processutils.ProcessExecutionError: [Errno 2] No such file or directory

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server [None
  req-76a536ab-394f-4347-8dec-30842c1ec1d2 tempest-
  ListImageFiltersTestJSON-1499681503 tempest-
  ListImageFiltersTestJSON-1499681503-project] Exception during message
  handling: glanceclient.exc.HTTPBadGateway: HTTP 502 Bad Gateway: Bad
  Gateway: The proxy server received an invalid: response from an
  upstream server.: Apache/2.4.41 (Ubuntu) Server at 158.69.72.121 Port
  80

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server Traceback (most
  recent call last):

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2916, in snapshot

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  metadata['location'] = root_disk.direct_snapshot(

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 452, in
  direct_snapshot

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     raise
  NotImplementedError(_('direct_snapshot() is not implemented'))

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server NotImplementedError:
  direct_snapshot() is not implemented

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server During handling of
  the above exception, another exception occurred:

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server Traceback (most
  recent call last):

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py",
  line 165, in _process_incoming

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     res =
  self.dispatcher.dispatch(message)

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-
  packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     return
  self._do_dispatch(endpoint, method, ctxt, args)

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-
  packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     result =
  func(ctxt, **new_args)

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/exception_wrapper.py", line 71, in wrapped

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  _emit_versioned_exception_notification(

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  227, in __exit__

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  self.force_reraise()

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  200, in force_reraise

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     raise self.value

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/exception_wrapper.py", line 63, in wrapped

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     return f(self,
  context, *args, **kw)

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 183, in
  decorated_function

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  LOG.warning("Failed to revert task state for instance. "

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  227, in __exit__

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  self.force_reraise()

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  200, in force_reraise

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     raise self.value

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 154, in
  decorated_function

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     return
  function(self, context, *args, **kwargs)

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/utils.py", line 1433, in
  decorated_function

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     return
  function(self, context, *args, **kwargs)

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 211, in
  decorated_function

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  compute_utils.add_instance_fault_from_exc(context,

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  227, in __exit__

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  self.force_reraise()

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  200, in force_reraise

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     raise self.value

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 200, in
  decorated_function

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     return
  function(self, context, *args, **kwargs)

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 231, in
  decorated_function

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  compute_utils.delete_image(

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  227, in __exit__

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  self.force_reraise()

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  200, in force_reraise

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     raise self.value

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 227, in
  decorated_function

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     return
  function(self, context, image_id, instance,

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 3917, in
  snapshot_instance

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  self._snapshot_instance(context, image_id, instance,

  Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 3950, in
  _snapshot_instance

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  self.driver.snapshot(context, instance, image_id,

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2987, in snapshot

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  self._image_api.update(context,

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/image/glance.py", line 1240, in update

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     return
  session.update(context, image_id, image_info, data=data,

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/image/glance.py", line 690, in update

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  _reraise_translated_image_exception(image_id)

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/image/glance.py", line 1028, in
  _reraise_translated_image_exception

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     raise
  new_exc.with_traceback(exc_trace)

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/image/glance.py", line 688, in update

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     image =
  self._update_v2(context, sent_service_image_meta, data)

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/image/glance.py", line 707, in _update_v2

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     image =
  self._upload_data(context, image_id, data)

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/image/glance.py", line 583, in _upload_data

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  utils.tpool_execute(self._client.call,

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/utils.py", line 687, in tpool_execute

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  eventlet.tpool.execute(func, *args, **kwargs)

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 132,
  in execute

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     six.reraise(c, e,
  tb)

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/six.py", line 719, in reraise

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     raise value

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 86,
  in tworker

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     rv = meth(*args,
  **kwargs)

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/image/glance.py", line 190, in call

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     result =
  getattr(controller, method)(*args, **kwargs)

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/glanceclient/common/utils.py",
  line 628, in inner

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     return
  RequestIdProxy(wrapped(*args, **kwargs))

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/glanceclient/v2/images.py",
  line 308, in upload

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     resp, body =
  self.http_client.put(url, headers=hdrs, data=body)

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/keystoneauth1/adapter.py",
  line 404, in put

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     return
  self.request(url, 'PUT', **kwargs)

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/glanceclient/common/http.py",
  line 380, in request

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     return
  self._handle_response(resp)

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/glanceclient/common/http.py",
  line 120, in _handle_response

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server     raise
  exc.from_response(resp, resp.content)

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server
  glanceclient.exc.HTTPBadGateway: HTTP 502 Bad Gateway: Bad Gateway:
  The proxy server received an invalid: response from an upstream
  server.: Apache/2.4.41 (Ubuntu) Server at 158.69.72.121 Port 80

  Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
  compute[106038]: ERROR oslo_messaging.rpc.server


  
  [1] https://zuul.openstack.org/builds?job_name=nova-lvm&branch=master
  [2] https://zuul.openstack.org/job/nova-lvm

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1938765/+subscriptions
Follow ups

[Bug 1938765] Re: nova-lvm job constantly fails on Glance image upload
From: OpenStack Infra, 2021-08-04