yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #86800
[Bug 1938765] Re: nova-lvm job constantly fails on Glance image upload
Reviewed: https://review.opendev.org/c/openstack/nova/+/803322
Committed: https://opendev.org/openstack/nova/commit/dbf3e276ba5971f1f6483e04b84cb0fc403cd592
Submitter: "Zuul (22348)"
Branch: master
commit dbf3e276ba5971f1f6483e04b84cb0fc403cd592
Author: Lee Yarwood <lyarwood@xxxxxxxxxx>
Date: Tue Aug 3 10:05:42 2021 +0100
zuul: Increase GLANCE_LIMIT_IMAGE_SIZE_TOTAL for nova-lvm
As seen before in bug #1913451 the nova-lvm job creates large RAW
snapshots during each run due to the nature of using LVM storage for the
ephemeral instance disks.
Id425aa546f1a5973bae8be9c017782d18f0b4a47 recently landed capping the
per teneant image size total to 1000 MiB that is far too low for this
job given the RAW snapshots it's creating. This in turn resulted in
504 failures when n-cpu attempted to upload snapshots to g-api.
This change simply increases the per tenant limit to 10 GiB that should
be more than enough for all of the tests covered by the nova-lvm job.
Closes-Bug: #1938765
Change-Id: I824655387a10ac9b813c1b7b2399e25ed95f7fc3
** Changed in: nova
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1938765
Title:
nova-lvm job constantly fails on Glance image upload
Status in OpenStack Compute (nova):
Fix Released
Bug description:
The nova-lvm job seems to fail for every change [1] which prevents us
to merge any change touching nova/virt/libvirt [2]
All failures seem to relate to the same Tempest tests (6 of them)
failing with the same problem : a Glance image upload issue as Glance
API returns a HTTP502.
Aug 01 03:50:12.036077 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR os_brick.initiator.linuxscsi [None req-8fa95fb0-15e8-4bf7-8314-0d2dac1c9b9c tempest-VolumesAdminNegativeTest-243853426 tempest-VolumesAdminNegativeTest-243853426-project] multipathd is not running: exit code None: oslo_concurrency.processutils.ProcessExecutionError: [Errno 2] No such file or directory
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server [None
req-76a536ab-394f-4347-8dec-30842c1ec1d2 tempest-
ListImageFiltersTestJSON-1499681503 tempest-
ListImageFiltersTestJSON-1499681503-project] Exception during message
handling: glanceclient.exc.HTTPBadGateway: HTTP 502 Bad Gateway: Bad
Gateway: The proxy server received an invalid: response from an
upstream server.: Apache/2.4.41 (Ubuntu) Server at 158.69.72.121 Port
80
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server Traceback (most
recent call last):
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 2916, in snapshot
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
metadata['location'] = root_disk.direct_snapshot(
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 452, in
direct_snapshot
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server raise
NotImplementedError(_('direct_snapshot() is not implemented'))
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server NotImplementedError:
direct_snapshot() is not implemented
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server During handling of
the above exception, another exception occurred:
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server Traceback (most
recent call last):
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py",
line 165, in _process_incoming
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-
packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-
packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/exception_wrapper.py", line 71, in wrapped
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
_emit_versioned_exception_notification(
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
227, in __exit__
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self.force_reraise()
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
200, in force_reraise
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server raise self.value
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/exception_wrapper.py", line 63, in wrapped
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return f(self,
context, *args, **kw)
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 183, in
decorated_function
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
LOG.warning("Failed to revert task state for instance. "
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
227, in __exit__
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self.force_reraise()
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
200, in force_reraise
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server raise self.value
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 154, in
decorated_function
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/utils.py", line 1433, in
decorated_function
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 211, in
decorated_function
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
compute_utils.add_instance_fault_from_exc(context,
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
227, in __exit__
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self.force_reraise()
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
200, in force_reraise
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server raise self.value
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 200, in
decorated_function
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 231, in
decorated_function
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
compute_utils.delete_image(
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
227, in __exit__
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self.force_reraise()
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
200, in force_reraise
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server raise self.value
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 227, in
decorated_function
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
function(self, context, image_id, instance,
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 3917, in
snapshot_instance
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self._snapshot_instance(context, image_id, instance,
Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 3950, in
_snapshot_instance
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self.driver.snapshot(context, instance, image_id,
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 2987, in snapshot
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
self._image_api.update(context,
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/image/glance.py", line 1240, in update
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
session.update(context, image_id, image_info, data=data,
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/image/glance.py", line 690, in update
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
_reraise_translated_image_exception(image_id)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/image/glance.py", line 1028, in
_reraise_translated_image_exception
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server raise
new_exc.with_traceback(exc_trace)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/image/glance.py", line 688, in update
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server image =
self._update_v2(context, sent_service_image_meta, data)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/image/glance.py", line 707, in _update_v2
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server image =
self._upload_data(context, image_id, data)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/image/glance.py", line 583, in _upload_data
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
utils.tpool_execute(self._client.call,
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/utils.py", line 687, in tpool_execute
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
eventlet.tpool.execute(func, *args, **kwargs)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 132,
in execute
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server six.reraise(c, e,
tb)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/six.py", line 719, in reraise
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server raise value
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 86,
in tworker
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server rv = meth(*args,
**kwargs)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/image/glance.py", line 190, in call
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server result =
getattr(controller, method)(*args, **kwargs)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/glanceclient/common/utils.py",
line 628, in inner
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
RequestIdProxy(wrapped(*args, **kwargs))
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/glanceclient/v2/images.py",
line 308, in upload
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server resp, body =
self.http_client.put(url, headers=hdrs, data=body)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/keystoneauth1/adapter.py",
line 404, in put
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
self.request(url, 'PUT', **kwargs)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/glanceclient/common/http.py",
line 380, in request
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server return
self._handle_response(resp)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/glanceclient/common/http.py",
line 120, in _handle_response
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server raise
exc.from_response(resp, resp.content)
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
glanceclient.exc.HTTPBadGateway: HTTP 502 Bad Gateway: Bad Gateway:
The proxy server received an invalid: response from an upstream
server.: Apache/2.4.41 (Ubuntu) Server at 158.69.72.121 Port 80
Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-
compute[106038]: ERROR oslo_messaging.rpc.server
[1] https://zuul.openstack.org/builds?job_name=nova-lvm&branch=master
[2] https://zuul.openstack.org/job/nova-lvm
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1938765/+subscriptions
References