yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #85687
[Bug 1921893] [NEW] Snapshots fail: Failed to add object to Swift
Public bug reported:
I'm using OpenStack Victoria.
When I try to take snapshots of instances above a certain size, they
disappear during saving state and glance shows the following error:
Mar 30 09:15:14 infra1-glance-container-99614ac2 glance-wsgi-api[96]: 2021-03-30 09:15:14.306 96 INFO glance.api.v2.image_data [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Unable to create trust: no such option collect_timing in group [keystone_authtoken] Use the existing user token.
Mar 30 09:16:10 infra1-glance-container-99614ac2 uwsgi[96]: Tue Mar 30 09:16:10 2021 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2/images/c83f6c4d-0083-4600-95f5-d21b4623af47/file (ip 192.168.110.214) !!!
Mar 30 09:16:11 infra1-glance-container-99614ac2 glance-wsgi-api[96]: 2021-03-30 09:16:11.427 96 ERROR glance_store._drivers.swift.store [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Error during chunked upload to backend, deleting stale chunks.: swiftclient.exceptions.ClientException: put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to reset contents for reupload.
Mar 30 09:16:11 infra1-glance-container-99614ac2 glance-wsgi-api[96]: 2021-03-30 09:16:11.494 96 ERROR glance_store._drivers.swift.store [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to add object to Swift.
Got error from Swift: put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to reset contents for reupload..: swiftclient.exceptions.ClientException: put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to reset contents for reupload.
Mar 30 09:16:11 infra1-glance-container-99614ac2 glance-wsgi-api[96]: 2021-03-30 09:16:11.497 96 ERROR glance.api.v2.image_data [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to upload image data due to internal error: glance_store.exceptions.BackendException: Failed to add object to Swift.
Mar 30 09:16:11 infra1-glance-container-99614ac2 glance-wsgi-api[96]: 2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Caught error: Failed to add object to Swift.
Got error from Swift: put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to reset contents for reupload..: glance_store.exceptions.BackendException: Failed to add object to Swift.
Got error from Swift: put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to reset contents for reupload..
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi Traceback (most recent call last):
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1014, in add
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self._delete_stale_chunks(
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self.force_reraise()
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi raise value
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1003, in add
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi manager.get_connection().put_object(
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1960, in put_object
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return self._retry(reset_func, put_object, container, obj, contents,
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1843, in _retry
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi reset_func(func, *args, **kwargs)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1940, in _default_reset
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi raise ClientException('put_object(%r, %r, ...) failure and no '
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi swiftclient.exceptions.ClientException: put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to reset contents for reupload.
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi During handling of the above exception, another exception occurred:
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi Traceback (most recent call last):
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1347, in __call__
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action,
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1391, in dispatch
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return method(*args, **kwargs)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/utils.py", line 416, in wrapped
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return func(self, req, *args, **kwargs)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 298, in upload
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self._restore(image_repo, image)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self.force_reraise()
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi raise value
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 163, in upload
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi image.set_data(data, size, backend=backend)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/domain/proxy.py", line 208, in set_data
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self.base.set_data(data, size, backend=backend, set_active=set_active)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 501, in set_data
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self.force_reraise()
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi raise value
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 447, in set_data
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self.repo.set_data(data, size, backend=backend,
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/policy.py", line 198, in set_data
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/quota/__init__.py", line 318, in set_data
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self.image.set_data(data, size=size, backend=backend,
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 567, in set_data
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 458, in _upload_to_store
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi multihash, loc_meta) = self.store_api.add_with_multihash(
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 398, in add_with_multihash
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return store_add_to_backend_with_multihash(
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 480, in store_add_to_backend_with_multihash
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi (location, size, checksum, multihash, metadata) = store.add(
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/driver.py", line 279, in add_adapter
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/capabilities.py", line 176, in op_checker
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1082, in add
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi raise glance_store.BackendException(msg)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi glance_store.exceptions.BackendException: Failed to add object to Swift.
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi Got error from Swift: put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to reset contents for reupload..
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
I don't see errors in the swift logs:
Mar 30 09:15:14 infra1-swift-proxy-container-27169fa7 proxy-server[1068]: - - 30/Mar/2021/09/15/14 HEAD /v1/AUTH_024cc551782f41e395d3c9f13582ef7d%3Fformat%3Djson HTTP/1.0 204 - Swift - - - - txf319964369254af4974a7-006062ec22 - 0.0135 RL - 1617095714.876815081 1617095714.890305758 -
Mar 30 09:15:14 infra1-swift-proxy-container-27169fa7 proxy-server[1068]: - - 30/Mar/2021/09/15/14 HEAD /v1/AUTH_024cc551782f41e395d3c9f13582ef7d%3Fformat%3Djson HTTP/1.0 204 - Swift - - - - txf319964369254af4974a7-006062ec22 - 0.0131 CS - 1617095714.892540216 1617095714.905595303 -
Mar 30 09:15:14 infra1-swift-proxy-container-27169fa7 proxy-server[1068]: - - 30/Mar/2021/09/15/14 HEAD /v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images%3Fformat%3Djson%26states%3Dlisting HTTP/1.0 204 - Swift - - - - txf319964369254af4974a7-006062ec22 - 0.0127 CS - 1617095714.907815695 1617095714.920493126 0
Mar 30 09:15:15 infra1-swift-proxy-container-27169fa7 proxy-server[1068]: 192.168.110.106 192.168.110.211 30/Mar/2021/09/15/15 HEAD /v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images%3Fformat%3Djson%26states%3Dlisting HTTP/1.0 204 - python-swiftclient-3.10.1 gAAAAABgYuwin4R-... - - - txf319964369254af4974a7-006062ec22 - 0.2779 - - 1617095714.874005318 1617095715.151889324 0
Mar 30 09:15:17 infra1-swift-proxy-container-27169fa7 proxy-server[1071]: 192.168.110.106 192.168.110.211 30/Mar/2021/09/15/17 PUT /v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001 HTTP/1.0 201 - python-swiftclient-3.10.1 gAAAAABgYuwin4R-... 204800000 - - txc05fff07e3384a68a7fa9-006062ec23 - 2.8218 - - 1617095715.163244247 1617095717.985031843 0
Mar 30 09:16:10 infra1-swift-proxy-container-27169fa7 proxy-server[1067]: Client disconnected without sending last chunk (txn: tx49b0efe394364bf1975bb-006062ec25) (client_ip: 192.168.110.106)
Mar 30 09:16:10 infra1-swift-proxy-container-27169fa7 proxy-server[1067]: 192.168.110.106 192.168.110.211 30/Mar/2021/09/16/10 PUT /v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00002 HTTP/1.0 499 - python-swiftclient-3.10.1 gAAAAABgYuwin4R-... 204800000 89 - tx49b0efe394364bf1975bb-006062ec25 - 52.2224 - - 1617095717.995234489 1617095770.217619896 0
Mar 30 09:16:11 infra1-swift-proxy-container-27169fa7 proxy-server[1080]: 192.168.110.106 192.168.110.211 30/Mar/2021/09/16/11 DELETE /v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001 HTTP/1.0 204 - python-swiftclient-3.10.1 gAAAAABgYuwin4R-... - - - tx98a9247419a2445fafcbf-006062ec5b - 0.0486 - - 1617095771.441193104 1617095771.489775896 0
and
Mar 30 09:15:17 bc1bl12 object-server[694851]: 10.0.3.211 - - [30/Mar/2021:09:15:17 +0000] "PUT /os-objects/245/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001" 201 - "PUT http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001"; "txc05fff07e3384a68a7fa9-006062ec23" "proxy-server 1071" 2.8030 "-" 694851 0
Mar 30 09:16:10 bc1bl12 object-server[694850]: 10.0.3.211 - - [30/Mar/2021:09:16:10 +0000] "PUT /os-objects/43/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00002" 499 89 "PUT http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00002"; "tx49b0efe394364bf1975bb-006062ec25" "proxy-server 1067" 52.1551 "-" 694850 0
Mar 30 09:16:11 bc1bl12 object-server[694854]: 10.0.3.211 - - [30/Mar/2021:09:16:11 +0000] "DELETE /os-objects/245/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001" 204 - "DELETE http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001"; "tx98a9247419a2445fafcbf-006062ec5b" "proxy-server 1080" 0.0321 "-" 694854 0
nova-compute shows the following:
Mar 30 09:13:24 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:24.975 650019 INFO nova.compute.manager [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] instance snapshotting
Mar 30 09:13:25 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:25.038 650019 INFO nova.virt.libvirt.driver [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] Beginning live snapshot process
Mar 30 09:13:25 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:25.527 650019 INFO nova.virt.libvirt.driver [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] Skipping quiescing instance: QEMU guest agent is not enabled.
Mar 30 09:13:56 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:56.384 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] image c27e2055-8c3c-49d7-9c12-44999b1e7e0f at (/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478): checking
Mar 30 09:13:56 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:56.386 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] image e73512b8-7099-4691-aa48-966aa35b59ff at (/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3): checking
Mar 30 09:13:56 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:56.983 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] Active base files: /var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478 /var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3
Mar 30 09:14:57 bc1bl14 nova-compute[650019]: 2021-03-30 09:14:57.390 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] image c27e2055-8c3c-49d7-9c12-44999b1e7e0f at (/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478): checking
Mar 30 09:14:57 bc1bl14 nova-compute[650019]: 2021-03-30 09:14:57.391 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] image e73512b8-7099-4691-aa48-966aa35b59ff at (/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3): checking
Mar 30 09:14:58 bc1bl14 nova-compute[650019]: 2021-03-30 09:14:58.006 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] Active base files: /var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478 /var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3
Mar 30 09:15:13 bc1bl14 nova-compute[650019]: 2021-03-30 09:15:13.861 650019 INFO nova.virt.libvirt.driver [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] Snapshot extracted, beginning image upload
Mar 30 09:15:58 bc1bl14 nova-compute[650019]: 2021-03-30 09:15:58.395 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] image c27e2055-8c3c-49d7-9c12-44999b1e7e0f at (/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478): checking
Mar 30 09:15:58 bc1bl14 nova-compute[650019]: 2021-03-30 09:15:58.397 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] image e73512b8-7099-4691-aa48-966aa35b59ff at (/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3): checking
Mar 30 09:15:58 bc1bl14 nova-compute[650019]: 2021-03-30 09:15:58.997 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] Active base files: /var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478 /var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3
Mar 30 09:16:12 bc1bl14 nova-compute[650019]: 2021-03-30 09:16:12.530 650019 INFO nova.compute.manager [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] Successfully reverted task state from image_uploading on failure for instance.
Mar 30 09:16:12 bc1bl14 nova-compute[650019]: 2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Exception during message handling: glanceclient.exc.HTTPInternalServerError: HTTP 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation.
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 2478, in snapshot
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server metadata['location'] = root_disk.direct_snapshot(
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/imagebackend.py", line 452, in direct_snapshot
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server raise NotImplementedError(_('direct_snapshot() is not implemented'))
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server NotImplementedError: direct_snapshot() is not implemented
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/exception_wrapper.py", line 76, in wrapped
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server _emit_exception_notification(
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server raise value
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/exception_wrapper.py", line 69, in wrapped
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 188, in decorated_function
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server LOG.warning("Failed to revert task state for instance. "
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server raise value
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 159, in decorated_function
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/utils.py", line 1434, in decorated_function
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 216, in decorated_function
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server compute_utils.add_instance_fault_from_exc(context,
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server raise value
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 205, in decorated_function
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 236, in decorated_function
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server compute_utils.delete_image(
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server raise value
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 232, in decorated_function
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return function(self, context, image_id, instance,
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 3908, in snapshot_instance
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server self._snapshot_instance(context, image_id, instance,
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 3941, in _snapshot_instance
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server self.driver.snapshot(context, instance, image_id,
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 2549, in snapshot
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server self._image_api.update(context,
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 1247, in update
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return session.update(context, image_id, image_info, data=data,
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 696, in update
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server _reraise_translated_image_exception(image_id)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 1035, in _reraise_translated_image_exception
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server raise new_exc.with_traceback(exc_trace)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 694, in update
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server image = self._update_v2(context, sent_service_image_meta, data)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 713, in _update_v2
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server image = self._upload_data(context, image_id, data)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 589, in _upload_data
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server utils.tpool_execute(self._client.call,
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/utils.py", line 694, in tpool_execute
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server eventlet.tpool.execute(func, *args, **kwargs)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/eventlet/tpool.py", line 129, in execute
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server six.reraise(c, e, tb)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server raise value
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/eventlet/tpool.py", line 83, in tworker
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 192, in call
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server result = getattr(controller, method)(*args, **kwargs)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/utils.py", line 600, in inner
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return RequestIdProxy(wrapped(*args, **kwargs))
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/v2/images.py", line 289, in upload
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server resp, body = self.http_client.put(url, headers=hdrs, data=body)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/keystoneauth1/adapter.py", line 404, in put
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return self.request(url, 'PUT', **kwargs)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/http.py", line 380, in request
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return self._handle_response(resp)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/http.py", line 120, in _handle_response
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server raise exc.from_response(resp, resp.content)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server glanceclient.exc.HTTPInternalServerError: HTTP 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation.
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server
Mar 30 09:16:59 bc1bl14 nova-compute[650019]: 2021-03-30 09:16:59.391 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] image c27e2055-8c3c-49d7-9c12-44999b1e7e0f at (/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478): checking
Mar 30 09:16:59 bc1bl14 nova-compute[650019]: 2021-03-30 09:16:59.392 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] image e73512b8-7099-4691-aa48-966aa35b59ff at (/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3): checking
Mar 30 09:16:59 bc1bl14 nova-compute[650019]: 2021-03-30 09:16:59.997 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] Active base files: /var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478 /var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3
** Affects: glance
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1921893
Title:
Snapshots fail: Failed to add object to Swift
Status in Glance:
New
Bug description:
I'm using OpenStack Victoria.
When I try to take snapshots of instances above a certain size, they
disappear during saving state and glance shows the following error:
Mar 30 09:15:14 infra1-glance-container-99614ac2 glance-wsgi-api[96]: 2021-03-30 09:15:14.306 96 INFO glance.api.v2.image_data [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Unable to create trust: no such option collect_timing in group [keystone_authtoken] Use the existing user token.
Mar 30 09:16:10 infra1-glance-container-99614ac2 uwsgi[96]: Tue Mar 30 09:16:10 2021 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2/images/c83f6c4d-0083-4600-95f5-d21b4623af47/file (ip 192.168.110.214) !!!
Mar 30 09:16:11 infra1-glance-container-99614ac2 glance-wsgi-api[96]: 2021-03-30 09:16:11.427 96 ERROR glance_store._drivers.swift.store [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Error during chunked upload to backend, deleting stale chunks.: swiftclient.exceptions.ClientException: put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to reset contents for reupload.
Mar 30 09:16:11 infra1-glance-container-99614ac2 glance-wsgi-api[96]: 2021-03-30 09:16:11.494 96 ERROR glance_store._drivers.swift.store [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to add object to Swift.
Got error from Swift: put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to reset contents for reupload..: swiftclient.exceptions.ClientException: put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to reset contents for reupload.
Mar 30 09:16:11 infra1-glance-container-99614ac2 glance-wsgi-api[96]: 2021-03-30 09:16:11.497 96 ERROR glance.api.v2.image_data [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to upload image data due to internal error: glance_store.exceptions.BackendException: Failed to add object to Swift.
Mar 30 09:16:11 infra1-glance-container-99614ac2 glance-wsgi-api[96]: 2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Caught error: Failed to add object to Swift.
Got error from Swift: put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to reset contents for reupload..: glance_store.exceptions.BackendException: Failed to add object to Swift.
Got error from Swift: put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to reset contents for reupload..
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi Traceback (most recent call last):
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1014, in add
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self._delete_stale_chunks(
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self.force_reraise()
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi raise value
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1003, in add
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi manager.get_connection().put_object(
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1960, in put_object
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return self._retry(reset_func, put_object, container, obj, contents,
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1843, in _retry
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi reset_func(func, *args, **kwargs)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1940, in _default_reset
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi raise ClientException('put_object(%r, %r, ...) failure and no '
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi swiftclient.exceptions.ClientException: put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to reset contents for reupload.
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi During handling of the above exception, another exception occurred:
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi Traceback (most recent call last):
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1347, in __call__
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action,
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1391, in dispatch
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return method(*args, **kwargs)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/utils.py", line 416, in wrapped
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return func(self, req, *args, **kwargs)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 298, in upload
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self._restore(image_repo, image)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self.force_reraise()
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi raise value
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 163, in upload
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi image.set_data(data, size, backend=backend)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/domain/proxy.py", line 208, in set_data
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self.base.set_data(data, size, backend=backend, set_active=set_active)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 501, in set_data
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self.force_reraise()
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi raise value
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 447, in set_data
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self.repo.set_data(data, size, backend=backend,
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/policy.py", line 198, in set_data
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/quota/__init__.py", line 318, in set_data
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self.image.set_data(data, size=size, backend=backend,
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 567, in set_data
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 458, in _upload_to_store
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi multihash, loc_meta) = self.store_api.add_with_multihash(
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 398, in add_with_multihash
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return store_add_to_backend_with_multihash(
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 480, in store_add_to_backend_with_multihash
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi (location, size, checksum, multihash, metadata) = store.add(
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/driver.py", line 279, in add_adapter
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/capabilities.py", line 176, in op_checker
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1082, in add
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi raise glance_store.BackendException(msg)
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi glance_store.exceptions.BackendException: Failed to add object to Swift.
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi Got error from Swift: put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to reset contents for reupload..
2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
I don't see errors in the swift logs:
Mar 30 09:15:14 infra1-swift-proxy-container-27169fa7 proxy-server[1068]: - - 30/Mar/2021/09/15/14 HEAD /v1/AUTH_024cc551782f41e395d3c9f13582ef7d%3Fformat%3Djson HTTP/1.0 204 - Swift - - - - txf319964369254af4974a7-006062ec22 - 0.0135 RL - 1617095714.876815081 1617095714.890305758 -
Mar 30 09:15:14 infra1-swift-proxy-container-27169fa7 proxy-server[1068]: - - 30/Mar/2021/09/15/14 HEAD /v1/AUTH_024cc551782f41e395d3c9f13582ef7d%3Fformat%3Djson HTTP/1.0 204 - Swift - - - - txf319964369254af4974a7-006062ec22 - 0.0131 CS - 1617095714.892540216 1617095714.905595303 -
Mar 30 09:15:14 infra1-swift-proxy-container-27169fa7 proxy-server[1068]: - - 30/Mar/2021/09/15/14 HEAD /v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images%3Fformat%3Djson%26states%3Dlisting HTTP/1.0 204 - Swift - - - - txf319964369254af4974a7-006062ec22 - 0.0127 CS - 1617095714.907815695 1617095714.920493126 0
Mar 30 09:15:15 infra1-swift-proxy-container-27169fa7 proxy-server[1068]: 192.168.110.106 192.168.110.211 30/Mar/2021/09/15/15 HEAD /v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images%3Fformat%3Djson%26states%3Dlisting HTTP/1.0 204 - python-swiftclient-3.10.1 gAAAAABgYuwin4R-... - - - txf319964369254af4974a7-006062ec22 - 0.2779 - - 1617095714.874005318 1617095715.151889324 0
Mar 30 09:15:17 infra1-swift-proxy-container-27169fa7 proxy-server[1071]: 192.168.110.106 192.168.110.211 30/Mar/2021/09/15/17 PUT /v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001 HTTP/1.0 201 - python-swiftclient-3.10.1 gAAAAABgYuwin4R-... 204800000 - - txc05fff07e3384a68a7fa9-006062ec23 - 2.8218 - - 1617095715.163244247 1617095717.985031843 0
Mar 30 09:16:10 infra1-swift-proxy-container-27169fa7 proxy-server[1067]: Client disconnected without sending last chunk (txn: tx49b0efe394364bf1975bb-006062ec25) (client_ip: 192.168.110.106)
Mar 30 09:16:10 infra1-swift-proxy-container-27169fa7 proxy-server[1067]: 192.168.110.106 192.168.110.211 30/Mar/2021/09/16/10 PUT /v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00002 HTTP/1.0 499 - python-swiftclient-3.10.1 gAAAAABgYuwin4R-... 204800000 89 - tx49b0efe394364bf1975bb-006062ec25 - 52.2224 - - 1617095717.995234489 1617095770.217619896 0
Mar 30 09:16:11 infra1-swift-proxy-container-27169fa7 proxy-server[1080]: 192.168.110.106 192.168.110.211 30/Mar/2021/09/16/11 DELETE /v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001 HTTP/1.0 204 - python-swiftclient-3.10.1 gAAAAABgYuwin4R-... - - - tx98a9247419a2445fafcbf-006062ec5b - 0.0486 - - 1617095771.441193104 1617095771.489775896 0
and
Mar 30 09:15:17 bc1bl12 object-server[694851]: 10.0.3.211 - - [30/Mar/2021:09:15:17 +0000] "PUT /os-objects/245/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001" 201 - "PUT http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001"; "txc05fff07e3384a68a7fa9-006062ec23" "proxy-server 1071" 2.8030 "-" 694851 0
Mar 30 09:16:10 bc1bl12 object-server[694850]: 10.0.3.211 - - [30/Mar/2021:09:16:10 +0000] "PUT /os-objects/43/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00002" 499 89 "PUT http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00002"; "tx49b0efe394364bf1975bb-006062ec25" "proxy-server 1067" 52.1551 "-" 694850 0
Mar 30 09:16:11 bc1bl12 object-server[694854]: 10.0.3.211 - - [30/Mar/2021:09:16:11 +0000] "DELETE /os-objects/245/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001" 204 - "DELETE http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001"; "tx98a9247419a2445fafcbf-006062ec5b" "proxy-server 1080" 0.0321 "-" 694854 0
nova-compute shows the following:
Mar 30 09:13:24 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:24.975 650019 INFO nova.compute.manager [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] instance snapshotting
Mar 30 09:13:25 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:25.038 650019 INFO nova.virt.libvirt.driver [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] Beginning live snapshot process
Mar 30 09:13:25 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:25.527 650019 INFO nova.virt.libvirt.driver [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] Skipping quiescing instance: QEMU guest agent is not enabled.
Mar 30 09:13:56 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:56.384 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] image c27e2055-8c3c-49d7-9c12-44999b1e7e0f at (/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478): checking
Mar 30 09:13:56 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:56.386 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] image e73512b8-7099-4691-aa48-966aa35b59ff at (/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3): checking
Mar 30 09:13:56 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:56.983 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] Active base files: /var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478 /var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3
Mar 30 09:14:57 bc1bl14 nova-compute[650019]: 2021-03-30 09:14:57.390 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] image c27e2055-8c3c-49d7-9c12-44999b1e7e0f at (/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478): checking
Mar 30 09:14:57 bc1bl14 nova-compute[650019]: 2021-03-30 09:14:57.391 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] image e73512b8-7099-4691-aa48-966aa35b59ff at (/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3): checking
Mar 30 09:14:58 bc1bl14 nova-compute[650019]: 2021-03-30 09:14:58.006 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] Active base files: /var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478 /var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3
Mar 30 09:15:13 bc1bl14 nova-compute[650019]: 2021-03-30 09:15:13.861 650019 INFO nova.virt.libvirt.driver [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] Snapshot extracted, beginning image upload
Mar 30 09:15:58 bc1bl14 nova-compute[650019]: 2021-03-30 09:15:58.395 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] image c27e2055-8c3c-49d7-9c12-44999b1e7e0f at (/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478): checking
Mar 30 09:15:58 bc1bl14 nova-compute[650019]: 2021-03-30 09:15:58.397 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] image e73512b8-7099-4691-aa48-966aa35b59ff at (/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3): checking
Mar 30 09:15:58 bc1bl14 nova-compute[650019]: 2021-03-30 09:15:58.997 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] Active base files: /var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478 /var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3
Mar 30 09:16:12 bc1bl14 nova-compute[650019]: 2021-03-30 09:16:12.530 650019 INFO nova.compute.manager [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] Successfully reverted task state from image_uploading on failure for instance.
Mar 30 09:16:12 bc1bl14 nova-compute[650019]: 2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Exception during message handling: glanceclient.exc.HTTPInternalServerError: HTTP 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation.
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 2478, in snapshot
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server metadata['location'] = root_disk.direct_snapshot(
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/imagebackend.py", line 452, in direct_snapshot
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server raise NotImplementedError(_('direct_snapshot() is not implemented'))
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server NotImplementedError: direct_snapshot() is not implemented
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/exception_wrapper.py", line 76, in wrapped
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server _emit_exception_notification(
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server raise value
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/exception_wrapper.py", line 69, in wrapped
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 188, in decorated_function
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server LOG.warning("Failed to revert task state for instance. "
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server raise value
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 159, in decorated_function
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/utils.py", line 1434, in decorated_function
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 216, in decorated_function
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server compute_utils.add_instance_fault_from_exc(context,
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server raise value
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 205, in decorated_function
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 236, in decorated_function
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server compute_utils.delete_image(
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server raise value
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 232, in decorated_function
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return function(self, context, image_id, instance,
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 3908, in snapshot_instance
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server self._snapshot_instance(context, image_id, instance,
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 3941, in _snapshot_instance
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server self.driver.snapshot(context, instance, image_id,
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 2549, in snapshot
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server self._image_api.update(context,
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 1247, in update
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return session.update(context, image_id, image_info, data=data,
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 696, in update
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server _reraise_translated_image_exception(image_id)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 1035, in _reraise_translated_image_exception
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server raise new_exc.with_traceback(exc_trace)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 694, in update
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server image = self._update_v2(context, sent_service_image_meta, data)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 713, in _update_v2
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server image = self._upload_data(context, image_id, data)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 589, in _upload_data
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server utils.tpool_execute(self._client.call,
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/utils.py", line 694, in tpool_execute
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server eventlet.tpool.execute(func, *args, **kwargs)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/eventlet/tpool.py", line 129, in execute
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server six.reraise(c, e, tb)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server raise value
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/eventlet/tpool.py", line 83, in tworker
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 192, in call
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server result = getattr(controller, method)(*args, **kwargs)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/utils.py", line 600, in inner
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return RequestIdProxy(wrapped(*args, **kwargs))
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/v2/images.py", line 289, in upload
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server resp, body = self.http_client.put(url, headers=hdrs, data=body)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/keystoneauth1/adapter.py", line 404, in put
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return self.request(url, 'PUT', **kwargs)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/http.py", line 380, in request
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return self._handle_response(resp)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/http.py", line 120, in _handle_response
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server raise exc.from_response(resp, resp.content)
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server glanceclient.exc.HTTPInternalServerError: HTTP 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation.
2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server
Mar 30 09:16:59 bc1bl14 nova-compute[650019]: 2021-03-30 09:16:59.391 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] image c27e2055-8c3c-49d7-9c12-44999b1e7e0f at (/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478): checking
Mar 30 09:16:59 bc1bl14 nova-compute[650019]: 2021-03-30 09:16:59.392 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] image e73512b8-7099-4691-aa48-966aa35b59ff at (/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3): checking
Mar 30 09:16:59 bc1bl14 nova-compute[650019]: 2021-03-30 09:16:59.997 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] Active base files: /var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478 /var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3
To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1921893/+subscriptions
