yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #94410
[Bug 2076437] [NEW] glance doesn't convert images to RAW if setting is enabled
Public bug reported:
DESCRIPTION
===========
Glance 26.0.0, Openstack Antelope deployed with Juju Charms.
Settings in glance-api.conf:
-------------------
[image_import_opts]
image_import_plugins = ['image_conversion']
[image_conversion]
output_format = raw
------------------
I am syncing images using simplestreams-sync latest/edge. SS pulls IMG filenames for Ubuntu 24.04 amd 64 resulting in download of the respective image which is QCOW2 format.
If SS is configured to enable-image-conversion (Glance has to have that
option enabled which it has) then SS assumes that downloaded image is
RAW and it pushes that to Glance:
--------------------
glance create_kwargs {'name': 'auto-sync/ubuntu-noble-24.04-amd64-server-20240809-disk1.img', 'container_format': 'bare', 'visibility': 'public', 'disk_format': 'raw'}
--------------------
Glance should check with its inspectors in the image_conversion.py if
the uploaded image is indeed RAW or not, it seems the check fails:
-------------------
Source is already in target format, not doing conversion for 6a79cbd5-cd19-4aea-a155-f982ed75ac62 _execute /usr/lib/python3/dist-packages/glance/async_/flows/plugins/image_conversion.py:181
------------------
What happens in that file, the check is running "qemu-img info -f
source_format --format=json /uploadedfilebyss which in our case would
translale to "qemu-img info -f raw ...."
That results always in RAW because the source_format provided by image
create command from SS is RAW hence the message "Source is already in
target format".
Then the image shows in Images as RAW but it has 568M, if I download it
from glance it is actually QCOW2 and not RAW.
example below:
ubuntu-24.04-server-cloudimg-amd64.img => downloaded from official
cloud-images
qemu-img info --output=json ubuntu-24.04-server-cloudimg-amd64.img
{
"virtual-size": 3758096384,
"filename": "ubuntu-24.04-server-cloudimg-amd64.img",
"cluster-size": 65536,
"format": "qcow2",
"actual-size": 585830400,
"format-specific": {
"type": "qcow2",
"data": {
"compat": "1.1",
"compression-type": "zlib",
"lazy-refcounts": false,
"refcount-bits": 16,
"corrupt": false,
"extended-l2": false
}
},
"dirty-flag": false
}
running the same command as the image_conversion plugin is doing:
qemu-img info -f raw --output=json ubuntu-24.04-server-cloudimg-
amd64.img
{
"virtual-size": 585826304,
"filename": "ubuntu-24.04-server-cloudimg-amd64.img",
"format": "raw",
"actual-size": 585830400,
"dirty-flag": false
}
I manually removed the -f source_format from image_conversion.py so the
command actually returns the correct format and i get this error :
"Failed to execute task a805c37c-ea60-4b1b-9a75-9dee11e4e5cc: Image
metadata disagrees about format: RuntimeError: Image metadata disagrees
about format"
I reverted my change and disabled image_conversion in simplestreams, and
now SS uploads the image to Glance as QCOW2 as it normally is.
But now, no image conversion happens at Glance level even though is
configured, and the system actually skips the step altogether,
proceeding to upload the image to RBD as normal.
====cut here=====
2024-08-09 15:09:41.472 543146 DEBUG glance.location [None req-443b8021-070f-462c-acd3-1831cc85f080 e8ed0a8415ef498ebcece070bef55de2 021dc8ae82ae4a5f9242eeec1ac86005 - - 021a9c699be4425c889335019f91910e 021a9c699be4425c889335019f91910e] Enabling in-flight format inspection for qcow2 set_data /usr/lib/python3/dist-packages/glance/location.py:581
2024-08-09 15:09:41.473 543146 DEBUG glance_store.multi_backend [None req-443b8021-070f-462c-acd3-1831cc85f080 e8ed0a8415ef498ebcece070bef55de2 021dc8ae82ae4a5f9242eeec1ac86005 - - 021a9c699be4425c889335019f91910e 021a9c699be4425c889335019f91910e] Attempting to import store rbd _load_multi_store /usr/lib/python3/dist-packages/glance_store/multi_backend.py:170
2024-08-09 15:09:41.495 543146 DEBUG glance_store.capabilities [None req-443b8021-070f-462c-acd3-1831cc85f080 e8ed0a8415ef498ebcece070bef55de2 021dc8ae82ae4a5f9242eeec1ac86005 - - 021a9c699be4425c889335019f91910e 021a9c699be4425c889335019f91910e] Store glance_store._drivers.rbd.Store doesn't support updating dynamic storage capabilities. Please overwrite 'update_capabilities' method of the store to implement updating logics if needed. update_capabilities /usr/lib/python3/dist-packages/glance_store/capabilities.py:91
2024-08-09 15:09:41.496 543146 DEBUG glance_store.driver [None req-443b8021-070f-462c-acd3-1831cc85f080 e8ed0a8415ef498ebcece070bef55de2 021dc8ae82ae4a5f9242eeec1ac86005 - - 021a9c699be4425c889335019f91910e 021a9c699be4425c889335019f91910e] Late loading location class glance_store._drivers.rbd.StoreLocation get_store_location_class /usr/lib/python3/dist-packages/glance_store/driver.py:116
2024-08-09 15:09:41.496 543146 DEBUG glance_store.location [None req-443b8021-070f-462c-acd3-1831cc85f080 e8ed0a8415ef498ebcece070bef55de2 021dc8ae82ae4a5f9242eeec1ac86005 - - 021a9c699be4425c889335019f91910e 021a9c699be4425c889335019f91910e] Registering scheme rbd with {'ceph': {'store': <glance_store._drivers.rbd.Store object at 0x7f004ce495a0>, 'location_class': <class 'glance_store._drivers.rbd.StoreLocation'>, 'store_entry': 'rbd'}} register_scheme_backend_map /usr/lib/python3/dist-packages/glance_store/location.py:132
2024-08-09 15:09:41.520 543146 DEBUG glance_store._drivers.rbd [None req-443b8021-070f-462c-acd3-1831cc85f080 e8ed0a8415ef498ebcece070bef55de2 021dc8ae82ae4a5f9242eeec1ac86005 - - 021a9c699be4425c889335019f91910e 021a9c699be4425c889335019f91910e] creating image c3ccd12e-971f-4a81-aaf3-6652da62571c with order 23 and size 0 add /usr/lib/python3/dist-packages/glance_store/_drivers/rbd.py:573
2024-08-09 15:09:41.520 543146 WARNING glance_store._drivers.rbd [None req-443b8021-070f-462c-acd3-1831cc85f080 e8ed0a8415ef498ebcece070bef55de2 021dc8ae82ae4a5f9242eeec1ac86005 - - 021a9c699be4425c889335019f91910e 021a9c699be4425c889335019f91910e] Since image size is zero we will be doing resize-before-write which will be slower than normal
2024-08-09 15:09:41.700 543146 DEBUG glance_store._drivers.rbd [None req-443b8021-070f-462c-acd3-1831cc85f080 e8ed0a8415ef498ebcece070bef55de2 021dc8ae82ae4a5f9242eeec1ac86005 - - 021a9c699be4425c889335019f91910e 021a9c699be4425c889335019f91910e] resizing image to 8192.0 KiB _resize_on_write /usr/lib/python3/dist-packages/glance_store/_drivers/rbd.py:523
2024-08-09 15:09:42.056 543146 DEBUG glance_store._drivers.rbd [None req-443b8021-070f-462c-acd3-1831cc85f080 e8ed0a8415ef498ebcece070bef55de2 021dc8ae82ae4a5f9242eeec1ac86005 - - 021a9c699be4425c889335019f91910e 021a9c699be4425c889335019f91910e] resizing image to 24576.0 KiB _resize_on_write /usr/lib/python3/dist-packages/glance_store/_drivers/rbd.py:523
2024-08-09 15:09:42.924 543146 DEBUG glance_store._drivers.rbd [None req-443b8021-070f-462c-acd3-1831cc85f080 e8ed0a8415ef498ebcece070bef55de2 021dc8ae82ae4a5f9242eeec1ac86005 - - 021a9c699be4425c889335019f91910e 021a9c699be4425c889335019f91910e] resizing image to 57344.0 KiB _resize_on_write /usr/lib/python3/dist-packages/glance_store/_drivers/rbd.py:523
2024-08-09 15:09:43.041 543147 INFO eventlet.wsgi.server [-] 10.35.1.23,127.0.0.1 - - [09/Aug/2024 15:09:43] "GET /healthcheck HTTP/1.1" 200 142 0.003914
2024-08-09 15:09:44.412 543146 DEBUG glance_store._drivers.rbd [None req-443b8021-070f-462c-acd3-1831cc85f080 e8ed0a8415ef498ebcece070bef55de2 021dc8ae82ae4a5f9242eeec1ac86005 - - 021a9c699be4425c889335019f91910e 021a9c699be4425c889335019f91910e] resizing image to 122880.0 KiB _resize_on_write /usr/lib/python3/dist-
====cut here====
It seems there is a combination of bugs here:
1. Glance doesn't really check properly what the client uploads when a
specific format is provided. (SS provided RAW format but uploaded a
QCOW2 image, no conversion was done)
2. qemu-img should return an error when running a specific format for
"info" command on an image that is not in that format. (see above,
running qemu-img info normally returns the correct format, adding -f raw
returns totally wrong data)
3. simplestreams on its own doesn't do any checks against what formats
it downloads in this specific case (IMG type) it just pushes with -t RAW
if image conversion is enabled on SS side, or -t QCOW2 if it's not.
How to reproduce:
===================
enable image conversion in Glance 26.0.0, set it to output format RAW
enable image conversion in simplestreams sync (latest edge: 0.1.0-aba6969)
set SS to download any amd64 Ubuntu Linux of file type "img" with this
filter:
[{url: http://cloud-images.ubuntu.com/releases/, name_prefix:
ubuntu:released, path: streams/v1/index.sjson, max: 1, item_filters:
[release~(noble), arch~(amd64), ftype~(img)]}]
run simplestreams sync either manually or with juju sync-images if
openstack is deployed like so.
Expected result:
================
Customer sends a wrong type of RAW image but uploads a QCOW2 image type,
Glance should detect and convert it to the specified format configured.
(in this case RAW).
Customer sends a correct type of image which is known and accepted by
Glance (in this case QCow2), Glance should run the image conversion
mechanism to convert it to RAW as instructed in the config file or
output and error if that conversion is not possible. (In this case it is
possible, with qemu-img convert from qcow2 to raw)
Actual result:
==============
Customer sends a QCoW2 image as -t raw , Glance accepts and image conversion plugin says there is no difference between types and proceeds to store the image as RAW in Glance, but it is QCow2. In this case instances cannot be launched with this image.
Customer sends a correct QCow2 image as -t qcow2, Glance accepts and
immediately proceeds to store the image into the backend, in my case
CEPH RBD, instead of launching image_conversion to RAW as the
configuration is instructing to do.
** Affects: glance
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/2076437
Title:
glance doesn't convert images to RAW if setting is enabled
Status in Glance:
New
Bug description:
DESCRIPTION
===========
Glance 26.0.0, Openstack Antelope deployed with Juju Charms.
Settings in glance-api.conf:
-------------------
[image_import_opts]
image_import_plugins = ['image_conversion']
[image_conversion]
output_format = raw
------------------
I am syncing images using simplestreams-sync latest/edge. SS pulls IMG filenames for Ubuntu 24.04 amd 64 resulting in download of the respective image which is QCOW2 format.
If SS is configured to enable-image-conversion (Glance has to have
that option enabled which it has) then SS assumes that downloaded
image is RAW and it pushes that to Glance:
--------------------
glance create_kwargs {'name': 'auto-sync/ubuntu-noble-24.04-amd64-server-20240809-disk1.img', 'container_format': 'bare', 'visibility': 'public', 'disk_format': 'raw'}
--------------------
Glance should check with its inspectors in the image_conversion.py if
the uploaded image is indeed RAW or not, it seems the check fails:
-------------------
Source is already in target format, not doing conversion for 6a79cbd5-cd19-4aea-a155-f982ed75ac62 _execute /usr/lib/python3/dist-packages/glance/async_/flows/plugins/image_conversion.py:181
------------------
What happens in that file, the check is running "qemu-img info -f
source_format --format=json /uploadedfilebyss which in our case would
translale to "qemu-img info -f raw ...."
That results always in RAW because the source_format provided by image
create command from SS is RAW hence the message "Source is already in
target format".
Then the image shows in Images as RAW but it has 568M, if I download
it from glance it is actually QCOW2 and not RAW.
example below:
ubuntu-24.04-server-cloudimg-amd64.img => downloaded from official
cloud-images
qemu-img info --output=json ubuntu-24.04-server-cloudimg-amd64.img
{
"virtual-size": 3758096384,
"filename": "ubuntu-24.04-server-cloudimg-amd64.img",
"cluster-size": 65536,
"format": "qcow2",
"actual-size": 585830400,
"format-specific": {
"type": "qcow2",
"data": {
"compat": "1.1",
"compression-type": "zlib",
"lazy-refcounts": false,
"refcount-bits": 16,
"corrupt": false,
"extended-l2": false
}
},
"dirty-flag": false
}
running the same command as the image_conversion plugin is doing:
qemu-img info -f raw --output=json ubuntu-24.04-server-cloudimg-
amd64.img
{
"virtual-size": 585826304,
"filename": "ubuntu-24.04-server-cloudimg-amd64.img",
"format": "raw",
"actual-size": 585830400,
"dirty-flag": false
}
I manually removed the -f source_format from image_conversion.py so
the command actually returns the correct format and i get this error :
"Failed to execute task a805c37c-ea60-4b1b-9a75-9dee11e4e5cc: Image
metadata disagrees about format: RuntimeError: Image metadata
disagrees about format"
I reverted my change and disabled image_conversion in simplestreams,
and now SS uploads the image to Glance as QCOW2 as it normally is.
But now, no image conversion happens at Glance level even though is
configured, and the system actually skips the step altogether,
proceeding to upload the image to RBD as normal.
====cut here=====
2024-08-09 15:09:41.472 543146 DEBUG glance.location [None req-443b8021-070f-462c-acd3-1831cc85f080 e8ed0a8415ef498ebcece070bef55de2 021dc8ae82ae4a5f9242eeec1ac86005 - - 021a9c699be4425c889335019f91910e 021a9c699be4425c889335019f91910e] Enabling in-flight format inspection for qcow2 set_data /usr/lib/python3/dist-packages/glance/location.py:581
2024-08-09 15:09:41.473 543146 DEBUG glance_store.multi_backend [None req-443b8021-070f-462c-acd3-1831cc85f080 e8ed0a8415ef498ebcece070bef55de2 021dc8ae82ae4a5f9242eeec1ac86005 - - 021a9c699be4425c889335019f91910e 021a9c699be4425c889335019f91910e] Attempting to import store rbd _load_multi_store /usr/lib/python3/dist-packages/glance_store/multi_backend.py:170
2024-08-09 15:09:41.495 543146 DEBUG glance_store.capabilities [None req-443b8021-070f-462c-acd3-1831cc85f080 e8ed0a8415ef498ebcece070bef55de2 021dc8ae82ae4a5f9242eeec1ac86005 - - 021a9c699be4425c889335019f91910e 021a9c699be4425c889335019f91910e] Store glance_store._drivers.rbd.Store doesn't support updating dynamic storage capabilities. Please overwrite 'update_capabilities' method of the store to implement updating logics if needed. update_capabilities /usr/lib/python3/dist-packages/glance_store/capabilities.py:91
2024-08-09 15:09:41.496 543146 DEBUG glance_store.driver [None req-443b8021-070f-462c-acd3-1831cc85f080 e8ed0a8415ef498ebcece070bef55de2 021dc8ae82ae4a5f9242eeec1ac86005 - - 021a9c699be4425c889335019f91910e 021a9c699be4425c889335019f91910e] Late loading location class glance_store._drivers.rbd.StoreLocation get_store_location_class /usr/lib/python3/dist-packages/glance_store/driver.py:116
2024-08-09 15:09:41.496 543146 DEBUG glance_store.location [None req-443b8021-070f-462c-acd3-1831cc85f080 e8ed0a8415ef498ebcece070bef55de2 021dc8ae82ae4a5f9242eeec1ac86005 - - 021a9c699be4425c889335019f91910e 021a9c699be4425c889335019f91910e] Registering scheme rbd with {'ceph': {'store': <glance_store._drivers.rbd.Store object at 0x7f004ce495a0>, 'location_class': <class 'glance_store._drivers.rbd.StoreLocation'>, 'store_entry': 'rbd'}} register_scheme_backend_map /usr/lib/python3/dist-packages/glance_store/location.py:132
2024-08-09 15:09:41.520 543146 DEBUG glance_store._drivers.rbd [None req-443b8021-070f-462c-acd3-1831cc85f080 e8ed0a8415ef498ebcece070bef55de2 021dc8ae82ae4a5f9242eeec1ac86005 - - 021a9c699be4425c889335019f91910e 021a9c699be4425c889335019f91910e] creating image c3ccd12e-971f-4a81-aaf3-6652da62571c with order 23 and size 0 add /usr/lib/python3/dist-packages/glance_store/_drivers/rbd.py:573
2024-08-09 15:09:41.520 543146 WARNING glance_store._drivers.rbd [None req-443b8021-070f-462c-acd3-1831cc85f080 e8ed0a8415ef498ebcece070bef55de2 021dc8ae82ae4a5f9242eeec1ac86005 - - 021a9c699be4425c889335019f91910e 021a9c699be4425c889335019f91910e] Since image size is zero we will be doing resize-before-write which will be slower than normal
2024-08-09 15:09:41.700 543146 DEBUG glance_store._drivers.rbd [None req-443b8021-070f-462c-acd3-1831cc85f080 e8ed0a8415ef498ebcece070bef55de2 021dc8ae82ae4a5f9242eeec1ac86005 - - 021a9c699be4425c889335019f91910e 021a9c699be4425c889335019f91910e] resizing image to 8192.0 KiB _resize_on_write /usr/lib/python3/dist-packages/glance_store/_drivers/rbd.py:523
2024-08-09 15:09:42.056 543146 DEBUG glance_store._drivers.rbd [None req-443b8021-070f-462c-acd3-1831cc85f080 e8ed0a8415ef498ebcece070bef55de2 021dc8ae82ae4a5f9242eeec1ac86005 - - 021a9c699be4425c889335019f91910e 021a9c699be4425c889335019f91910e] resizing image to 24576.0 KiB _resize_on_write /usr/lib/python3/dist-packages/glance_store/_drivers/rbd.py:523
2024-08-09 15:09:42.924 543146 DEBUG glance_store._drivers.rbd [None req-443b8021-070f-462c-acd3-1831cc85f080 e8ed0a8415ef498ebcece070bef55de2 021dc8ae82ae4a5f9242eeec1ac86005 - - 021a9c699be4425c889335019f91910e 021a9c699be4425c889335019f91910e] resizing image to 57344.0 KiB _resize_on_write /usr/lib/python3/dist-packages/glance_store/_drivers/rbd.py:523
2024-08-09 15:09:43.041 543147 INFO eventlet.wsgi.server [-] 10.35.1.23,127.0.0.1 - - [09/Aug/2024 15:09:43] "GET /healthcheck HTTP/1.1" 200 142 0.003914
2024-08-09 15:09:44.412 543146 DEBUG glance_store._drivers.rbd [None req-443b8021-070f-462c-acd3-1831cc85f080 e8ed0a8415ef498ebcece070bef55de2 021dc8ae82ae4a5f9242eeec1ac86005 - - 021a9c699be4425c889335019f91910e 021a9c699be4425c889335019f91910e] resizing image to 122880.0 KiB _resize_on_write /usr/lib/python3/dist-
====cut here====
It seems there is a combination of bugs here:
1. Glance doesn't really check properly what the client uploads when a
specific format is provided. (SS provided RAW format but uploaded a
QCOW2 image, no conversion was done)
2. qemu-img should return an error when running a specific format for
"info" command on an image that is not in that format. (see above,
running qemu-img info normally returns the correct format, adding -f
raw returns totally wrong data)
3. simplestreams on its own doesn't do any checks against what formats
it downloads in this specific case (IMG type) it just pushes with -t
RAW if image conversion is enabled on SS side, or -t QCOW2 if it's
not.
How to reproduce:
===================
enable image conversion in Glance 26.0.0, set it to output format RAW
enable image conversion in simplestreams sync (latest edge: 0.1.0-aba6969)
set SS to download any amd64 Ubuntu Linux of file type "img" with this
filter:
[{url: http://cloud-images.ubuntu.com/releases/, name_prefix:
ubuntu:released, path: streams/v1/index.sjson, max: 1, item_filters:
[release~(noble), arch~(amd64), ftype~(img)]}]
run simplestreams sync either manually or with juju sync-images if
openstack is deployed like so.
Expected result:
================
Customer sends a wrong type of RAW image but uploads a QCOW2 image
type, Glance should detect and convert it to the specified format
configured. (in this case RAW).
Customer sends a correct type of image which is known and accepted by
Glance (in this case QCow2), Glance should run the image conversion
mechanism to convert it to RAW as instructed in the config file or
output and error if that conversion is not possible. (In this case it
is possible, with qemu-img convert from qcow2 to raw)
Actual result:
==============
Customer sends a QCoW2 image as -t raw , Glance accepts and image conversion plugin says there is no difference between types and proceeds to store the image as RAW in Glance, but it is QCow2. In this case instances cannot be launched with this image.
Customer sends a correct QCow2 image as -t qcow2, Glance accepts and
immediately proceeds to store the image into the backend, in my case
CEPH RBD, instead of launching image_conversion to RAW as the
configuration is instructing to do.
To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/2076437/+subscriptions
