yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2110222] [NEW] Nova server image uploaded to S3 uses singlepart instead of multipart upload

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Erik Dekker <2110222@xxxxxxxxxxxxxxxxxx>
Date: Thu, 08 May 2025 14:35:34 -0000
Reply-to: Bug 2110222 <2110222@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

When creating an image from a server using the `openstack server image
create` command, a snapshot is created by nova and uploaded to S3 via
glanceclient upload method.

The size of the image is not supplied to glanceclient (however it can be
known to nova by checking the size of temporary file nova created), and
therefore it will be set to 0 when its None. This causes the S3 driver
to assume the size is smaller than the store_large_object_size and call
the _add_singlepart method instead of _add_multipart. This results in
poor upload performance and high memory usage on the glance-api server
because the whole upload is read in memory.

The (virtual) size will eventually be calculated when the upload to S3
is completed, and then get saved in the database.


Steps to reproduce:
* Configure an S3 backend like MinIO as default_backend in devstack
* openstack server create --flavor m1.small --image debian12 --network public vm1
* openstack server image create vm1

Expected result:
* Image is uploaded to S3 in parts using multipart

Actual result:
* Image is uploaded to S3 using _add_singlepart, causing high memory consumption and duration.


Performance impact singlepart vs multipart:

singlepart:

May 01 19:54:23 devstack-1 nova-compute[502984]: INFO nova.virt.libvirt.driver [None req-cf59e8a9-d12a-4424-8221-9e64fb62b5f7 demo admin] [instance: c3f80570-8ce5-40c4-a815-6acad873bba5] Snapshot extracted, beginning image upload
May 01 19:55:49 devstack-1 nova-compute[502984]: INFO nova.virt.libvirt.driver [None req-cf59e8a9-d12a-4424-8221-9e64fb62b5f7 demo admin] [instance: c3f80570-8ce5-40c4-a815-6acad873bba5] Snapshot image upload complete
May 01 19:55:49 devstack-1 nova-compute[502984]: INFO nova.compute.manager [None req-cf59e8a9-d12a-4424-8221-9e64fb62b5f7 demo admin] [instance: c3f80570-8ce5-40c4-a815-6acad873bba5] Took 89.70 seconds to snapshot the instance on the hypervisor.

multipart:

May 01 19:57:24 devstack-1 nova-compute[502984]: INFO nova.virt.libvirt.driver [None req-272205be-2fea-417d-8412-2a2a45daefb5 demo admin] [instance: c3f80570-8ce5-40c4-a815-6acad873bba5] Snapshot extracted, beginning image upload
May 01 19:57:38 devstack-1 nova-compute[502984]: INFO nova.virt.libvirt.driver [None req-272205be-2fea-417d-8412-2a2a45daefb5 demo admin] [instance: c3f80570-8ce5-40c4-a815-6acad873bba5] Snapshot image upload complete
May 01 19:57:38 devstack-1 nova-compute[502984]: INFO nova.compute.manager [None req-272205be-2fea-417d-8412-2a2a45daefb5 demo admin] [instance: c3f80570-8ce5-40c4-a815-6acad873bba5] Took 18.83 seconds to snapshot the instance on the hypervisor


I've tried forcing multipart uploads by using this patch https://review.opendev.org/c/openstack/glance_store/+/946382/1/glance_store/_drivers/s3.py and setting the large_object_size to 0. This fixes the issue, but creates another issue when creating an image of volume-backed instances. In that case, a 0 byte dummy object is created as placeholder, and metadata with a reference to the volume snapshot id is saved in glance. This errors out:

ERROR glance.common.wsgi botocore.errorfactory.InvalidRequest: An error
occurred (InvalidRequest) when calling the CompleteMultipartUpload
operation: You must specify at least one part

Related to this similar bug:
https://bugs.launchpad.net/glance-store/+bug/2043740

** Affects: glance-store
     Importance: Undecided
         Status: New

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: glance glance-store glanceclient multipart singlepart

** Also affects: nova
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2110222

Title:
  Nova server image uploaded to S3 uses singlepart instead of multipart
  upload

Status in glance_store:
  New
Status in OpenStack Compute (nova):
  New

Bug description:
  When creating an image from a server using the `openstack server image
  create` command, a snapshot is created by nova and uploaded to S3 via
  glanceclient upload method.

  The size of the image is not supplied to glanceclient (however it can
  be known to nova by checking the size of temporary file nova created),
  and therefore it will be set to 0 when its None. This causes the S3
  driver to assume the size is smaller than the store_large_object_size
  and call the _add_singlepart method instead of _add_multipart. This
  results in poor upload performance and high memory usage on the
  glance-api server because the whole upload is read in memory.

  The (virtual) size will eventually be calculated when the upload to S3
  is completed, and then get saved in the database.

  
  Steps to reproduce:
  * Configure an S3 backend like MinIO as default_backend in devstack
  * openstack server create --flavor m1.small --image debian12 --network public vm1
  * openstack server image create vm1

  Expected result:
  * Image is uploaded to S3 in parts using multipart

  Actual result:
  * Image is uploaded to S3 using _add_singlepart, causing high memory consumption and duration.

  
  Performance impact singlepart vs multipart:

  singlepart:

  May 01 19:54:23 devstack-1 nova-compute[502984]: INFO nova.virt.libvirt.driver [None req-cf59e8a9-d12a-4424-8221-9e64fb62b5f7 demo admin] [instance: c3f80570-8ce5-40c4-a815-6acad873bba5] Snapshot extracted, beginning image upload
  May 01 19:55:49 devstack-1 nova-compute[502984]: INFO nova.virt.libvirt.driver [None req-cf59e8a9-d12a-4424-8221-9e64fb62b5f7 demo admin] [instance: c3f80570-8ce5-40c4-a815-6acad873bba5] Snapshot image upload complete
  May 01 19:55:49 devstack-1 nova-compute[502984]: INFO nova.compute.manager [None req-cf59e8a9-d12a-4424-8221-9e64fb62b5f7 demo admin] [instance: c3f80570-8ce5-40c4-a815-6acad873bba5] Took 89.70 seconds to snapshot the instance on the hypervisor.

  multipart:

  May 01 19:57:24 devstack-1 nova-compute[502984]: INFO nova.virt.libvirt.driver [None req-272205be-2fea-417d-8412-2a2a45daefb5 demo admin] [instance: c3f80570-8ce5-40c4-a815-6acad873bba5] Snapshot extracted, beginning image upload
  May 01 19:57:38 devstack-1 nova-compute[502984]: INFO nova.virt.libvirt.driver [None req-272205be-2fea-417d-8412-2a2a45daefb5 demo admin] [instance: c3f80570-8ce5-40c4-a815-6acad873bba5] Snapshot image upload complete
  May 01 19:57:38 devstack-1 nova-compute[502984]: INFO nova.compute.manager [None req-272205be-2fea-417d-8412-2a2a45daefb5 demo admin] [instance: c3f80570-8ce5-40c4-a815-6acad873bba5] Took 18.83 seconds to snapshot the instance on the hypervisor

  
  I've tried forcing multipart uploads by using this patch https://review.opendev.org/c/openstack/glance_store/+/946382/1/glance_store/_drivers/s3.py and setting the large_object_size to 0. This fixes the issue, but creates another issue when creating an image of volume-backed instances. In that case, a 0 byte dummy object is created as placeholder, and metadata with a reference to the volume snapshot id is saved in glance. This errors out:

  ERROR glance.common.wsgi botocore.errorfactory.InvalidRequest: An
  error occurred (InvalidRequest) when calling the
  CompleteMultipartUpload operation: You must specify at least one part

  Related to this similar bug:
  https://bugs.launchpad.net/glance-store/+bug/2043740

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance-store/+bug/2110222/+subscriptions