yahoo-eng-team team mailing list archive

[Koya Watanabe <2121144@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Description:
After upgrading Glance from Dalmatian to Epoxy, uploading an image to an S3-compatible storage backend fails.
The failure only occurs on Epoxy and correlates with the boto3 1.36.x upgrade, which enabled new default S3 checksum behaviors.
Many S3-compatible storage services (e.g., Ceph RGW) do not yet support these behaviors, which causes boto3/botocore to raise errors and the Glance upload to fail.

This is a regression from Dalmatian (uploads succeeded there).


Environment:
Glance - Epoxy
boto3 - 1.36.26 (installed via Epoxy upper-constraints; https://github.com/openstack/requirements/blob/bfd91c04caf498be2bfda0732a2a0f9fa9306e07/upper-constraints.txt#L306)
Storage backend - Ceph RGW (S3-compatible storage)


Steps to Reproduce:
1. Configure Glance to use an S3-compatible storage backend (e.g., Ceph RGW).
2. Attempt to upload any image and observe that the upload fails, for example;

```
$ openstack image create "cirros-0.6.2-x86_64" --file ./cirros-0.6.2-x86_64-disk.img --disk-format qcow2 --container-format bare
HttpException: 500: Server Error for url: https://XXXXXXXXXX/v2/images/258cdf6f-ba48-4c03-8cf1-d90114f5da8e/file, Internal Server Error
Error: exit status 1
```

error log;
```
2025-08-21T07:08:05.467 13 ERROR glance.api.v2.image_data [F] [None req-1139a1ef-1339-4f34-aa19-844de43261cb f209e8b906b549fe987161b2c42dbbfd 604177366ff8401a911cf16aacfe6b4b - - default default] Failed to upload image data due to internal error: botocore.exceptions.ClientError: An error occurred (MissingContentLength) when calling the PutObject operation: The internal error code is -2011
```

Suspected Root Cause:
Starting in boto3/botocore 1.36.0, the SDK enables new S3 “data integrity” protections by default (request checksum calculation and response checksum validation).
S3-compatible storage that does not implement these features rejects the new headers or fails validation, causing Glance uploads to fail.
Related: https://github.com/boto/boto3/issues/4392


Workaround:
Set the following environment variables for the Glance API process so that checksum behavior is only used when required by the API;

```
export AWS_REQUEST_CHECKSUM_CALCULATION=WHEN_REQUIRED
export AWS_RESPONSE_CHECKSUM_VALIDATION=WHEN_REQUIRED
```

Guidance:
https://github.com/boto/boto3/issues/4398#issuecomment-2727003405


Proposed Fix (ideas):
- In Glance/glance_store’s S3 path, default boto3’s S3 client config to request_checksum_calculation="when_required" and response_checksum_validation="when_required".
- Document the required environment variables prominently in Glance Epoxy release notes.

** Affects: glance
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/2121144

Title:
  Image uploads to S3-compatible storage fail with Epoxy version after
  upgrade from Dalmatian due to boto3 checksum behavior changes

Status in Glance:
  New

Bug description:
  Description:
  After upgrading Glance from Dalmatian to Epoxy, uploading an image to an S3-compatible storage backend fails.
  The failure only occurs on Epoxy and correlates with the boto3 1.36.x upgrade, which enabled new default S3 checksum behaviors.
  Many S3-compatible storage services (e.g., Ceph RGW) do not yet support these behaviors, which causes boto3/botocore to raise errors and the Glance upload to fail.

  This is a regression from Dalmatian (uploads succeeded there).

  
  Environment:
  Glance - Epoxy
  boto3 - 1.36.26 (installed via Epoxy upper-constraints; https://github.com/openstack/requirements/blob/bfd91c04caf498be2bfda0732a2a0f9fa9306e07/upper-constraints.txt#L306)
  Storage backend - Ceph RGW (S3-compatible storage)

  
  Steps to Reproduce:
  1. Configure Glance to use an S3-compatible storage backend (e.g., Ceph RGW).
  2. Attempt to upload any image and observe that the upload fails, for example;

  ```
  $ openstack image create "cirros-0.6.2-x86_64" --file ./cirros-0.6.2-x86_64-disk.img --disk-format qcow2 --container-format bare
  HttpException: 500: Server Error for url: https://XXXXXXXXXX/v2/images/258cdf6f-ba48-4c03-8cf1-d90114f5da8e/file, Internal Server Error
  Error: exit status 1
  ```

  error log;
  ```
  2025-08-21T07:08:05.467 13 ERROR glance.api.v2.image_data [F] [None req-1139a1ef-1339-4f34-aa19-844de43261cb f209e8b906b549fe987161b2c42dbbfd 604177366ff8401a911cf16aacfe6b4b - - default default] Failed to upload image data due to internal error: botocore.exceptions.ClientError: An error occurred (MissingContentLength) when calling the PutObject operation: The internal error code is -2011
  ```

  Suspected Root Cause:
  Starting in boto3/botocore 1.36.0, the SDK enables new S3 “data integrity” protections by default (request checksum calculation and response checksum validation).
  S3-compatible storage that does not implement these features rejects the new headers or fails validation, causing Glance uploads to fail.
  Related: https://github.com/boto/boto3/issues/4392

  
  Workaround:
  Set the following environment variables for the Glance API process so that checksum behavior is only used when required by the API;

  ```
  export AWS_REQUEST_CHECKSUM_CALCULATION=WHEN_REQUIRED
  export AWS_RESPONSE_CHECKSUM_VALIDATION=WHEN_REQUIRED
  ```

  Guidance:
  https://github.com/boto/boto3/issues/4398#issuecomment-2727003405

  
  Proposed Fix (ideas):
  - In Glance/glance_store’s S3 path, default boto3’s S3 client config to request_checksum_calculation="when_required" and response_checksum_validation="when_required".
  - Document the required environment variables prominently in Glance Epoxy release notes.

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/2121144/+subscriptions