yahoo-eng-team team mailing list archive

[Thaddeus Fortenberry <1989013@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

We are trying to upload images to an on-premise minio-based S3
compatible storage cluster. We have been using it for ~30GB images for
months.  However when increasing the size to >30GB, the image is
successfully uploaded and then glance seems to tell it to delete.

Successful log looks like this:
Sep 6, 2022 @ 08:40:22.812	1	1	rPR1E4MBBdpF5mC-rsYE	test_logs	-	ttcloud-image	ttcloud-image	CompleteMultipartUpload	CompleteMultipartUpload
Sep 6, 2022 @ 08:39:39.038	1	1	q_R1E4MBBdpF5mC-A8YF	test_logs	-	ttcloud-image	ttcloud-image	PutObjectPart	PutObjectPart
...
Whole bunch of PutObjectPart...


Failure log looks like this:
Sep 2, 2022 @ 18:11:46.067	1	1	PvTnAIMBBdpF5mC-XLv7	test_logs	-	ttcloud-image	ttcloud-image	DeleteObject	DeleteObject
Sep 2, 2022 @ 18:11:45.987	1	1	PfTnAIMBBdpF5mC-XLur	test_logs	-	ttcloud-image	ttcloud-image	HeadObject	HeadObject
Sep 2, 2022 @ 18:11:45.927	1	1	PPTnAIMBBdpF5mC-XLtw	test_logs	-	ttcloud-image	ttcloud-image	CompleteMultipartUpload	CompleteMultipartUpload
Sep 2, 2022 @ 18:10:49.370	1	1	O_TmAIMBBdpF5mC-f7uE	test_logs	-	ttcloud-image	ttcloud-image	PutObjectPart	PutObjectPart


We were able to get the 38GB image to work by increasing the chunk size from 10MB to 100MB.  However this does not help the 50GB or 100GB.

The storage system is correctly responding to all S3 request with either
200 or 204 replies. It appears that something important is happening
with the HeadObject which is then causing glance to send a DeleteObject.

I've attached shortened versions of the S3 logs - I've deleted the
majority of PutObjectPart entries to keep the size down.

We're not finding any information. What is causing this behavior?

** Affects: glance
     Importance: Undecided
         Status: New


** Tags: complete delete deleteobject headobject s3

** Attachment added: "S3 Logs"
   https://bugs.launchpad.net/bugs/1989013/+attachment/5614450/+files/S3%20-%20Fail%20+38GB%20S3%20Transfer%20%28short%29.numbers

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1989013

Title:
  Large images uploaded but then deleted

Status in Glance:
  New

Bug description:
  We are trying to upload images to an on-premise minio-based S3
  compatible storage cluster. We have been using it for ~30GB images for
  months.  However when increasing the size to >30GB, the image is
  successfully uploaded and then glance seems to tell it to delete.

  Successful log looks like this:
  Sep 6, 2022 @ 08:40:22.812	1	1	rPR1E4MBBdpF5mC-rsYE	test_logs	-	ttcloud-image	ttcloud-image	CompleteMultipartUpload	CompleteMultipartUpload
  Sep 6, 2022 @ 08:39:39.038	1	1	q_R1E4MBBdpF5mC-A8YF	test_logs	-	ttcloud-image	ttcloud-image	PutObjectPart	PutObjectPart
  ...
  Whole bunch of PutObjectPart...

  
  Failure log looks like this:
  Sep 2, 2022 @ 18:11:46.067	1	1	PvTnAIMBBdpF5mC-XLv7	test_logs	-	ttcloud-image	ttcloud-image	DeleteObject	DeleteObject
  Sep 2, 2022 @ 18:11:45.987	1	1	PfTnAIMBBdpF5mC-XLur	test_logs	-	ttcloud-image	ttcloud-image	HeadObject	HeadObject
  Sep 2, 2022 @ 18:11:45.927	1	1	PPTnAIMBBdpF5mC-XLtw	test_logs	-	ttcloud-image	ttcloud-image	CompleteMultipartUpload	CompleteMultipartUpload
  Sep 2, 2022 @ 18:10:49.370	1	1	O_TmAIMBBdpF5mC-f7uE	test_logs	-	ttcloud-image	ttcloud-image	PutObjectPart	PutObjectPart

  
  We were able to get the 38GB image to work by increasing the chunk size from 10MB to 100MB.  However this does not help the 50GB or 100GB.

  The storage system is correctly responding to all S3 request with
  either 200 or 204 replies. It appears that something important is
  happening with the HeadObject which is then causing glance to send a
  DeleteObject.

  I've attached shortened versions of the S3 logs - I've deleted the
  majority of PutObjectPart entries to keep the size down.

  We're not finding any information. What is causing this behavior?

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1989013/+subscriptions