yahoo-eng-team team mailing list archive

[Brian Rosmaita <1989013@xxxxxxxxxxxxxxxxxx>]

I suspect this is a glance_store issue, but keeping both components
until more research is done.

** Also affects: glance-store
   Importance: Undecided
       Status: New

** Summary changed:

- Large images uploaded but then deleted
+ s3: Large images uploaded but then deleted

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1989013

Title:
  s3: Large images uploaded but then deleted

Status in Glance:
  New
Status in glance_store:
  New

Bug description:
  We are trying to upload images to an on-premise minio-based S3
  compatible storage cluster. We have been using it for ~30GB images for
  months.  However when increasing the size to >30GB, the image is
  successfully uploaded and then glance seems to tell it to delete.

  Successful log looks like this:
  Sep 6, 2022 @ 08:40:22.812	1	1	rPR1E4MBBdpF5mC-rsYE	test_logs	-	ttcloud-image	ttcloud-image	CompleteMultipartUpload	CompleteMultipartUpload
  Sep 6, 2022 @ 08:39:39.038	1	1	q_R1E4MBBdpF5mC-A8YF	test_logs	-	ttcloud-image	ttcloud-image	PutObjectPart	PutObjectPart
  ...
  Whole bunch of PutObjectPart...

  
  Failure log looks like this:
  Sep 2, 2022 @ 18:11:46.067	1	1	PvTnAIMBBdpF5mC-XLv7	test_logs	-	ttcloud-image	ttcloud-image	DeleteObject	DeleteObject
  Sep 2, 2022 @ 18:11:45.987	1	1	PfTnAIMBBdpF5mC-XLur	test_logs	-	ttcloud-image	ttcloud-image	HeadObject	HeadObject
  Sep 2, 2022 @ 18:11:45.927	1	1	PPTnAIMBBdpF5mC-XLtw	test_logs	-	ttcloud-image	ttcloud-image	CompleteMultipartUpload	CompleteMultipartUpload
  Sep 2, 2022 @ 18:10:49.370	1	1	O_TmAIMBBdpF5mC-f7uE	test_logs	-	ttcloud-image	ttcloud-image	PutObjectPart	PutObjectPart

  
  We were able to get the 38GB image to work by increasing the chunk size from 10MB to 100MB.  However this does not help the 50GB or 100GB.

  The storage system is correctly responding to all S3 request with
  either 200 or 204 replies. It appears that something important is
  happening with the HeadObject which is then causing glance to send a
  DeleteObject.

  I've attached shortened versions of the S3 logs - I've deleted the
  majority of PutObjectPart entries to keep the size down.

  We're not finding any information. What is causing this behavior?

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1989013/+subscriptions