yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1890580] [NEW] HTTP 409 error from glance-api with swift backend

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Rajiv Mucheli <1890580@xxxxxxxxxxxxxxxxxx>
Date: Thu, 06 Aug 2020 11:53:39 -0000
Reply-to: Bug 1890580 <1890580@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Hi Team,

I would like to understand why and how does glance-api pod generate HTTP
409 errors, i did look into the available documentation and code but had
no conclusion. I referred the below :

https://github.com/openstack/glance/blob/stable/train/doc/source/user/glanceapi.rst
https://github.com/openstack/glance/blob/54329c6a21b0d3f845b09e79f710fc795976a175/releasenotes/source/locale/ja/LC_MESSAGES/releasenotes.po
https://bugs.launchpad.net/glance/+bug/1229823
https://docs.openstack.org/glance/pike/configuration/configuring.html#configuring-the-swift-storage-backend

I wonder the HTTP 409 conflicts occur due to the below flags, are the
api calls parallel or sequential for deletion ? i.e. a HTTP 409 means a
Conflict response status code, maybe the deletion calls dont wait until
the deletion is completed ? :

swift_store_large_object_size = 5120
swift_store_large_object_chunk_size = 512 (default is 200Mb (i dont see HTTP 409), would it generate HTTP 409 if its increased to 500Mb)

glance-api logs :

2020-07-18 01:55:46,627.627 52 ERROR glance.common.wsgi [req-236a9c8c-
396e-42a1-8987-f847923c7e13
f1083795e1da57ca00ff8c967ad0c3d80751fe341a1e64046869e0ae0770cc1d
7e49c7a15b4a4f149cae86a0c1366afa - ec213443e8834473b579f7bea9e8c194
ec213443e8834473b579f7bea9e8c194] Caught error: Container DELETE failed:
https://xxx:443/v1/AUTH_7e49c7a15b4a4f149cae86a0c1366afa/glance_1352ddc3
-12ba-4afe-9c89-304cefd90ef5 409 Conflict [first 60 chars of response]
b'<html><h1>Conflict</h1><p>There was a conflict when trying t':
swiftclient.exceptions.ClientException: Container DELETE failed:
https://xxx:443/v1/AUTH_7e49c7a15b4a4f149cae86a0c1366afa/glance_1352ddc3
-12ba-4afe-9c89-304cefd90ef5 409 Conflict [first 60 chars of response]
b'<html><h1>Conflict</h1><p>There was a conflict when trying t'

2020-07-18 01:55:46,707.707 52 INFO eventlet.wsgi.server [req-236a9c8c-
396e-42a1-8987-f847923c7e13
f1083795e1da57ca00ff8c967ad0c3d80751fe341a1e64046869e0ae0770cc1d
7e49c7a15b4a4f149cae86a0c1366afa - ec213443e8834473b579f7bea9e8c194
ec213443e8834473b579f7bea9e8c194] 10.46.14.92,100.85.0.29 - -
[18/Jul/2020 01:55:46] "DELETE /v2/images/1352ddc3-12ba-4afe-
9c89-304cefd90ef5 HTTP/1.1" 500 449 11.140969

Openstack Glance Version : Train
Glance-api.conf : https://github.com/sapcc/helm-charts/blob/master/openstack/glance/templates/etc/_glance-api.conf.tpl
Swift-api.conf 
Swift conf file : https://github.com/sapcc/helm-charts/blob/master/openstack/swift/templates/etc/_proxy-server.conf.tpl

** Affects: glance
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1890580

Title:
  HTTP 409 error from glance-api with swift backend

Status in Glance:
  New

Bug description:
  Hi Team,

  I would like to understand why and how does glance-api pod generate
  HTTP 409 errors, i did look into the available documentation and code
  but had no conclusion. I referred the below :

  https://github.com/openstack/glance/blob/stable/train/doc/source/user/glanceapi.rst
  https://github.com/openstack/glance/blob/54329c6a21b0d3f845b09e79f710fc795976a175/releasenotes/source/locale/ja/LC_MESSAGES/releasenotes.po
  https://bugs.launchpad.net/glance/+bug/1229823
  https://docs.openstack.org/glance/pike/configuration/configuring.html#configuring-the-swift-storage-backend

  I wonder the HTTP 409 conflicts occur due to the below flags, are the
  api calls parallel or sequential for deletion ? i.e. a HTTP 409 means
  a Conflict response status code, maybe the deletion calls dont wait
  until the deletion is completed ? :

  swift_store_large_object_size = 5120
  swift_store_large_object_chunk_size = 512 (default is 200Mb (i dont see HTTP 409), would it generate HTTP 409 if its increased to 500Mb)

  glance-api logs :

  2020-07-18 01:55:46,627.627 52 ERROR glance.common.wsgi [req-236a9c8c-
  396e-42a1-8987-f847923c7e13
  f1083795e1da57ca00ff8c967ad0c3d80751fe341a1e64046869e0ae0770cc1d
  7e49c7a15b4a4f149cae86a0c1366afa - ec213443e8834473b579f7bea9e8c194
  ec213443e8834473b579f7bea9e8c194] Caught error: Container DELETE
  failed:
  https://xxx:443/v1/AUTH_7e49c7a15b4a4f149cae86a0c1366afa/glance_1352ddc3
  -12ba-4afe-9c89-304cefd90ef5 409 Conflict [first 60 chars of response]
  b'<html><h1>Conflict</h1><p>There was a conflict when trying t':
  swiftclient.exceptions.ClientException: Container DELETE failed:
  https://xxx:443/v1/AUTH_7e49c7a15b4a4f149cae86a0c1366afa/glance_1352ddc3
  -12ba-4afe-9c89-304cefd90ef5 409 Conflict [first 60 chars of response]
  b'<html><h1>Conflict</h1><p>There was a conflict when trying t'

  2020-07-18 01:55:46,707.707 52 INFO eventlet.wsgi.server [req-
  236a9c8c-396e-42a1-8987-f847923c7e13
  f1083795e1da57ca00ff8c967ad0c3d80751fe341a1e64046869e0ae0770cc1d
  7e49c7a15b4a4f149cae86a0c1366afa - ec213443e8834473b579f7bea9e8c194
  ec213443e8834473b579f7bea9e8c194] 10.46.14.92,100.85.0.29 - -
  [18/Jul/2020 01:55:46] "DELETE /v2/images/1352ddc3-12ba-4afe-
  9c89-304cefd90ef5 HTTP/1.1" 500 449 11.140969

  Openstack Glance Version : Train
  Glance-api.conf : https://github.com/sapcc/helm-charts/blob/master/openstack/glance/templates/etc/_glance-api.conf.tpl
  Swift-api.conf 
  Swift conf file : https://github.com/sapcc/helm-charts/blob/master/openstack/swift/templates/etc/_proxy-server.conf.tpl

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1890580/+subscriptions