yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1789748] [NEW] Glance should set container ACLs when using swift store

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: David Ames <david.ames@xxxxxxxxxxxxx>
Date: Wed, 29 Aug 2018 22:00:16 -0000
Reply-to: Bug 1789748 <1789748@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

With glance v2 and swift as the store, glance is unable to delete images.
Tested with Queens and Rocky.

Code snippit:
import glanceclient.v2 as glance_clientv2
glance = glance_clientv2.Client("2", session=keystone.session)
image = glance.images.create(name=image_name, disk_format="qcow2", visibility="public", container_format="bare")
glance.images.upload(image.id, open(local_path, 'rb'))
glance.images.delete(image.id)

Throws:
glanceclient.exc.HTTPInternalServerError: 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

Leads to the following error in glance-api.log

2018-08-29 21:49:03.367 21964 INFO swiftclient [req-4f5b43a9-c3f1-4c71-b351-d08c2d221860 094897deb7424a4da2caccacc8c9d4ff ce38b0769f8c4037b641f8296bfd858b - bbda55ddd16e4faca419e2a13a113176 bbda55ddd16e4faca419e2a13a113176] REQ: curl -i http://10.5.0.34:8080/v1/AUTH_d213d2090cf948a0ba07e14491d58fc7/glance/02a6c228-249c-41f4-b8e3-04220a1d7529 -I -H "X-Auth-Token: gAAAAABbhxTOAmCn..."
2018-08-29 21:49:03.368 21964 INFO swiftclient [req-4f5b43a9-c3f1-4c71-b351-d08c2d221860 094897deb7424a4da2caccacc8c9d4ff ce38b0769f8c4037b641f8296bfd858b - bbda55ddd16e4faca419e2a13a113176 bbda55ddd16e4faca419e2a13a113176] RESP STATUS: 403 Forbidden
2018-08-29 21:49:03.368 21964 INFO swiftclient [req-4f5b43a9-c3f1-4c71-b351-d08c2d221860 094897deb7424a4da2caccacc8c9d4ff ce38b0769f8c4037b641f8296bfd858b - bbda55ddd16e4faca419e2a13a113176 bbda55ddd16e4faca419e2a13a113176] RESP HEADERS: {u'Date': u'Wed, 29 Aug 2018 21:49:03 GMT', u'Content-Length': u'73', u'Content-Type': u'text/html; charset=UTF-8', u'X-Openstack-Request-Id': u'tx3c78666a736e42a4b32a3-005b8714ce', u'X-Trans-Id': u'tx3c78666a736e42a4b32a3-005b8714ce'}
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi [req-4f5b43a9-c3f1-4c71-b351-d08c2d221860 094897deb7424a4da2caccacc8c9d4ff ce38b0769f8c4037b641f8296bfd858b - bbda55ddd16e4faca419e2a13a113176 bbda55ddd16e4faca419e2a13a113176] Caught error: Object HEAD failed: http://10.5.0.34:8080/v1/AUTH_d213d2090cf948a0ba07e14491d58fc7/glance/02a6c228-249c-41f4-b8e3-04220a1d7529 403 Forbidden: ClientException: Object HEAD failed: http://10.5.0.34:8080/v1/AUTH_d213d2090cf948a0ba07e14491d58fc7/glance/02a6c228-249c-41f4-b8e3-04220a1d7529 403 Forbidden
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi Traceback (most recent call last):
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 1274, in __call__
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     request, **action_args)
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 1317, in dispatch
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     return method(*args, **kwargs)
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance/common/utils.py", line 417, in wrapped
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     return func(self, req, *args, **kwargs)
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance/api/v2/images.py", line 343, in delete
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     image.delete()
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance/domain/proxy.py", line 189, in delete
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     self.base.delete()
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance/domain/proxy.py", line 189, in delete
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     self.base.delete()
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance/api/policy.py", line 172, in delete
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     return self.image.delete()
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance/domain/proxy.py", line 189, in delete
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     self.base.delete()
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance/location.py", line 431, in delete
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     location)
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance/common/store_utils.py", line 123, in delete_image_location_from_backend
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     safe_delete_from_backend(context, image_id, location)
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance/common/store_utils.py", line 56, in safe_delete_from_backend
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     context=context)
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance_store/backend.py", line 409, in delete_from_backend
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     return store.delete(loc, context=context)
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance_store/capabilities.py", line 225, in op_checker
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     return store_op_fun(store, *args, **kwargs)
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance_store/_drivers/swift/store.py", line 1093, in delete
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     location.container, location.obj)
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1826, in head_object
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     query_string=query_string)
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1722, in _retry
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     service_token=self.service_token, **kwargs)
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1252, in head_object
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     raise ClientException.from_response(resp, 'Object HEAD failed', body)
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi ClientException: Object HEAD failed: http://10.5.0.34:8080/v1/AUTH_d213d2090cf948a0ba07e14491d58fc7/glance/02a6c228-249c-41f4-b8e3-04220a1d7529 403 Forbidden
2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi
2018-08-29 21:49:03.376 21964 INFO eventlet.wsgi.server [req-4f5b43a9-c3f1-4c71-b351-d08c2d221860 094897deb7424a4da2caccacc8c9d4ff ce38b0769f8c4037b641f8296bfd858b - bbda55ddd16e4faca419e2a13a113176 bbda55ddd16e4faca419e2a13a113176] 10.5.0.12 - - [29/Aug/2018 21:49:03] "DELETE /v2/images/02a6c228-249c-41f4-b8e3-04220a1d7529 HTTP/1.1" 500 454 2.932388


Tested with openstack client for create and delete with same results.

As the glance user using the swift client directly, the initial image
part is not readable:

$ swift list glance
02a6c228-249c-41f4-b8e3-04220a1d7529
02a6c228-249c-41f4-b8e3-04220a1d7529-00001

$ swift stat glance 02a6c228-249c-41f4-b8e3-04220a1d7529-00001
               Account: AUTH_d213d2090cf948a0ba07e14491d58fc7
             Container: glance
                Object: 02a6c228-249c-41f4-b8e3-04220a1d7529-00001
          Content Type: application/octet-stream
        Content Length: 12716032
         Last Modified: Wed, 29 Aug 2018 21:49:00 GMT
                  ETag: 443b7623e27ecf03dc9e01ee93f67afe
         Accept-Ranges: bytes
           X-Timestamp: 1535579339.25431
            X-Trans-Id: txd57d93cc856846f98015f-005b87155d
X-Openstack-Request-Id: txd57d93cc856846f98015f-005b87155d

$ swift stat glance 02a6c228-249c-41f4-b8e3-04220a1d7529
Object HEAD failed: http://10.5.0.34:8080/v1/AUTH_d213d2090cf948a0ba07e14491d58fc7/glance/02a6c228-249c-41f4-b8e3-04220a1d7529 403 Forbidden
Failed Transaction ID: tx088744539df247ec8e2c0-005b871566

Note the glance container has no ACLs set:
$ swift stat glance
               Account: AUTH_d213d2090cf948a0ba07e14491d58fc7
             Container: glance
               Objects: 2
                 Bytes: 12716032
              Read ACL:
             Write ACL:
               Sync To:
              Sync Key:
         Accept-Ranges: bytes
      X-Storage-Policy: Policy-0
         Last-Modified: Wed, 29 Aug 2018 21:49:00 GMT
           X-Timestamp: 1535579339.45744
            X-Trans-Id: txdc539510bdc84b7f84f28-005b87154b
          Content-Type: application/json; charset=utf-8
X-Openstack-Request-Id: txdc539510bdc84b7f84f28-005b87154b

After adding read ACLs to the container, the image is deleteable
swift post glance --read-acl ".r:*,.rlistings"

Glance v2 should create its swift store container with correct read
ACLS.

** Affects: glance
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1789748

Title:
  Glance should set container ACLs when using swift store

Status in Glance:
  New

Bug description:
  With glance v2 and swift as the store, glance is unable to delete images.
  Tested with Queens and Rocky.

  Code snippit:
  import glanceclient.v2 as glance_clientv2
  glance = glance_clientv2.Client("2", session=keystone.session)
  image = glance.images.create(name=image_name, disk_format="qcow2", visibility="public", container_format="bare")
  glance.images.upload(image.id, open(local_path, 'rb'))
  glance.images.delete(image.id)

  Throws:
  glanceclient.exc.HTTPInternalServerError: 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

  Leads to the following error in glance-api.log

  2018-08-29 21:49:03.367 21964 INFO swiftclient [req-4f5b43a9-c3f1-4c71-b351-d08c2d221860 094897deb7424a4da2caccacc8c9d4ff ce38b0769f8c4037b641f8296bfd858b - bbda55ddd16e4faca419e2a13a113176 bbda55ddd16e4faca419e2a13a113176] REQ: curl -i http://10.5.0.34:8080/v1/AUTH_d213d2090cf948a0ba07e14491d58fc7/glance/02a6c228-249c-41f4-b8e3-04220a1d7529 -I -H "X-Auth-Token: gAAAAABbhxTOAmCn..."
  2018-08-29 21:49:03.368 21964 INFO swiftclient [req-4f5b43a9-c3f1-4c71-b351-d08c2d221860 094897deb7424a4da2caccacc8c9d4ff ce38b0769f8c4037b641f8296bfd858b - bbda55ddd16e4faca419e2a13a113176 bbda55ddd16e4faca419e2a13a113176] RESP STATUS: 403 Forbidden
  2018-08-29 21:49:03.368 21964 INFO swiftclient [req-4f5b43a9-c3f1-4c71-b351-d08c2d221860 094897deb7424a4da2caccacc8c9d4ff ce38b0769f8c4037b641f8296bfd858b - bbda55ddd16e4faca419e2a13a113176 bbda55ddd16e4faca419e2a13a113176] RESP HEADERS: {u'Date': u'Wed, 29 Aug 2018 21:49:03 GMT', u'Content-Length': u'73', u'Content-Type': u'text/html; charset=UTF-8', u'X-Openstack-Request-Id': u'tx3c78666a736e42a4b32a3-005b8714ce', u'X-Trans-Id': u'tx3c78666a736e42a4b32a3-005b8714ce'}
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi [req-4f5b43a9-c3f1-4c71-b351-d08c2d221860 094897deb7424a4da2caccacc8c9d4ff ce38b0769f8c4037b641f8296bfd858b - bbda55ddd16e4faca419e2a13a113176 bbda55ddd16e4faca419e2a13a113176] Caught error: Object HEAD failed: http://10.5.0.34:8080/v1/AUTH_d213d2090cf948a0ba07e14491d58fc7/glance/02a6c228-249c-41f4-b8e3-04220a1d7529 403 Forbidden: ClientException: Object HEAD failed: http://10.5.0.34:8080/v1/AUTH_d213d2090cf948a0ba07e14491d58fc7/glance/02a6c228-249c-41f4-b8e3-04220a1d7529 403 Forbidden
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi Traceback (most recent call last):
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 1274, in __call__
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     request, **action_args)
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 1317, in dispatch
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     return method(*args, **kwargs)
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance/common/utils.py", line 417, in wrapped
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     return func(self, req, *args, **kwargs)
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance/api/v2/images.py", line 343, in delete
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     image.delete()
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance/domain/proxy.py", line 189, in delete
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     self.base.delete()
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance/domain/proxy.py", line 189, in delete
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     self.base.delete()
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance/api/policy.py", line 172, in delete
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     return self.image.delete()
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance/domain/proxy.py", line 189, in delete
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     self.base.delete()
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance/location.py", line 431, in delete
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     location)
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance/common/store_utils.py", line 123, in delete_image_location_from_backend
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     safe_delete_from_backend(context, image_id, location)
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance/common/store_utils.py", line 56, in safe_delete_from_backend
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     context=context)
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance_store/backend.py", line 409, in delete_from_backend
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     return store.delete(loc, context=context)
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance_store/capabilities.py", line 225, in op_checker
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     return store_op_fun(store, *args, **kwargs)
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/glance_store/_drivers/swift/store.py", line 1093, in delete
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     location.container, location.obj)
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1826, in head_object
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     query_string=query_string)
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1722, in _retry
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     service_token=self.service_token, **kwargs)
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi   File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1252, in head_object
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi     raise ClientException.from_response(resp, 'Object HEAD failed', body)
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi ClientException: Object HEAD failed: http://10.5.0.34:8080/v1/AUTH_d213d2090cf948a0ba07e14491d58fc7/glance/02a6c228-249c-41f4-b8e3-04220a1d7529 403 Forbidden
  2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi
  2018-08-29 21:49:03.376 21964 INFO eventlet.wsgi.server [req-4f5b43a9-c3f1-4c71-b351-d08c2d221860 094897deb7424a4da2caccacc8c9d4ff ce38b0769f8c4037b641f8296bfd858b - bbda55ddd16e4faca419e2a13a113176 bbda55ddd16e4faca419e2a13a113176] 10.5.0.12 - - [29/Aug/2018 21:49:03] "DELETE /v2/images/02a6c228-249c-41f4-b8e3-04220a1d7529 HTTP/1.1" 500 454 2.932388

  
  Tested with openstack client for create and delete with same results.

  As the glance user using the swift client directly, the initial image
  part is not readable:

  $ swift list glance
  02a6c228-249c-41f4-b8e3-04220a1d7529
  02a6c228-249c-41f4-b8e3-04220a1d7529-00001

  $ swift stat glance 02a6c228-249c-41f4-b8e3-04220a1d7529-00001
                 Account: AUTH_d213d2090cf948a0ba07e14491d58fc7
               Container: glance
                  Object: 02a6c228-249c-41f4-b8e3-04220a1d7529-00001
            Content Type: application/octet-stream
          Content Length: 12716032
           Last Modified: Wed, 29 Aug 2018 21:49:00 GMT
                    ETag: 443b7623e27ecf03dc9e01ee93f67afe
           Accept-Ranges: bytes
             X-Timestamp: 1535579339.25431
              X-Trans-Id: txd57d93cc856846f98015f-005b87155d
  X-Openstack-Request-Id: txd57d93cc856846f98015f-005b87155d

  $ swift stat glance 02a6c228-249c-41f4-b8e3-04220a1d7529
  Object HEAD failed: http://10.5.0.34:8080/v1/AUTH_d213d2090cf948a0ba07e14491d58fc7/glance/02a6c228-249c-41f4-b8e3-04220a1d7529 403 Forbidden
  Failed Transaction ID: tx088744539df247ec8e2c0-005b871566

  Note the glance container has no ACLs set:
  $ swift stat glance
                 Account: AUTH_d213d2090cf948a0ba07e14491d58fc7
               Container: glance
                 Objects: 2
                   Bytes: 12716032
                Read ACL:
               Write ACL:
                 Sync To:
                Sync Key:
           Accept-Ranges: bytes
        X-Storage-Policy: Policy-0
           Last-Modified: Wed, 29 Aug 2018 21:49:00 GMT
             X-Timestamp: 1535579339.45744
              X-Trans-Id: txdc539510bdc84b7f84f28-005b87154b
            Content-Type: application/json; charset=utf-8
  X-Openstack-Request-Id: txdc539510bdc84b7f84f28-005b87154b

  After adding read ACLs to the container, the image is deleteable
  swift post glance --read-acl ".r:*,.rlistings"

  Glance v2 should create its swift store container with correct read
  ACLS.

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1789748/+subscriptions