yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1077045] Re: Cannot delete images when rbd store is used

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Brian Waldon <bcwaldon+launchpad@xxxxxxxxx>
Date: Mon, 04 Mar 2013 22:54:58 -0000
Reply-to: Bug 1077045 <1077045@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: glance
       Status: Incomplete => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1077045

Title:
  Cannot delete images when rbd store is used

Status in OpenStack Image Registry and Delivery Service (Glance):
  Invalid

Bug description:
  I have a multinode installation of openstack
  ubuntu 12.04 using the ubuntu cloud repo for Folsom codebase with quantum
  Glance Version: 2012.2-0ubuntu2~cloud0
  Nova version : 2012.2-0ubuntu5~cloud0
  ceph version 53
  apt-get update apt-get upgrade completed on all nodes (as of 9th Nov 2012)

  1 only  Cloud controller system
  2 only cloud compute nodes nodes
  4 only ceph nodes (3 monitors)
  5 only swift nodes (1 proxy)

  I have configured ceph , nova-volume and Glance as per this documentaton for setup in openstack:
  http://www.sebastien-han.fr/blog/2012/06/10/introducing-ceph-to-openstack/

  problem:
  when an RBD store is specified for glance, i am unable to delete or clear images. I can successfully add/update or use image only
  If i configure glance with a file backend i can add/update/delete/clear as normal
  If i configure glance with a swift backend i can add/update/delete/clear as normal 

  error generated by client:

   glance delete fb11234a-9e85-4496-a999-186cc71ca2da
  Delete image fb11234a-9e85-4496-a999-186cc71ca2da? [y/N] y
  Request returned failure status.
  None
  HTTPInternalServerError (HTTP 500)

  steps to reproduce:
  Configure to Glance to use ceph backend
  upload an image via glance add
  glance delete <imageID> or glance clear(for all images) - both will generate the same error

  workarounds: - none
  alternative configuration :
  Configure glance to use Filesystem or swift backends for glance for image storage

  glance api.log fragment:
  2012-11-09 15:58:28 5780 INFO sqlalchemy.engine.base.Engine [-] SELECT images.created_at AS images_created_at, images.updated_at AS images_updated_at, image s.deleted_at AS images_deleted_at, images.deleted AS images_deleted, images.id AS images_id, images.name AS images_name, images.disk_format AS images_disk_f ormat, images.container_format AS images_container_format, images.size AS images_size, images.status AS images_status, images.is_public AS images_is_public,  images.location AS images_location, images.checksum AS images_checksum, images.min_disk AS images_min_disk, images.min_ram AS images_min_ram, images.owner  AS images_owner, images.protected AS images_protected, image_properties_1.created_at AS image_properties_1_created_at, image_properties_1.updated_at AS imag e_properties_1_updated_at, image_properties_1.deleted_at AS image_properties_1_deleted_at, image_properties_1.deleted AS image_properties_1_deleted, image_p roperties_1.id AS image_properties_1_id, image_properties_1.image_id AS image_properties_1_image_id, image_properties_1.name AS image_properties_1_name, ima ge_properties_1.value AS image_properties_1_value
  FROM images LEFT OUTER JOIN image_properties AS image_properties_1 ON images.id = image_properties_1.image_id
  WHERE images.id = %s
  2012-11-09 15:58:28 5780 INFO sqlalchemy.engine.base.Engine [-] ('fb11234a-9e85-4496-a999-186cc71ca2da',)
  2012-11-09 15:58:28 5780 DEBUG sqlalchemy.engine.base.Engine [-] Col ('images_created_at', 'images_updated_at', 'images_deleted_at', 'images_deleted', 'imag es_id', 'images_name', 'images_disk_format', 'images_container_format', 'images_size', 'images_status', 'images_is_public', 'images_location', 'images_check sum', 'images_min_disk', 'images_min_ram', 'images_owner', 'images_protected', 'image_properties_1_created_at', 'image_properties_1_updated_at', 'image_prop erties_1_deleted_at', 'image_properties_1_deleted', 'image_properties_1_id', 'image_properties_1_image_id', 'image_properties_1_name', 'image_properties_1_v alue') __init__ /usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py:2789
  2012-11-09 15:58:28 5780 DEBUG sqlalchemy.engine.base.Engine [-] Row (datetime.datetime(2012, 11, 9, 14, 38, 37), datetime.datetime(2012, 11, 9, 15, 40, 57) , None, 0, 'fb11234a-9e85-4496-a999-186cc71ca2da', 'glanceload-ubu2-ceph', 'qcow2', 'ovf', 1133772800L, 'active', 0, 'rbd://b2125094-25b7-4009-9431-4946a921 b778/images/fb11234a-9e85-4496-a999-186cc71ca2da/snap', 'd8d6e78896d009899f5fbd66bdafb09e', 0L, 0L, '2cea4cff2003475f954b031049dfd66c', 0, None, None, None,  None, None, None, None, None) process_rows /usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py:3194
  2012-11-09 15:58:28 INFO glance.registry.api.v1.images [ea996b6e-7678-46bb-91c8-f76515beb75b 0bab1d02fee141799787115a8bc219e4 2cea4cff2003475f954b031049dfd6 6c] Successfully retrieved image fb11234a-9e85-4496-a999-186cc71ca2da
  2012-11-09 15:58:28 5780 INFO sqlalchemy.engine.base.Engine [-] SELECT images.created_at AS images_created_at, images.updated_at AS images_updated_at, image s.deleted_at AS images_deleted_at, images.deleted AS images_deleted, images.id AS images_id, images.name AS images_name, images.disk_format AS images_disk_f ormat, images.container_format AS images_container_format, images.size AS images_size, images.status AS images_status, images.is_public AS images_is_public,  images.location AS images_location, images.checksum AS images_checksum, images.min_disk AS images_min_disk, images.min_ram AS images_min_ram, images.owner  AS images_owner, images.protected AS images_protected, image_properties_1.created_at AS image_properties_1_created_at, image_properties_1.updated_at AS imag e_properties_1_updated_at, image_properties_1.deleted_at AS image_properties_1_deleted_at, image_properties_1.deleted AS image_properties_1_deleted, image_p roperties_1.id AS image_properties_1_id, image_properties_1.image_id AS image_properties_1_image_id, image_properties_1.name AS image_properties_1_name, ima ge_properties_1.value AS image_properties_1_value
  FROM images LEFT OUTER JOIN image_properties AS image_properties_1 ON images.id = image_properties_1.image_id
  WHERE images.id = %s
  2012-11-09 15:58:28 5780 INFO sqlalchemy.engine.base.Engine [-] ('fb11234a-9e85-4496-a999-186cc71ca2da',)
  2012-11-09 15:58:28 5780 DEBUG sqlalchemy.engine.base.Engine [-] Col ('images_created_at', 'images_updated_at', 'images_deleted_at', 'images_deleted', 'imag es_id', 'images_name', 'images_disk_format', 'images_container_format', 'images_size', 'images_status', 'images_is_public', 'images_location', 'images_check sum', 'images_min_disk', 'images_min_ram', 'images_owner', 'images_protected', 'image_properties_1_created_at', 'image_properties_1_updated_at', 'image_prop erties_1_deleted_at', 'image_properties_1_deleted', 'image_properties_1_id', 'image_properties_1_image_id', 'image_properties_1_name', 'image_properties_1_v alue') __init__ /usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py:2789
  2012-11-09 15:58:28 5780 DEBUG sqlalchemy.engine.base.Engine [-] Row (datetime.datetime(2012, 11, 9, 14, 38, 37), datetime.datetime(2012, 11, 9, 15, 40, 57) , None, 0, 'fb11234a-9e85-4496-a999-186cc71ca2da', 'glanceload-ubu2-ceph', 'qcow2', 'ovf', 1133772800L, 'active', 0, 'rbd://b2125094-25b7-4009-9431-4946a921 b778/images/fb11234a-9e85-4496-a999-186cc71ca2da/snap', 'd8d6e78896d009899f5fbd66bdafb09e', 0L, 0L, '2cea4cff2003475f954b031049dfd66c', 0, None, None, None,  None, None, None, None, None) process_rows /usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py:3194
  2012-11-09 15:58:28 INFO glance.registry.api.v1.images [3e5adfc4-15ad-46db-a8de-1357c20d13fa 0bab1d02fee141799787115a8bc219e4 2cea4cff2003475f954b031049dfd6 6c] Successfully retrieved image fb11234a-9e85-4496-a999-186cc71ca2da

  glance-registry log fragment:
  2012-11-09 15:58:27 7009 DEBUG glance.api.middleware.version_negotiation [-] Determining version of request: HEAD /v1/images/fb11234a-9e85-4496-a999-186cc71ca2da Accept:  process_request /usr/lib/python2.7/dist-packages/glance/api/middleware/version_negotiation.py:45
  2012-11-09 15:58:27 7009 DEBUG glance.api.middleware.version_negotiation [-] Using url versioning process_request /usr/lib/python2.7/dist-packages/glance/api/middleware/version_negotiation.py:58
  2012-11-09 15:58:27 7009 DEBUG glance.api.middleware.version_negotiation [-] Matched version: v1 process_request /usr/lib/python2.7/dist-packages/glance/api/middleware/version_negotiation.py:70
  2012-11-09 15:58:27 7009 DEBUG glance.api.middleware.version_negotiation [-] new uri /v1/images/fb11234a-9e85-4496-a999-186cc71ca2da process_request /usr/lib/python2.7/dist-packages/glance/api/middleware/version_negotiation.py:71
  2012-11-09 15:58:28 DEBUG glance.api.policy [f18a4e64-df12-41a1-9977-c77ebc238519 0bab1d02fee141799787115a8bc219e4 2cea4cff2003475f954b031049dfd66c] Loaded policy rules: {u'default': [], u'manage_image_cache': [[u'role:admin']]} load_rules /usr/lib/python2.7/dist-packages/glance/api/policy.py:63
  2012-11-09 15:58:28 DEBUG glance.common.client [f18a4e64-df12-41a1-9977-c77ebc238519 0bab1d02fee141799787115a8bc219e4 2cea4cff2003475f954b031049dfd66c] Constructed URL: http://172.28.137.65:9191/images/fb11234a-9e85-4496-a999-186cc71ca2da _construct_url /usr/lib/python2.7/dist-packages/glance/common/client.py:464
  2012-11-09 15:58:28 DEBUG glance.registry.client [f18a4e64-df12-41a1-9977-c77ebc238519 0bab1d02fee141799787115a8bc219e4 2cea4cff2003475f954b031049dfd66c] Registry request GET /images/fb11234a-9e85-4496-a999-186cc71ca2da HTTP 200 request id req-ea996b6e-7678-46bb-91c8-f76515beb75b do_request /usr/lib/python2.7/dist-packages/glance/registry/client.py:94
  2012-11-09 15:58:28 7009 DEBUG glance.api.middleware.version_negotiation [-] Determining version of request: DELETE /v1/images/fb11234a-9e85-4496-a999-186cc71ca2da Accept:  process_request /usr/lib/python2.7/dist-packages/glance/api/middleware/version_negotiation.py:45
  2012-11-09 15:58:28 7009 DEBUG glance.api.middleware.version_negotiation [-] Using url versioning process_request /usr/lib/python2.7/dist-packages/glance/api/middleware/version_negotiation.py:58
  2012-11-09 15:58:28 7009 DEBUG glance.api.middleware.version_negotiation [-] Matched version: v1 process_request /usr/lib/python2.7/dist-packages/glance/api/middleware/version_negotiation.py:70
  2012-11-09 15:58:28 7009 DEBUG glance.api.middleware.version_negotiation [-] new uri /v1/images/fb11234a-9e85-4496-a999-186cc71ca2da process_request /usr/lib/python2.7/dist-packages/glance/api/middleware/version_negotiation.py:71
  2012-11-09 15:58:28 DEBUG glance.api.policy [629ba27d-3fb9-4b19-aa8f-cf589fac4751 0bab1d02fee141799787115a8bc219e4 2cea4cff2003475f954b031049dfd66c] Loaded policy rules: {u'default': [], u'manage_image_cache': [[u'role:admin']]} load_rules /usr/lib/python2.7/dist-packages/glance/api/policy.py:63
  2012-11-09 15:58:28 DEBUG glance.common.client [629ba27d-3fb9-4b19-aa8f-cf589fac4751 0bab1d02fee141799787115a8bc219e4 2cea4cff2003475f954b031049dfd66c] Constructed URL: http://172.28.137.65:9191/images/fb11234a-9e85-4496-a999-186cc71ca2da _construct_url /usr/lib/python2.7/dist-packages/glance/common/client.py:464
  2012-11-09 15:58:28 DEBUG glance.registry.client [629ba27d-3fb9-4b19-aa8f-cf589fac4751 0bab1d02fee141799787115a8bc219e4 2cea4cff2003475f954b031049dfd66c] Registry request GET /images/fb11234a-9e85-4496-a999-186cc71ca2da HTTP 200 request id req-3e5adfc4-15ad-46db-a8de-1357c20d13fa do_request /usr/lib/python2.7/dist-packages/glance/registry/client.py:94

  any help or guidance as to how we can resolve this or work around this
  issue with glance and ceph would be really appreciated

  thanks

  Steve A

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1077045/+subscriptions