← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #93182

[Bug 2045769] [NEW] Image is not getting deleted even after image data is deleted from the rbd store

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

When glance is using rbd store, during image download, if delete call
for the same image is attempted, delete fails with InUseByStore raised
from the store but image data also gets deleted from the store and still
image remains in active state.

Steps to reproduce:

1. Create image:
$ glance image-create --name test-new-image --file gentoo_root.img --disk-format raw --container-format bare  

2. Download Image and simultaneously delete that image in other terminal:
$ glance image-download 1e1f9830-29e0-4901-8eff-ef05b7faf725 --file hhh.txt
[Errno 32] Corrupt image download. Hash was bc0b615f1a34f1bde273f496fa6246d33dbd760ac20e7687e55c1d06ab070907bfe087b2a25bceca3cd4ba819b096286d6e4f28a5f27398ed7afb0b0ba18721a expected e4f21997407b9cb0df347f6eba2feaeb14c19f15cf784da06b78e1d5ff776a419535c894dea10a859fa72bcb234e94ada0fc86de0ff127bf9280eede8d473edb

$ glance image-delete 1e1f9830-29e0-4901-8eff-ef05b7faf725
Unable to delete image '1e1f9830-29e0-4901-8eff-ef05b7faf725' because it is in use.

3. Try to Download the same Image again
$ glance image-download 1e1f9830-29e0-4901-8eff-ef05b7faf725 --file hhh.txt
Image 1e1f9830-29e0-4901-8eff-ef05b7faf725 has no data.


Log:

WARNING glance_store._drivers.rbd [None req-9d5aeaae-efc9-4d43-8496-2c742a52a4da admin admin] Remove image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 failed. It is in use.: rbd.ImageBusy: [errno 16] RBD image is busy (error removing image)
Dec 06 11:40:41 pdeore-bobdev devstack@g-api.service[813992]: WARNING glance.api.v2.images [None req-9d5aeaae-efc9-4d43-8496-2c742a52a4da admin admin] Image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 could not be deleted because it is in use: The image cannot be deleted because it is in use through the backend store outside of Glance.: glance_store.exceptions.InUseByStore: The image cannot be deleted because it is in use through the backend store outside of Glance.


Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: WARNING glance.image_cache.drivers.sqlite [None req-e71cfe15-f144-418b-b9f7-15afbece570b admin admin] ("Fetch of cache file failed (%(e)s), rolling back by moving '%(incomplete_path)s' to '%(invalid_path)s'", {'e': NotFound('RBD image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 does not exist'), 'incomplete_path': '/opt/stack/data/glance/cache/incomplete/c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4', 'invalid_path': '/opt/stack/data/glance/cache/invalid/c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4'}): glance_store.exceptions.NotFound: RBD image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 does not exist
Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache [None req-e71cfe15-f144-418b-b9f7-15afbece570b admin admin] Exception encountered while tee'ing image 'c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4' into cache: RBD image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 does not exist. Continuing with response.: glance_store.exceptions.NotFound: RBD image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 does not exist
Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache Traceback (most recent call last):
Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache   File "/usr/local/lib/python3.10/dist-packages/glance_store/_drivers/rbd.py", line 269, in __iter__
Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache     with rbd.Image(ioctx, self.name,
Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache   File "rbd.pyx", line 2894, in rbd.Image.__init__
Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache rbd.ImageNotFound: [errno 2] RBD image not found (error opening image b'c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4' at snapshot b'snap')
Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache 
Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache During handling of the above exception, another exception occurred:
Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache 
Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache Traceback (most recent call last):
Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache   File "/opt/stack/glance/glance/image_cache/__init__.py", line 352, in cache_tee_iter
Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache     for chunk in image_iter:
Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache   File "/opt/stack/glance/glance/notifier.py", line 412, in _get_chunk_data_iterator
Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache     for chunk in data:
Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache   File "/usr/local/lib/python3.10/dist-packages/glance_store/_drivers/rbd.py", line 282, in __iter__
Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache     raise exceptions.NotFound(
Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache glance_store.exceptions.NotFound: RBD image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 does not exist
Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache

** Affects: glance
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/2045769

Title:
  Image is not getting deleted even after image data is deleted from the
  rbd store

Status in Glance:
  New

Bug description:
  When glance is using rbd store, during image download, if delete call
  for the same image is attempted, delete fails with InUseByStore raised
  from the store but image data also gets deleted from the store and
  still image remains in active state.

  Steps to reproduce:

  1. Create image:
  $ glance image-create --name test-new-image --file gentoo_root.img --disk-format raw --container-format bare  

  2. Download Image and simultaneously delete that image in other terminal:
  $ glance image-download 1e1f9830-29e0-4901-8eff-ef05b7faf725 --file hhh.txt
  [Errno 32] Corrupt image download. Hash was bc0b615f1a34f1bde273f496fa6246d33dbd760ac20e7687e55c1d06ab070907bfe087b2a25bceca3cd4ba819b096286d6e4f28a5f27398ed7afb0b0ba18721a expected e4f21997407b9cb0df347f6eba2feaeb14c19f15cf784da06b78e1d5ff776a419535c894dea10a859fa72bcb234e94ada0fc86de0ff127bf9280eede8d473edb

  $ glance image-delete 1e1f9830-29e0-4901-8eff-ef05b7faf725
  Unable to delete image '1e1f9830-29e0-4901-8eff-ef05b7faf725' because it is in use.

  3. Try to Download the same Image again
  $ glance image-download 1e1f9830-29e0-4901-8eff-ef05b7faf725 --file hhh.txt
  Image 1e1f9830-29e0-4901-8eff-ef05b7faf725 has no data.


  Log:

  WARNING glance_store._drivers.rbd [None req-9d5aeaae-efc9-4d43-8496-2c742a52a4da admin admin] Remove image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 failed. It is in use.: rbd.ImageBusy: [errno 16] RBD image is busy (error removing image)
  Dec 06 11:40:41 pdeore-bobdev devstack@g-api.service[813992]: WARNING glance.api.v2.images [None req-9d5aeaae-efc9-4d43-8496-2c742a52a4da admin admin] Image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 could not be deleted because it is in use: The image cannot be deleted because it is in use through the backend store outside of Glance.: glance_store.exceptions.InUseByStore: The image cannot be deleted because it is in use through the backend store outside of Glance.

  
  Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: WARNING glance.image_cache.drivers.sqlite [None req-e71cfe15-f144-418b-b9f7-15afbece570b admin admin] ("Fetch of cache file failed (%(e)s), rolling back by moving '%(incomplete_path)s' to '%(invalid_path)s'", {'e': NotFound('RBD image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 does not exist'), 'incomplete_path': '/opt/stack/data/glance/cache/incomplete/c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4', 'invalid_path': '/opt/stack/data/glance/cache/invalid/c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4'}): glance_store.exceptions.NotFound: RBD image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 does not exist
  Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache [None req-e71cfe15-f144-418b-b9f7-15afbece570b admin admin] Exception encountered while tee'ing image 'c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4' into cache: RBD image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 does not exist. Continuing with response.: glance_store.exceptions.NotFound: RBD image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 does not exist
  Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache Traceback (most recent call last):
  Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache   File "/usr/local/lib/python3.10/dist-packages/glance_store/_drivers/rbd.py", line 269, in __iter__
  Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache     with rbd.Image(ioctx, self.name,
  Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache   File "rbd.pyx", line 2894, in rbd.Image.__init__
  Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache rbd.ImageNotFound: [errno 2] RBD image not found (error opening image b'c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4' at snapshot b'snap')
  Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache 
  Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache During handling of the above exception, another exception occurred:
  Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache 
  Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache Traceback (most recent call last):
  Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache   File "/opt/stack/glance/glance/image_cache/__init__.py", line 352, in cache_tee_iter
  Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache     for chunk in image_iter:
  Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache   File "/opt/stack/glance/glance/notifier.py", line 412, in _get_chunk_data_iterator
  Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache     for chunk in data:
  Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache   File "/usr/local/lib/python3.10/dist-packages/glance_store/_drivers/rbd.py", line 282, in __iter__
  Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache     raise exceptions.NotFound(
  Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache glance_store.exceptions.NotFound: RBD image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 does not exist
  Dec 06 11:40:44 pdeore-bobdev devstack@g-api.service[813991]: ERROR glance.image_cache

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/2045769/+subscriptions
  Thread Previous • Date Previous • • Thread Next