← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #95965

[Bug 2109428] Re: Image delete fails in tempest nova-ceph-multistore job with: "The image cannot be deleted because it is in use through the backend store outside of Glance"

  Thread Previous • Date Previous • • Thread Next 
I haven't seen this issue in nova CI results in the last 7 days so I'm
closing this as fixed.

❯ logsearch storedsearch --days 7 bug-2109428-glance-cannot-delete-image-in-use
Running stored search:
  bug-2109428-glance-cannot-delete-image-in-use:
    files:
    - controller/logs/screen-n-api.txt
    job-groups:
    - nova-devstack
    project: openstack/nova
    regex:
    - The image cannot be deleted because it is in use through the backend store outside
      of Glance.
    result: FAILURE
  
Found builds:
+----------------------------------+---------------------+-----------------+-----------------------------------+---------------------+--------------------------------------------+
| uuid                             | finished            | pipeline        | review                            | branch              | job                                        |
+----------------------------------+---------------------+-----------------+-----------------------------------+---------------------+--------------------------------------------+
| d8eb68d0def24d0f858f0a7696308395 | 2025-06-03T14:27:54 | check           | https://review.opendev.org/948450 | master              | nova-grenade-multinode                     |
| a92b9d4ac7af432ea8e2554791bd1944 | 2025-05-31T08:53:49 | periodic-weekly | None                              | stable/2025.1       | tempest-integrated-compute-centos-9-stream |
| 5dc534073c364921a2a0c039b67ed343 | 2025-05-31T08:43:54 | periodic-weekly | None                              | unmaintained/zed    | tempest-integrated-compute-centos-9-stream |
| 8e96f5d1817d4a2998982e2f9fded2be | 2025-05-31T08:42:38 | periodic-weekly | None                              | unmaintained/2023.1 | tempest-integrated-compute-centos-9-stream |
| 4774df90113d4d7681c8963264eb67cd | 2025-05-31T08:26:44 | periodic-weekly | None                              | stable/2024.1       | tempest-integrated-compute-centos-9-stream |
| 29277ea6287d4b8c9df09429e2c2d46f | 2025-05-31T08:15:58 | periodic-weekly | None                              | stable/2024.2       | tempest-integrated-compute-centos-9-stream |
| a43b0392f2954e4693c00169aef3a35a | 2025-05-30T02:33:53 | check           | https://review.opendev.org/951226 | master              | nova-grenade-multinode                     |
| 7a13ba355c27400faa7b79aafd3cc7d2 | 2025-05-29T09:57:11 | check           | https://review.opendev.org/950623 | master              | nova-next                                  |
| ab6f4741f911428ebc208a26823ebacc | 2025-05-29T08:23:24 | check           | https://review.opendev.org/949754 | master              | grenade-skip-level-always                  |
| 20301661f37941ceb1a4a2a588296291 | 2025-05-29T08:08:32 | check           | https://review.opendev.org/949754 | master              | tempest-integrated-compute                 |
| 4cd8dfae211443cb8f640afddc88896a | 2025-05-29T09:09:42 | check           | https://review.opendev.org/949754 | master              | nova-ceph-multistore                       |
| e9f742f169d24b84a85ac9a10107de87 | 2025-05-29T08:12:06 | check           | https://review.opendev.org/949754 | master              | nova-grenade-multinode                     |
| f74784093a8946dab2a15f95ab342bec | 2025-05-29T07:55:15 | check           | https://review.opendev.org/949754 | master              | nova-lvm                                   |
| 7bf0c69b97d5417da026346dda011200 | 2025-05-29T07:50:27 | check           | https://review.opendev.org/949754 | master              | nova-multi-cell                            |
| 662427e2ea804c44a248596f56f4a5c9 | 2025-05-29T07:21:28 | check           | https://review.opendev.org/949754 | master              | tempest-ipv6-only                          |
| aaf0fb9d955f4943adb49493b1fe74b8 | 2025-05-28T13:26:55 | check           | https://review.opendev.org/950623 | master              | nova-grenade-multinode                     |
| ab3ce94011634332a7f1ca8187fbba98 | 2025-05-28T07:09:08 | check           | https://review.opendev.org/951084 | master              | nova-next                                  |
| 0e3046fc797c4a74a1ec5930ecf7e581 | 2025-05-27T23:11:28 | check           | https://review.opendev.org/942973 | master              | nova-multi-cell                            |
+----------------------------------+---------------------+-----------------+-----------------------------------+---------------------+--------------------------------------------+
Downloading logs:
Builds with matching logs 0/18:
+------+----------+----------+--------+--------+-----+
| uuid | finished | pipeline | review | branch | job |
+------+----------+----------+--------+--------+-----+
+------+----------+----------+--------+--------+-----+


** Changed in: glance
       Status: New => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/2109428

Title:
  Image delete fails in tempest nova-ceph-multistore job with: "The
  image cannot be deleted because it is in use through the backend store
  outside of Glance"

Status in Glance:
  Fix Released

Bug description:
  Example hits from the last 14 days:

  ❯ logsearch log --project openstack/nova --job nova-ceph-multistore
  --result FAILURE --days 14  "The image cannot be deleted because it is
  in use through the backend store outside of Glance"

  Builds with matching logs 5/15:
  +----------------------------------+---------------------+----------+-----------------------------------+--------+
  | uuid                             | finished            | pipeline | review                            | branch |
  +----------------------------------+---------------------+----------+-----------------------------------+--------+
  | 5f7dccf493cf4b9d87d87a8f0a052192 | 2025-04-27T12:37:50 | check    | https://review.opendev.org/947212 | master |
  | c7f28ff023b747849862c96817ffccdb | 2025-04-26T12:36:32 | check    | https://review.opendev.org/947212 | master |
  | f94c9e6667874556849f625a01964367 | 2025-04-25T14:16:05 | check    | https://review.opendev.org/922496 | master |
  | f169e50cabbf4e65a89a922b6ac9b29e | 2025-04-25T13:08:02 | check    | https://review.opendev.org/947966 | master |
  | 0e53881b2e694ff8996b55bf7c477e79 | 2025-04-25T08:27:00 | check    | https://review.opendev.org/947212 | master |
  +----------------------------------+---------------------+----------+-----------------------------------+--------+

  I only see it in the nova-ceph-multistore job but in various test cases:
  * https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_5f7/openstack/5f7dccf493cf4b9d87d87a8f0a052192/testr_results.html
  * https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f94/openstack/f94c9e6667874556849f625a01964367/testr_results.html
  * https://40d1580bb656fd0ed240-3f272db0dacf207a646e9867f60c7e03.ssl.cf1.rackcdn.com/openstack/f169e50cabbf4e65a89a922b6ac9b29e/testr_results.html

  I looked a bit into the occurrence
  https://zuul.opendev.org/t/openstack/build/5f7dccf493cf4b9d87d87a8f0a052192/logs
  below.

  glance API logs:
  https://zuul.opendev.org/t/openstack/build/5f7dccf493cf4b9d87d87a8f0a052192/log/controller/logs/screen-g-api.txt#12123-12132

  Apr 27 11:52:19.736399 np0040547480 glance-api[98085]: DEBUG oslo_policy.policy [None req-20c620eb-2315-4620-b145-98df5a0ed217 tempest-ServerActionsTestOtherB-1525757365 tempest-ServerActionsTestOtherB-1525757365-project-member] enforce: rule="context_is_admin" creds={"domain_id": null, "is_admin_project": true, "project_domain_id": "default", "project_id": "495c6985d5064e17970e2b26b58e56c6", "roles": ["reader", "member"], "service_project_domain_id": null, "service_project_id": null, "service_roles": [], "service_user_domain_id": null, "service_user_id": null, "system_scope": null, "user_domain_id": "default", "user_id": "ec6739598d2540019167fc2432e4d312"} target={"auth_token": "***", "domain": null, "global_request_id": null, "is_admin": false, "is_admin_project": true, "project": "495c6985d5064e17970e2b26b58e56c6", "project_domain": "default", "project_id": "495c6985d5064e17970e2b26b58e56c6", "read_only": false, "request_id": "req-20c620eb-2315-4620-b145-98df5a0ed217", "resource_uuid": null, "roles": ["reader", "member"], "service_catalog": [{"endpoints": [{"publicURL": "https://158.69.64.188/identity";, "region": "RegionOne"}], "name": "keystone", "type": "identity"}, {"endpoints": [{"publicURL": "https://158.69.64.188/compute/v2/495c6985d5064e17970e2b26b58e56c6";, "region": "RegionOne"}], "name": "nova_legacy", "type": "compute_legacy"}, {"endpoints": [{"publicURL": "https://158.69.64.188/volume/v3";, "region": "RegionOne"}], "name": "cinder", "type": "block-storage"}, {"endpoints": [{"publicURL": "https://158.69.64.188/placement";, "region": "RegionOne"}], "name": "placement", "type": "placement"}, {"endpoints": [{"publicURL": "https://158.69.64.188/image";, "region": "RegionOne"}], "name": "glance", "type": "image"}, {"endpoints": [{"publicURL": "https://158.69.64.188/compute/v2.1";, "region": "RegionOne"}], "name": "nova", "type": "compute"}, {"endpoints": [{"publicURL": "https://158.69.64.188:9696/networking";, "region": "RegionOne"}], "name": "neutron", "type": "network"}], "show_deleted": false, "system_scope": null, "user": "ec6739598d2540019167fc2432e4d312", "user_domain": "default", "user_identity": "ec6739598d2540019167fc2432e4d312 495c6985d5064e17970e2b26b58e56c6 - default default"} {{(pid=98085) enforce /opt/stack/data/venv/lib/python3.12/site-packages/oslo_policy/policy.py:1040}}
  Apr 27 11:52:19.748380 np0040547480 glance-api[98085]: DEBUG glance.common.utils [None req-20c620eb-2315-4620-b145-98df5a0ed217 tempest-ServerActionsTestOtherB-1525757365 tempest-ServerActionsTestOtherB-1525757365-project-member] Sorted locations: [{'store_name': 'robust'}] {{(pid=98085) sort_image_locations /opt/stack/glance/glance/common/utils.py:735}}
  Apr 27 11:52:19.750448 np0040547480 glance-api[98085]: DEBUG oslo_policy.policy [None req-20c620eb-2315-4620-b145-98df5a0ed217 tempest-ServerActionsTestOtherB-1525757365 tempest-ServerActionsTestOtherB-1525757365-project-member] enforce: rule="delete_image" creds={"domain_id": null, "is_admin_project": true, "project_domain_id": "default", "project_id": "495c6985d5064e17970e2b26b58e56c6", "roles": ["reader", "member"], "service_project_domain_id": null, "service_project_id": null, "service_roles": [], "service_user_domain_id": null, "service_user_id": null, "system_scope": null, "user_domain_id": "default", "user_id": "ec6739598d2540019167fc2432e4d312"} target={"backup_type": "daily", "base_image_ref": "08f6a6f3-85eb-46c1-a2ab-e260218b0429", "boot_roles": "reader,member", "checksum": null, "container_format": "bare", "created_at": "2025-04-27T11:52:11.000000", "disk_format": "raw", "extra_properties": {}, "hw_cdrom_bus": "ide", "hw_disk_bus": "virtio", "hw_machine_type": "pc", "hw_video_model": "virtio", "hw_vif_model": "virtio", "image_id": "345b1922-5189-4d6f-a7af-bf28c7226051", "image_location": "snapshot", "image_state": "available", "image_type": "backup", "instance_uuid": "79f599ec-27d2-47cf-9072-be2ab65bee1b", "member": null, "min_disk": 1, "min_ram": 0, "name": "tempest-backup-3-804999709", "os_glance_import_task": "a39f3c07-8a30-48c3-ba47-5464019146e0", "os_hash_algo": "sha512", "os_hash_value": null, "os_hidden": false, "owner": "495c6985d5064e17970e2b26b58e56c6", "owner_id": "495c6985d5064e17970e2b26b58e56c6", "owner_project_name": "tempest-ServerActionsTestOtherB-1525757365", "owner_specified.openstack.md5": "", "owner_specified.openstack.object": "images/cirros-raw", "owner_specified.openstack.sha256": "", "owner_user_name": "tempest-ServerActionsTestOtherB-1525757365-project-member", "project_id": "495c6985d5064e17970e2b26b58e56c6", "protected": false, "size": 1073741824, "status": "active", "tags": [], "updated_at": "2025-04-27T11:52:18.000000", "user_id": "ec6739598d2540019167fc2432e4d312", "virtual_size": null, "visibility": "private"} {{(pid=98085) enforce /opt/stack/data/venv/lib/python3.12/site-packages/oslo_policy/policy.py:1040}}
  Apr 27 11:52:19.751720 np0040547480 glance-api[98085]: DEBUG glance_store.multi_backend [None req-20c620eb-2315-4620-b145-98df5a0ed217 tempest-ServerActionsTestOtherB-1525757365 tempest-ServerActionsTestOtherB-1525757365-project-member] Attempting to import store rbd {{(pid=98085) _load_multi_store /opt/stack/data/venv/lib/python3.12/site-packages/glance_store/multi_backend.py:170}}
  Apr 27 11:52:19.773854 np0040547480 glance-api[98085]: DEBUG glance_store.capabilities [None req-20c620eb-2315-4620-b145-98df5a0ed217 tempest-ServerActionsTestOtherB-1525757365 tempest-ServerActionsTestOtherB-1525757365-project-member] Store glance_store._drivers.rbd.Store doesn't support updating dynamic storage capabilities. Please overwrite 'update_capabilities' method of the store to implement updating logics if needed. {{(pid=98085) update_capabilities /opt/stack/data/venv/lib/python3.12/site-packages/glance_store/capabilities.py:91}}
  Apr 27 11:52:19.774138 np0040547480 glance-api[98085]: DEBUG glance_store.driver [None req-20c620eb-2315-4620-b145-98df5a0ed217 tempest-ServerActionsTestOtherB-1525757365 tempest-ServerActionsTestOtherB-1525757365-project-member] Late loading location class glance_store._drivers.rbd.StoreLocation {{(pid=98085) get_store_location_class /opt/stack/data/venv/lib/python3.12/site-packages/glance_store/driver.py:133}}
  Apr 27 11:52:19.774268 np0040547480 glance-api[98085]: DEBUG glance_store.location [None req-20c620eb-2315-4620-b145-98df5a0ed217 tempest-ServerActionsTestOtherB-1525757365 tempest-ServerActionsTestOtherB-1525757365-project-member] Registering scheme rbd with {'robust': {'store': <glance_store._drivers.rbd.Store object at 0x7687f29b94c0>, 'location_class': <class 'glance_store._drivers.rbd.StoreLocation'>, 'store_entry': 'rbd'}} {{(pid=98085) register_scheme_backend_map /opt/stack/data/venv/lib/python3.12/site-packages/glance_store/location.py:132}}
  Apr 27 11:52:20.682141 np0040547480 glance-api[98085]: WARNING glance_store._drivers.rbd [None req-20c620eb-2315-4620-b145-98df5a0ed217 tempest-ServerActionsTestOtherB-1525757365 tempest-ServerActionsTestOtherB-1525757365-project-member] Remove image 345b1922-5189-4d6f-a7af-bf28c7226051 failed. It is in use.: rbd.ImageBusy: [errno 16] RBD image is busy (error removing image)
  Apr 27 11:52:20.687798 np0040547480 glance-api[98085]: WARNING glance.api.v2.images [None req-20c620eb-2315-4620-b145-98df5a0ed217 tempest-ServerActionsTestOtherB-1525757365 tempest-ServerActionsTestOtherB-1525757365-project-member] Image 345b1922-5189-4d6f-a7af-bf28c7226051 could not be deleted because it is in use: The image cannot be deleted because it is in use through the backend store outside of Glance.: glance_store.exceptions.InUseByStore: The image cannot be deleted because it is in use through the backend store outside of Glance.
  Apr 27 11:52:20.689146 np0040547480 glance-api[98085]: INFO eventlet.wsgi.server [None req-20c620eb-2315-4620-b145-98df5a0ed217 tempest-ServerActionsTestOtherB-1525757365 tempest-ServerActionsTestOtherB-1525757365-project-member] 158.69.64.188,158.69.64.188,158.69.64.188 - - [27/Apr/2025 11:52:20] "DELETE /v2/images/345b1922-5189-4d6f-a7af-bf28c7226051 HTTP/1.1" 409 462 0.956102

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/2109428/+subscriptions

References

  Thread Previous • Date Previous • • Thread Next