yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2115906] [NEW] Glance image import fails on Glance over Cinder/NFS scenario

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Fernando Ferraz <2115906@xxxxxxxxxxxxxxxxxx>
Date: Thu, 03 Jul 2025 15:16:12 -0000
Reply-to: Bug 2115906 <2115906@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Bug Description:

Glance over Cinder/NFS configuration randomly fails with the following
message when create/delete image operations are performed concurrently.
This happens because the current lock mechanism in glance_store that
prevents race conditions on volume mount/unmount [1][2] is local to the
process, and fails when multiple Glance workers run concurrently. 

[1]
https://opendev.org/openstack/glance_store/src/commit/9a5271ccddbe01256be0819c1a74fbfc93ab0a2d/glance_store/common/fs_mount.py#L181

[2]
https://opendev.org/openstack/glance_store/src/commit/8ac98ddf6170f66d4bdefac159b633c713a75dd5/glance_store/common/fs_mount.py#L242

Jun 03 19:37:41.316051 np0041005539 devstack@g-api.service[90321]: WARNING glance.image_cache.drivers.centralized_db [None req-3aff3146-9e2a-4de4-9f7b-4d8d622c03da tempest-ImageDependencyTests-1910387855 tempest-ImageDependencyTests-1910387855-project-member] ("Fetch of cache file failed (%(e)s), rolling back by moving '%(incomplete_path)s' to '%(invalid_path)s'", {'e': FileExistsError(17, 'File exists'), 'incomplete_path': '/opt/stack/data/glance/cache/incomplete/bc2f9475-0d11-4f96-b8e6-b22fdce5cace', 'invalid_path': '/opt/stack/data/glance/cache/invalid/bc2f9475-0d11-4f96-b8e6-b22fdce5cace'}): FileExistsError: [Errno 17] File exists: '/opt/stack/data/glance/images/mnt/8db6edfa5db8c4a4b76e2ad4526a0f97806edb8c809ce63ffcdb8d1a551aa57a'
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache [None req-3aff3146-9e2a-4de4-9f7b-4d8d622c03da tempest-ImageDependencyTests-1910387855 tempest-ImageDependencyTests-1910387855-project-member] Exception encountered while tee'ing image 'bc2f9475-0d11-4f96-b8e6-b22fdce5cace' into cache: [Errno 17] File exists: '/opt/stack/data/glance/images/mnt/8db6edfa5db8c4a4b76e2ad4526a0f97806edb8c809ce63ffcdb8d1a551aa57a'. Continuing with response.: FileExistsError: [Errno 17] File exists: '/opt/stack/data/glance/images/mnt/8db6edfa5db8c4a4b76e2ad4526a0f97806edb8c809ce63ffcdb8d1a551aa57a'
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache Traceback (most recent call last):
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache   File "/opt/stack/glance/glance/image_cache/__init__.py", line 360, in cache_tee_iter
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache     for chunk in image_iter:
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache   File "/opt/stack/glance/glance/notifier.py", line 412, in _get_chunk_data_iterator
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache     for chunk in data:
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache   File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/_drivers/cinder/store.py", line 813, in _cinder_volume_data_iterator
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache     with self._open_cinder_volume(client, volume, 'rb') as fp:
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache   File "/usr/lib/python3.12/contextlib.py", line 137, in __enter__
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache     return next(self.gen)
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache            ^^^^^^^^^^^^^^
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache   File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/_drivers/cinder/store.py", line 770, in _open_cinder_volume
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache     device = conn.connect_volume(volume)
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache   File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/_drivers/cinder/nfs.py", line 94, in connect_volume
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache     device = connect_volume_nfs()
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache              ^^^^^^^^^^^^^^^^^^^^
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py", line 415, in inner
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache     return f(*args, **kwargs)
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache            ^^^^^^^^^^^^^^^^^^
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache   File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/_drivers/cinder/nfs.py", line 89, in connect_volume_nfs
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache     mount.mount(
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache   File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/common/fs_mount.py", line 358, in mount
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache     mount_state.mount(fstype, export, vol_name, mountpoint, host,
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache   File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/common/fs_mount.py", line 247, in mount
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache     os.makedirs(mountpoint)
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache   File "<frozen os>", line 225, in makedirs
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache FileExistsError: [Errno 17] File exists: '/opt/stack/data/glance/images/mnt/8db6edfa5db8c4a4b76e2ad4526a0f97806edb8c809ce63ffcdb8d1a551aa57a'
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache


Steps to reproduce the issue:

Try to run multiple create/delete image operations concurrently in an
environment configured with Glance over Cinder/NFS.

Expected behavior:

All image create/delete operations should complete without errors.

Bug impact:

* Tempest tests are failing;
* Image create behavior is unreliable for environments running Glance over Cinder/NFS


Known workaround:

For the Glance service, set the number of workers to 1. This has a
significant performance impact, but currently it is the only way to
mitigate the issue.

** Affects: glance
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/2115906

Title:
  Glance image import fails on Glance over Cinder/NFS scenario

Status in Glance:
  New

Bug description:
  Bug Description:

  Glance over Cinder/NFS configuration randomly fails with the following
  message when create/delete image operations are performed
  concurrently. This happens because the current lock mechanism in
  glance_store that prevents race conditions on volume mount/unmount
  [1][2] is local to the process, and fails when multiple Glance workers
  run concurrently. 

  [1]
  https://opendev.org/openstack/glance_store/src/commit/9a5271ccddbe01256be0819c1a74fbfc93ab0a2d/glance_store/common/fs_mount.py#L181

  [2]
  https://opendev.org/openstack/glance_store/src/commit/8ac98ddf6170f66d4bdefac159b633c713a75dd5/glance_store/common/fs_mount.py#L242

  Jun 03 19:37:41.316051 np0041005539 devstack@g-api.service[90321]: WARNING glance.image_cache.drivers.centralized_db [None req-3aff3146-9e2a-4de4-9f7b-4d8d622c03da tempest-ImageDependencyTests-1910387855 tempest-ImageDependencyTests-1910387855-project-member] ("Fetch of cache file failed (%(e)s), rolling back by moving '%(incomplete_path)s' to '%(invalid_path)s'", {'e': FileExistsError(17, 'File exists'), 'incomplete_path': '/opt/stack/data/glance/cache/incomplete/bc2f9475-0d11-4f96-b8e6-b22fdce5cace', 'invalid_path': '/opt/stack/data/glance/cache/invalid/bc2f9475-0d11-4f96-b8e6-b22fdce5cace'}): FileExistsError: [Errno 17] File exists: '/opt/stack/data/glance/images/mnt/8db6edfa5db8c4a4b76e2ad4526a0f97806edb8c809ce63ffcdb8d1a551aa57a'
  Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache [None req-3aff3146-9e2a-4de4-9f7b-4d8d622c03da tempest-ImageDependencyTests-1910387855 tempest-ImageDependencyTests-1910387855-project-member] Exception encountered while tee'ing image 'bc2f9475-0d11-4f96-b8e6-b22fdce5cace' into cache: [Errno 17] File exists: '/opt/stack/data/glance/images/mnt/8db6edfa5db8c4a4b76e2ad4526a0f97806edb8c809ce63ffcdb8d1a551aa57a'. Continuing with response.: FileExistsError: [Errno 17] File exists: '/opt/stack/data/glance/images/mnt/8db6edfa5db8c4a4b76e2ad4526a0f97806edb8c809ce63ffcdb8d1a551aa57a'
  Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache Traceback (most recent call last):
  Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache   File "/opt/stack/glance/glance/image_cache/__init__.py", line 360, in cache_tee_iter
  Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache     for chunk in image_iter:
  Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache   File "/opt/stack/glance/glance/notifier.py", line 412, in _get_chunk_data_iterator
  Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache     for chunk in data:
  Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache   File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/_drivers/cinder/store.py", line 813, in _cinder_volume_data_iterator
  Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache     with self._open_cinder_volume(client, volume, 'rb') as fp:
  Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache   File "/usr/lib/python3.12/contextlib.py", line 137, in __enter__
  Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache     return next(self.gen)
  Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache            ^^^^^^^^^^^^^^
  Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache   File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/_drivers/cinder/store.py", line 770, in _open_cinder_volume
  Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache     device = conn.connect_volume(volume)
  Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache   File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/_drivers/cinder/nfs.py", line 94, in connect_volume
  Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache     device = connect_volume_nfs()
  Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache              ^^^^^^^^^^^^^^^^^^^^
  Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py", line 415, in inner
  Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache     return f(*args, **kwargs)
  Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache            ^^^^^^^^^^^^^^^^^^
  Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache   File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/_drivers/cinder/nfs.py", line 89, in connect_volume_nfs
  Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache     mount.mount(
  Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache   File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/common/fs_mount.py", line 358, in mount
  Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache     mount_state.mount(fstype, export, vol_name, mountpoint, host,
  Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache   File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/common/fs_mount.py", line 247, in mount
  Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache     os.makedirs(mountpoint)
  Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache   File "<frozen os>", line 225, in makedirs
  Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache FileExistsError: [Errno 17] File exists: '/opt/stack/data/glance/images/mnt/8db6edfa5db8c4a4b76e2ad4526a0f97806edb8c809ce63ffcdb8d1a551aa57a'
  Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache

  
  Steps to reproduce the issue:

  Try to run multiple create/delete image operations concurrently in an
  environment configured with Glance over Cinder/NFS.

  Expected behavior:

  All image create/delete operations should complete without errors.

  Bug impact:

  * Tempest tests are failing;
  * Image create behavior is unreliable for environments running Glance over Cinder/NFS

  
  Known workaround:

  For the Glance service, set the number of workers to 1. This has a
  significant performance impact, but currently it is the only way to
  mitigate the issue.

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/2115906/+subscriptions