← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #96244

[Bug 2119353] [NEW] Volume snapshot deletion of stopped Cinder NFS backed instance fails with RemoteFSInvalidBackingFile

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

This bug is the same problem as a past fixed bug [1] but somehow the fix
was incomplete and recently reported downstream [2].

When a boot-from-volume instance with Cinder backed by NFS, when
attempting to delete a volume snapshot after deleting a newer volume
snapshot while the instance is stopped, the delete fails when Cinder
raises RemoteFSInvalidBackingFile:

DEBUG cinder.volume.drivers.remotefs [None req-f4057282-a9a5-45e2-bca8-555831b07f22 tempest-TestVolumeBootPattern-1181586744 None] Deleting online snapshot 637a614e-0037-4dae-a8bf-68c28c5a1353 of volume 5032d631-b69a-4a03-a46d-526f2a026f0f {{(pid=89061) _delete_snapshot /opt/stack/cinder/cinder/volume/drivers/remotefs.py:1278}}
DEBUG cinder.volume.drivers.remotefs [None req-f4057282-a9a5-45e2-bca8-555831b07f22 tempest-TestVolumeBootPattern-1181586744 None] snapshot_file for this snap is: volume-5032d631-b69a-4a03-a46d-526f2a026f0f.144e7b21-bc8c-466b-a793-33d06ed0eacf {{(pid=89061) _delete_snapshot /opt/stack/cinder/cinder/volume/drivers/remotefs.py:1307}}
DEBUG oslo.privsep.daemon [-] privsep: reply[b1db715e-22bb-4ff7-b8d7-d4a53cde47a8]: (4, 'qcow2') {{(pid=98228) _call_back /opt/stack/data/venv/lib/python3.12/site-packages/oslo_privsep/daemon.py:512}}
DEBUG oslo_concurrency.processutils [None req-f4057282-a9a5-45e2-bca8-555831b07f22 tempest-TestVolumeBootPattern-1181586744 None] Running cmd (subprocess): /opt/stack/data/venv/bin/python3.12 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info -f qcow2 --output=json --force-share /opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-5032d631-b69a-4a03-a46d-526f2a026f0f.144e7b21-bc8c-466b-a793-33d06ed0eacf {{(pid=89061) execute /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}}
 : PWD=/ ; USER=root ; COMMAND=/opt/stack/data/venv/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info -f qcow2 --output=json --force-share /opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-5032d631-b69a-4a03-a46d-526f2a026f0f.144e7b21-bc8c-466b-a793-33d06ed0eacf
(sudo:session): session opened for user root(uid=0) by (uid=1001)
(sudo:session): session closed for user root
DEBUG oslo_concurrency.processutils [None req-f4057282-a9a5-45e2-bca8-555831b07f22 tempest-TestVolumeBootPattern-1181586744 None] CMD "/opt/stack/data/venv/bin/python3.12 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info -f qcow2 --output=json --force-share /opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-5032d631-b69a-4a03-a46d-526f2a026f0f.144e7b21-bc8c-466b-a793-33d06ed0eacf" returned: 0 in 0.229s {{(pid=89061) execute /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}}
DEBUG cinder.coordination [None req-f4057282-a9a5-45e2-bca8-555831b07f22 tempest-TestVolumeBootPattern-1181586744 None] Lock "cinder-nfs-5032d631-b69a-4a03-a46d-526f2a026f0f" released by "delete_snapshot" :: held 0.239s {{(pid=89061) __release /opt/stack/cinder/cinder/coordination.py:157}}
INFO cinder.message.api [None req-f4057282-a9a5-45e2-bca8-555831b07f22 tempest-TestVolumeBootPattern-1181586744 None] Creating message record for request_id = req-f4057282-a9a5-45e2-bca8-555831b07f22
DEBUG cinder.coordination [None req-f4057282-a9a5-45e2-bca8-555831b07f22 tempest-TestVolumeBootPattern-1181586744 None] Lock "cinder-637a614e-0037-4dae-a8bf-68c28c5a1353-delete_snapshot" released by "delete_snapshot" :: held 0.263s {{(pid=89061) __release /opt/stack/cinder/cinder/coordination.py:157}}
ERROR oslo_messaging.rpc.server [None req-f4057282-a9a5-45e2-bca8-555831b07f22 tempest-TestVolumeBootPattern-1181586744 None] Exception during message handling: cinder.exception.RemoteFSInvalidBackingFile: File /opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-5032d631-b69a-4a03-a46d-526f2a026f0f.144e7b21-bc8c-466b-a793-33d06ed0eacf has invalid backing file /opt/stack/data/nova/mnt/896fb15da6036b68a917322e72ebfe57/volume-5032d631-b69a-4a03-a46d-526f2a026f0f.
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/rpc/server.py", line 174, in _process_incoming
ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
ERROR oslo_messaging.rpc.server           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
ERROR oslo_messaging.rpc.server            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
ERROR oslo_messaging.rpc.server              ^^^^^^^^^^^^^^^^^^^^^^
ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/manager.py", line 219, in wrapper
ERROR oslo_messaging.rpc.server     skip_clean = func(self, context, snapshot, *args, **kwargs)
ERROR oslo_messaging.rpc.server                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.12/site-packages/decorator.py", line 235, in fun
ERROR oslo_messaging.rpc.server     return caller(func, *(extras + args), **kw)
ERROR oslo_messaging.rpc.server            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/coordination.py", line 239, in _synchronized
ERROR oslo_messaging.rpc.server     return f(*a, **k)
ERROR oslo_messaging.rpc.server            ^^^^^^^^^^
ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/manager.py", line 1351, in delete_snapshot
ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server     self.force_reraise()
ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
ERROR oslo_messaging.rpc.server     raise self.value
ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/manager.py", line 1337, in delete_snapshot
ERROR oslo_messaging.rpc.server     self.driver.delete_snapshot(snapshot)
ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.12/site-packages/decorator.py", line 235, in fun
ERROR oslo_messaging.rpc.server     return caller(func, *(extras + args), **kw)
ERROR oslo_messaging.rpc.server            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/coordination.py", line 239, in _synchronized
ERROR oslo_messaging.rpc.server     return f(*a, **k)
ERROR oslo_messaging.rpc.server            ^^^^^^^^^^
ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/drivers/nfs.py", line 620, in delete_snapshot
ERROR oslo_messaging.rpc.server     return self._delete_snapshot(snapshot)
ERROR oslo_messaging.rpc.server            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/drivers/remotefs.py", line 1312, in _delete_snapshot
ERROR oslo_messaging.rpc.server     snapshot_path_img_info = self._qemu_img_info(
ERROR oslo_messaging.rpc.server                              ^^^^^^^^^^^^^^^^^^^^
ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/drivers/nfs.py", line 585, in _qemu_img_info
ERROR oslo_messaging.rpc.server     return super(NfsDriver, self)._qemu_img_info_base(
ERROR oslo_messaging.rpc.server            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/drivers/remotefs.py", line 900, in _qemu_img_info_base
ERROR oslo_messaging.rpc.server     raise exception.RemoteFSInvalidBackingFile(
ERROR oslo_messaging.rpc.server cinder.exception.RemoteFSInvalidBackingFile: File /opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-5032d631-b69a-4a03-a46d-526f2a026f0f.144e7b21-bc8c-466b-a793-33d06ed0eacf has invalid backing file /opt/stack/data/nova/mnt/896fb15da6036b68a917322e72ebfe57/volume-5032d631-b69a-4a03-a46d-526f2a026f0f.
ERROR oslo_messaging.rpc.server

Steps to reproduce:

1. Create a volume boot instance
2. Create two (or more) volume snapshots from above volume
3. Stop instance
4. Delete the newest volume snapshot
5. Delete the older volume snapshot (fails)

The root cause is Nova using an absolute file path when specifying the
backing file when rebasing the qcow2 image during the deletion of the
newer volume snapshot. Then when the older volume snapshot is requested
to be deleted, on the Cinder side it fails because Cinder detects the
absolute path from Nova.

[1] https://bugs.launchpad.net/nova/+bug/1885528
[2] https://issues.redhat.com/browse/OSPRH-17787

** Affects: nova
     Importance: Undecided
     Assignee: melanie witt (melwitt)
         Status: New


** Tags: nfs volumes

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2119353

Title:
  Volume snapshot deletion of stopped Cinder NFS backed instance fails
  with RemoteFSInvalidBackingFile

Status in OpenStack Compute (nova):
  New

Bug description:
  This bug is the same problem as a past fixed bug [1] but somehow the
  fix was incomplete and recently reported downstream [2].

  When a boot-from-volume instance with Cinder backed by NFS, when
  attempting to delete a volume snapshot after deleting a newer volume
  snapshot while the instance is stopped, the delete fails when Cinder
  raises RemoteFSInvalidBackingFile:

  DEBUG cinder.volume.drivers.remotefs [None req-f4057282-a9a5-45e2-bca8-555831b07f22 tempest-TestVolumeBootPattern-1181586744 None] Deleting online snapshot 637a614e-0037-4dae-a8bf-68c28c5a1353 of volume 5032d631-b69a-4a03-a46d-526f2a026f0f {{(pid=89061) _delete_snapshot /opt/stack/cinder/cinder/volume/drivers/remotefs.py:1278}}
  DEBUG cinder.volume.drivers.remotefs [None req-f4057282-a9a5-45e2-bca8-555831b07f22 tempest-TestVolumeBootPattern-1181586744 None] snapshot_file for this snap is: volume-5032d631-b69a-4a03-a46d-526f2a026f0f.144e7b21-bc8c-466b-a793-33d06ed0eacf {{(pid=89061) _delete_snapshot /opt/stack/cinder/cinder/volume/drivers/remotefs.py:1307}}
  DEBUG oslo.privsep.daemon [-] privsep: reply[b1db715e-22bb-4ff7-b8d7-d4a53cde47a8]: (4, 'qcow2') {{(pid=98228) _call_back /opt/stack/data/venv/lib/python3.12/site-packages/oslo_privsep/daemon.py:512}}
  DEBUG oslo_concurrency.processutils [None req-f4057282-a9a5-45e2-bca8-555831b07f22 tempest-TestVolumeBootPattern-1181586744 None] Running cmd (subprocess): /opt/stack/data/venv/bin/python3.12 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info -f qcow2 --output=json --force-share /opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-5032d631-b69a-4a03-a46d-526f2a026f0f.144e7b21-bc8c-466b-a793-33d06ed0eacf {{(pid=89061) execute /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}}
   : PWD=/ ; USER=root ; COMMAND=/opt/stack/data/venv/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info -f qcow2 --output=json --force-share /opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-5032d631-b69a-4a03-a46d-526f2a026f0f.144e7b21-bc8c-466b-a793-33d06ed0eacf
  (sudo:session): session opened for user root(uid=0) by (uid=1001)
  (sudo:session): session closed for user root
  DEBUG oslo_concurrency.processutils [None req-f4057282-a9a5-45e2-bca8-555831b07f22 tempest-TestVolumeBootPattern-1181586744 None] CMD "/opt/stack/data/venv/bin/python3.12 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info -f qcow2 --output=json --force-share /opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-5032d631-b69a-4a03-a46d-526f2a026f0f.144e7b21-bc8c-466b-a793-33d06ed0eacf" returned: 0 in 0.229s {{(pid=89061) execute /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}}
  DEBUG cinder.coordination [None req-f4057282-a9a5-45e2-bca8-555831b07f22 tempest-TestVolumeBootPattern-1181586744 None] Lock "cinder-nfs-5032d631-b69a-4a03-a46d-526f2a026f0f" released by "delete_snapshot" :: held 0.239s {{(pid=89061) __release /opt/stack/cinder/cinder/coordination.py:157}}
  INFO cinder.message.api [None req-f4057282-a9a5-45e2-bca8-555831b07f22 tempest-TestVolumeBootPattern-1181586744 None] Creating message record for request_id = req-f4057282-a9a5-45e2-bca8-555831b07f22
  DEBUG cinder.coordination [None req-f4057282-a9a5-45e2-bca8-555831b07f22 tempest-TestVolumeBootPattern-1181586744 None] Lock "cinder-637a614e-0037-4dae-a8bf-68c28c5a1353-delete_snapshot" released by "delete_snapshot" :: held 0.263s {{(pid=89061) __release /opt/stack/cinder/cinder/coordination.py:157}}
  ERROR oslo_messaging.rpc.server [None req-f4057282-a9a5-45e2-bca8-555831b07f22 tempest-TestVolumeBootPattern-1181586744 None] Exception during message handling: cinder.exception.RemoteFSInvalidBackingFile: File /opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-5032d631-b69a-4a03-a46d-526f2a026f0f.144e7b21-bc8c-466b-a793-33d06ed0eacf has invalid backing file /opt/stack/data/nova/mnt/896fb15da6036b68a917322e72ebfe57/volume-5032d631-b69a-4a03-a46d-526f2a026f0f.
  ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/rpc/server.py", line 174, in _process_incoming
  ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  ERROR oslo_messaging.rpc.server           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
  ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  ERROR oslo_messaging.rpc.server            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
  ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  ERROR oslo_messaging.rpc.server              ^^^^^^^^^^^^^^^^^^^^^^
  ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/manager.py", line 219, in wrapper
  ERROR oslo_messaging.rpc.server     skip_clean = func(self, context, snapshot, *args, **kwargs)
  ERROR oslo_messaging.rpc.server                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.12/site-packages/decorator.py", line 235, in fun
  ERROR oslo_messaging.rpc.server     return caller(func, *(extras + args), **kw)
  ERROR oslo_messaging.rpc.server            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/coordination.py", line 239, in _synchronized
  ERROR oslo_messaging.rpc.server     return f(*a, **k)
  ERROR oslo_messaging.rpc.server            ^^^^^^^^^^
  ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/manager.py", line 1351, in delete_snapshot
  ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
  ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  ERROR oslo_messaging.rpc.server     self.force_reraise()
  ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  ERROR oslo_messaging.rpc.server     raise self.value
  ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/manager.py", line 1337, in delete_snapshot
  ERROR oslo_messaging.rpc.server     self.driver.delete_snapshot(snapshot)
  ERROR oslo_messaging.rpc.server   File "/opt/stack/data/venv/lib/python3.12/site-packages/decorator.py", line 235, in fun
  ERROR oslo_messaging.rpc.server     return caller(func, *(extras + args), **kw)
  ERROR oslo_messaging.rpc.server            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/coordination.py", line 239, in _synchronized
  ERROR oslo_messaging.rpc.server     return f(*a, **k)
  ERROR oslo_messaging.rpc.server            ^^^^^^^^^^
  ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/drivers/nfs.py", line 620, in delete_snapshot
  ERROR oslo_messaging.rpc.server     return self._delete_snapshot(snapshot)
  ERROR oslo_messaging.rpc.server            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/drivers/remotefs.py", line 1312, in _delete_snapshot
  ERROR oslo_messaging.rpc.server     snapshot_path_img_info = self._qemu_img_info(
  ERROR oslo_messaging.rpc.server                              ^^^^^^^^^^^^^^^^^^^^
  ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/drivers/nfs.py", line 585, in _qemu_img_info
  ERROR oslo_messaging.rpc.server     return super(NfsDriver, self)._qemu_img_info_base(
  ERROR oslo_messaging.rpc.server            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/drivers/remotefs.py", line 900, in _qemu_img_info_base
  ERROR oslo_messaging.rpc.server     raise exception.RemoteFSInvalidBackingFile(
  ERROR oslo_messaging.rpc.server cinder.exception.RemoteFSInvalidBackingFile: File /opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-5032d631-b69a-4a03-a46d-526f2a026f0f.144e7b21-bc8c-466b-a793-33d06ed0eacf has invalid backing file /opt/stack/data/nova/mnt/896fb15da6036b68a917322e72ebfe57/volume-5032d631-b69a-4a03-a46d-526f2a026f0f.
  ERROR oslo_messaging.rpc.server

  Steps to reproduce:

  1. Create a volume boot instance
  2. Create two (or more) volume snapshots from above volume
  3. Stop instance
  4. Delete the newest volume snapshot
  5. Delete the older volume snapshot (fails)

  The root cause is Nova using an absolute file path when specifying the
  backing file when rebasing the qcow2 image during the deletion of the
  newer volume snapshot. Then when the older volume snapshot is
  requested to be deleted, on the Cinder side it fails because Cinder
  detects the absolute path from Nova.

  [1] https://bugs.launchpad.net/nova/+bug/1885528
  [2] https://issues.redhat.com/browse/OSPRH-17787

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2119353/+subscriptions
  Thread Previous • Date Previous • • Thread Next