← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #85577

[Bug 1919487] [NEW] virDomainBlockCommit called when deleting a snapshot via os-assisted-volume-snapshots even when instance is shutoff

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Description
===========

Attempting to delete a NFS volume snapshot (via c-api and the the os-
assisted-volume-snapshots n-api) of a volume attached to a SHUTOFF
instance currently results in n-cpu attempting to fire off a
virDomainBlockCommit command even though the instance isn't running.


Steps to reproduce
==================
1. Create multiple volume snapshots against a volume.
2. Attach the volume to an ACTIVE instance.
3. Stop the instance and ensure it is SHUTOFF.
4. Attempt to delete the latest snapshot.

Expected result
===============
qemu-img commit or qemu-img rebase should be used to handle this offline.

Actual result
=============
virDomainBlockCommit is called even though the domain isn't running.

Environment
===========

1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/

   master

2. Which hypervisor did you use?
   (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
   What's the version of that?

   libvirt + KVM

2. Which storage type did you use?
   (For example: Ceph, LVM, GPFS, ...)
   What's the version of that?

   NFS c-vol

3. Which networking type did you use?
   (For example: nova-network, Neutron with OpenVSwitch, ...)

   N/A

Logs & Configs
==============

Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server [req-570281c6-566e-44a3-9953-eeb634513778 req-0fbbe87f-fd1d-4861-9fb3-21b8eb011e55 service nova] Exception during message handling: libvirt.libvirtError: Requested operation is not valid: domain is not >
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 273, in dispatch
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 193, in _do_dispatch
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 78, in wrapped
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     self.force_reraise()
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     raise value
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 69, in wrapped
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 3916, in volume_snapshot_delete
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     snapshot_id, delete_info)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3129, in volume_snapshot_delete
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     context, snapshot_id, 'error_deleting')
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     self.force_reraise()
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     raise value
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3122, in volume_snapshot_delete
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     snapshot_id, delete_info=delete_info)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3107, in _volume_snapshot_delete
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     result = dev.commit(commit_base, commit_top, relative=True)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 809, in commit
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     self._disk, base, top, self.COMMIT_DEFAULT_BANDWIDTH, flags=flags)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 190, in doit
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 148, in proxy_call
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 129, in execute
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     raise value
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 83, in tworker
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib64/python3.7/site-packages/libvirt.py", line 728, in blockCommit
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainBlockCommit() failed', dom=self)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server libvirt.libvirtError: Requested operation is not valid: domain is not running

** Affects: nova
     Importance: Medium
     Assignee: Lee Yarwood (lyarwood)
         Status: New


** Tags: nfs volumes

** Changed in: nova
     Assignee: (unassigned) => Lee Yarwood (lyarwood)

** Changed in: nova
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1919487

Title:
  virDomainBlockCommit called when deleting a snapshot via os-assisted-
  volume-snapshots even when instance is shutoff

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========

  Attempting to delete a NFS volume snapshot (via c-api and the the os-
  assisted-volume-snapshots n-api) of a volume attached to a SHUTOFF
  instance currently results in n-cpu attempting to fire off a
  virDomainBlockCommit command even though the instance isn't running.

  
  Steps to reproduce
  ==================
  1. Create multiple volume snapshots against a volume.
  2. Attach the volume to an ACTIVE instance.
  3. Stop the instance and ensure it is SHUTOFF.
  4. Attempt to delete the latest snapshot.

  Expected result
  ===============
  qemu-img commit or qemu-img rebase should be used to handle this offline.

  Actual result
  =============
  virDomainBlockCommit is called even though the domain isn't running.

  Environment
  ===========

  1. Exact version of OpenStack you are running. See the following
    list for all releases: http://docs.openstack.org/releases/

     master

  2. Which hypervisor did you use?
     (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
     What's the version of that?

     libvirt + KVM

  2. Which storage type did you use?
     (For example: Ceph, LVM, GPFS, ...)
     What's the version of that?

     NFS c-vol

  3. Which networking type did you use?
     (For example: nova-network, Neutron with OpenVSwitch, ...)

     N/A

  Logs & Configs
  ==============

  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server [req-570281c6-566e-44a3-9953-eeb634513778 req-0fbbe87f-fd1d-4861-9fb3-21b8eb011e55 service nova] Exception during message handling: libvirt.libvirtError: Requested operation is not valid: domain is not >
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 273, in dispatch
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 193, in _do_dispatch
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 78, in wrapped
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     raise value
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 69, in wrapped
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 3916, in volume_snapshot_delete
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     snapshot_id, delete_info)
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3129, in volume_snapshot_delete
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     context, snapshot_id, 'error_deleting')
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     raise value
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3122, in volume_snapshot_delete
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     snapshot_id, delete_info=delete_info)
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3107, in _volume_snapshot_delete
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     result = dev.commit(commit_base, commit_top, relative=True)
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 809, in commit
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     self._disk, base, top, self.COMMIT_DEFAULT_BANDWIDTH, flags=flags)
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 190, in doit
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 148, in proxy_call
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 129, in execute
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     raise value
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 83, in tworker
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib64/python3.7/site-packages/libvirt.py", line 728, in blockCommit
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainBlockCommit() failed', dom=self)
  Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server libvirt.libvirtError: Requested operation is not valid: domain is not running

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1919487/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next