yahoo-eng-team team mailing list archive

[OpenStack Infra <1980720@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/nova/+/852171
Committed: https://opendev.org/openstack/nova/commit/c8d5397b0a671fddfa9b9c27bc7790a439a18caf
Submitter: "Zuul (22348)"
Branch:    master

commit c8d5397b0a671fddfa9b9c27bc7790a439a18caf
Author: Amit Uniyal <auniyal@xxxxxxxxxx>
Date:   Thu Aug 4 13:42:14 2022 +0000

    Adds check for VM snapshot fail while quiesce
    
    Added check if quiesce fails because libvirt fails to connect with
    qemu guest agent inside instance
    
    Closes-Bug: #1980720
    Change-Id: I134a4060ace2678f76ae3606bf117c07194a8d92


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1980720

Title:
  "Unexpected API Error" when instance snapshot fails while quiescing

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Description
  ===========
  Creating a snapshot from a instance with root disk fails with an unexpected API Error.

  The reason is that the inital image has the property os_require_quiesce='True' set, but the
  instance has no qemu-guest-agent installed or running.

  
  Steps to reproduce
  ==================
  * Upload Image:
      $ openstack image create --disk-format raw --container-format bare \
        --property description=test --property hw_disk_bus=scsi \
        --property hw_qemu_guest_agent=yes \
        --property hw_scsi_model=virtio-scsi \
        --property os_require_quiesce=yes --file ubuntu2004.img test-image
  * Create a Instance with Root Disk:
      $ openstack server create --flavor $FLAVOR --network $NETWORK \
        --image $IMAGE --boot-from-volume 20 test-server
  * Create Snapshot from the Instance
      $ openstack server image create --name test-snap $INSTACE

  
  Expected result
  ===============
  User-friendly error message that libvirt cannnot connect 
  to the qemu-guest-agent inside the instance.

  
  Actual result
  =============
  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'nova.exception_Remote.InternalError_Remote'> (HTTP 500) (Request-ID: req-99b611e4-76d8-4f1c-8457-6abfdd6f344a)

  
  Environment
  ===========
  OpenStack Ussuri on Kolla based container

  
  Logs & Configs
  ==============

  INFO: api.py: Attempting to quiesce instance before volume snapshot.
  INFO: wsgi.py: HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
                 <class 'nova.exception_Remote.InternalError_Remote'>
  ERROR: wsgi.py: Unexpected exception in API method

  nova.exception_Remote.InternalError_Remote: Error from libvirt while quiescing instance-001f899f: [Error Code 86] Guest agent is not responding: QEMU guest agent is not connected
  Traceback (most recent call last):
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2626, in _set_quiesced
      guest.freeze_filesystems()
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 568, in freeze_filesystems
      self._domain.fsFreeze()
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit
      result = proxy_call(self._autowrap, f, *args, **kwargs)
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
      rv = execute(f, *args, **kwargs)
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
      six.reraise(c, e, tb)
    File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
      raise value
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
      rv = meth(*args, **kwargs)
    File "/usr/lib/python3/dist-packages/libvirt.py", line 3069, in fsFreeze
      if ret == -1: raise libvirtError ('virDomainFSFreeze() failed', dom=self)

  libvirt.libvirtError: Guest agent is not responding: QEMU guest agent
  is not connected

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
      return func(*args, **kwargs)
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 79, in wrapped
      function_name, call_dict, binary, tb)
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
      self.force_reraise()
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
      six.reraise(self.type_, self.value, self.tb)
    File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
      raise value
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 69, in wrapped
      return f(self, context, *args, **kw)
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 10518, in quiesce_instance
      self.driver.quiesce(context, instance, image_meta)
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2643, in quiesce
      self._set_quiesced(context, instance, image_meta, True)
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2636, in _set_quiesced
      raise exception.InternalError(msg)
  nova.exception.InternalError: Error from libvirt while quiescing instance-001f899f: [Error Code 86] Guest agent is not responding: QEMU guest agent is not connected

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1980720/+subscriptions