← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #89247

[Bug 1980720] [NEW] "Unexpected API Error" when instance snapshot fails while quiescing

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Description
===========
Creating a snapshot from a instance with root disk fails with an unexpected API Error.

The reason is that the inital image has the property os_require_quiesce='True' set, but the
instance has no qemu-guest-agent installed or running.


Steps to reproduce
==================
* Upload Image:
    $ openstack image create --disk-format raw --container-format bare \
      --property description=test --property hw_disk_bus=scsi \
      --property hw_qemu_guest_agent=yes \
      --property hw_scsi_model=virtio-scsi \
      --property os_require_quiesce=yes --file ubuntu2004.img test-image
* Create a Instance with Root Disk:
    $ openstack server create --flavor $FLAVOR --network $NETWORK \
      --image $IMAGE --boot-from-volume 20 test-server
* Create Snapshot from the Instance
    $ openstack server image create --name test-snap $INSTACE


Expected result
===============
User-friendly error message that libvirt cannnot connect 
to the qemu-guest-agent inside the instance.


Actual result
=============
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception_Remote.InternalError_Remote'> (HTTP 500) (Request-ID: req-99b611e4-76d8-4f1c-8457-6abfdd6f344a)


Environment
===========
OpenStack Ussuri on Kolla based container


Logs & Configs
==============

INFO: api.py: Attempting to quiesce instance before volume snapshot.
INFO: wsgi.py: HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
               <class 'nova.exception_Remote.InternalError_Remote'>
ERROR: wsgi.py: Unexpected exception in API method

nova.exception_Remote.InternalError_Remote: Error from libvirt while quiescing instance-001f899f: [Error Code 86] Guest agent is not responding: QEMU guest agent is not connected
Traceback (most recent call last):
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2626, in _set_quiesced
    guest.freeze_filesystems()
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 568, in freeze_filesystems
    self._domain.fsFreeze()
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit
    result = proxy_call(self._autowrap, f, *args, **kwargs)
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
    rv = execute(f, *args, **kwargs)
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
    six.reraise(c, e, tb)
  File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
    raise value
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
    rv = meth(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/libvirt.py", line 3069, in fsFreeze
    if ret == -1: raise libvirtError ('virDomainFSFreeze() failed', dom=self)

libvirt.libvirtError: Guest agent is not responding: QEMU guest agent is
not connected

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
    return func(*args, **kwargs)
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 79, in wrapped
    function_name, call_dict, binary, tb)
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
    raise value
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 69, in wrapped
    return f(self, context, *args, **kw)
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 10518, in quiesce_instance
    self.driver.quiesce(context, instance, image_meta)
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2643, in quiesce
    self._set_quiesced(context, instance, image_meta, True)
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2636, in _set_quiesced
    raise exception.InternalError(msg)
nova.exception.InternalError: Error from libvirt while quiescing instance-001f899f: [Error Code 86] Guest agent is not responding: QEMU guest agent is not connected

** Affects: nova
     Importance: Undecided
         Status: New

** Description changed:

  Description
  ===========
- 
- Creating a snapshot from a instance with root disk fails with an
- unexpected API Error.
+ Creating a snapshot from a instance with root disk fails with an unexpected API Error.
  
  The reason is that the inital image has the property os_require_quiesce='True' set, but the
  instance has no qemu-guest-agent installed or running.
  
  
  Steps to reproduce
  ==================
  * Upload Image:
-     $ openstack image create --disk-format raw --container-format bare \
-       --property description=test --property hw_disk_bus=scsi --property hw_qemu_guest_agent=yes \
-       --property hw_scsi_model=virtio-scsi --property os_require_quiesce=yes --file ubuntu2004.img test-image
+     $ openstack image create --disk-format raw --container-format bare \
+       --property description=test --property hw_disk_bus=scsi \
+       --property hw_qemu_guest_agent=yes \
+       --property hw_scsi_model=virtio-scsi \
+       --property os_require_quiesce=yes --file ubuntu2004.img test-image
  * Create a Instance with Root Disk:
-     $ openstack server create --flavor $FLAVOR --network $NETWORK --image $IMAGE --boot-from-volume 20 test-server
+     $ openstack server create --flavor $FLAVOR --network $NETWORK \
+       --image $IMAGE --boot-from-volume 20 test-server
  * Create Snapshot from the Instance
-     $ openstack server image create --name test-snap $INSTACE
+     $ openstack server image create --name test-snap $INSTACE
  
  
  Expected result
  ===============
- User-friendly error message that libvirt cannnot connect to the qemu-guest-agent inside the instance.
+ User-friendly error message that libvirt cannnot connect 
+ to the qemu-guest-agent inside the instance.
  
  
  Actual result
  =============
  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'nova.exception_Remote.InternalError_Remote'> (HTTP 500) (Request-ID: req-99b611e4-76d8-4f1c-8457-6abfdd6f344a)
  
  
  Environment
  ===========
  OpenStack Ussuri on Kolla based container
  
  
  Logs & Configs
  ==============
  
  INFO: api.py: Attempting to quiesce instance before volume snapshot.
  INFO: wsgi.py: HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
-                <class 'nova.exception_Remote.InternalError_Remote'>
+                <class 'nova.exception_Remote.InternalError_Remote'>
  ERROR: wsgi.py: Unexpected exception in API method
  
  nova.exception_Remote.InternalError_Remote: Error from libvirt while quiescing instance-001f899f: [Error Code 86] Guest agent is not responding: QEMU guest agent is not connected
  Traceback (most recent call last):
-   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2626, in _set_quiesced
-     guest.freeze_filesystems()
-   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 568, in freeze_filesystems
-     self._domain.fsFreeze()
-   File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit
-     result = proxy_call(self._autowrap, f, *args, **kwargs)
-   File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
-     rv = execute(f, *args, **kwargs)
-   File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
-     six.reraise(c, e, tb)
-   File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
-     raise value
-   File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
-     rv = meth(*args, **kwargs)
-   File "/usr/lib/python3/dist-packages/libvirt.py", line 3069, in fsFreeze
-     if ret == -1: raise libvirtError ('virDomainFSFreeze() failed', dom=self)
+   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2626, in _set_quiesced
+     guest.freeze_filesystems()
+   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 568, in freeze_filesystems
+     self._domain.fsFreeze()
+   File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit
+     result = proxy_call(self._autowrap, f, *args, **kwargs)
+   File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
+     rv = execute(f, *args, **kwargs)
+   File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
+     six.reraise(c, e, tb)
+   File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
+     raise value
+   File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
+     rv = meth(*args, **kwargs)
+   File "/usr/lib/python3/dist-packages/libvirt.py", line 3069, in fsFreeze
+     if ret == -1: raise libvirtError ('virDomainFSFreeze() failed', dom=self)
  
  libvirt.libvirtError: Guest agent is not responding: QEMU guest agent is
  not connected
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
-   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
-     return func(*args, **kwargs)
-   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 79, in wrapped
-     function_name, call_dict, binary, tb)
-   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
-     self.force_reraise()
-   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
-     six.reraise(self.type_, self.value, self.tb)
-   File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
-     raise value
-   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 69, in wrapped
-     return f(self, context, *args, **kw)
-   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 10518, in quiesce_instance
-     self.driver.quiesce(context, instance, image_meta)
-   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2643, in quiesce
-     self._set_quiesced(context, instance, image_meta, True)
-   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2636, in _set_quiesced
-     raise exception.InternalError(msg)
+   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
+     return func(*args, **kwargs)
+   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 79, in wrapped
+     function_name, call_dict, binary, tb)
+   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
+     self.force_reraise()
+   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
+     six.reraise(self.type_, self.value, self.tb)
+   File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
+     raise value
+   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 69, in wrapped
+     return f(self, context, *args, **kw)
+   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 10518, in quiesce_instance
+     self.driver.quiesce(context, instance, image_meta)
+   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2643, in quiesce
+     self._set_quiesced(context, instance, image_meta, True)
+   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2636, in _set_quiesced
+     raise exception.InternalError(msg)
  nova.exception.InternalError: Error from libvirt while quiescing instance-001f899f: [Error Code 86] Guest agent is not responding: QEMU guest agent is not connected

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1980720

Title:
  "Unexpected API Error" when instance snapshot fails while quiescing

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  Creating a snapshot from a instance with root disk fails with an unexpected API Error.

  The reason is that the inital image has the property os_require_quiesce='True' set, but the
  instance has no qemu-guest-agent installed or running.

  
  Steps to reproduce
  ==================
  * Upload Image:
      $ openstack image create --disk-format raw --container-format bare \
        --property description=test --property hw_disk_bus=scsi \
        --property hw_qemu_guest_agent=yes \
        --property hw_scsi_model=virtio-scsi \
        --property os_require_quiesce=yes --file ubuntu2004.img test-image
  * Create a Instance with Root Disk:
      $ openstack server create --flavor $FLAVOR --network $NETWORK \
        --image $IMAGE --boot-from-volume 20 test-server
  * Create Snapshot from the Instance
      $ openstack server image create --name test-snap $INSTACE

  
  Expected result
  ===============
  User-friendly error message that libvirt cannnot connect 
  to the qemu-guest-agent inside the instance.

  
  Actual result
  =============
  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'nova.exception_Remote.InternalError_Remote'> (HTTP 500) (Request-ID: req-99b611e4-76d8-4f1c-8457-6abfdd6f344a)

  
  Environment
  ===========
  OpenStack Ussuri on Kolla based container

  
  Logs & Configs
  ==============

  INFO: api.py: Attempting to quiesce instance before volume snapshot.
  INFO: wsgi.py: HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
                 <class 'nova.exception_Remote.InternalError_Remote'>
  ERROR: wsgi.py: Unexpected exception in API method

  nova.exception_Remote.InternalError_Remote: Error from libvirt while quiescing instance-001f899f: [Error Code 86] Guest agent is not responding: QEMU guest agent is not connected
  Traceback (most recent call last):
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2626, in _set_quiesced
      guest.freeze_filesystems()
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 568, in freeze_filesystems
      self._domain.fsFreeze()
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit
      result = proxy_call(self._autowrap, f, *args, **kwargs)
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
      rv = execute(f, *args, **kwargs)
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
      six.reraise(c, e, tb)
    File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
      raise value
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
      rv = meth(*args, **kwargs)
    File "/usr/lib/python3/dist-packages/libvirt.py", line 3069, in fsFreeze
      if ret == -1: raise libvirtError ('virDomainFSFreeze() failed', dom=self)

  libvirt.libvirtError: Guest agent is not responding: QEMU guest agent
  is not connected

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
      return func(*args, **kwargs)
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 79, in wrapped
      function_name, call_dict, binary, tb)
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
      self.force_reraise()
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
      six.reraise(self.type_, self.value, self.tb)
    File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
      raise value
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 69, in wrapped
      return f(self, context, *args, **kw)
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 10518, in quiesce_instance
      self.driver.quiesce(context, instance, image_meta)
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2643, in quiesce
      self._set_quiesced(context, instance, image_meta, True)
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2636, in _set_quiesced
      raise exception.InternalError(msg)
  nova.exception.InternalError: Error from libvirt while quiescing instance-001f899f: [Error Code 86] Guest agent is not responding: QEMU guest agent is not connected

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1980720/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next