yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1745529] Re: Failed to shelve paused instance with libvirt: "libvirtError: Requested operation is not valid: domain is not running"

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1745529@xxxxxxxxxxxxxxxxxx>
Date: Fri, 11 May 2018 01:51:52 -0000
Reply-to: Bug 1745529 <1745529@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/541660
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d5901f613cf98f61b5253a1568b22af1d9dd1a08
Submitter: Zuul
Branch:    master

commit d5901f613cf98f61b5253a1568b22af1d9dd1a08
Author: LeopardMa <mabao@xxxxxxxxxx>
Date:   Wed Feb 7 10:21:08 2018 +0800

    Fix shelving a paused instance
    
    It is possible to shelve a paused instance, but in that case the
    guest is already shutdown, and some hypervisors will fail when
    trying to perform a clean shutdown of a non-running guest. For
    example, attempting to shelve a paused libvirt instance will
    result in this error:
    
      libvirtError: Requested operation is not valid: domain is not running
    
    Therefore, if the instance is paused, we don't attempt a clean
    shutdown while shelving.
    
    Related Tempest test: https://review.openstack.org/564127/
    
    Closes-Bug: #1745529
    Change-Id: I8ca25d9847d50001fbe8513a6c1dba8b697c24e4


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1745529

Title:
  Failed to shelve paused instance with libvirt: "libvirtError:
  Requested operation is not valid: domain is not running"

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) ocata series:
  Confirmed
Status in OpenStack Compute (nova) pike series:
  Confirmed
Status in OpenStack Compute (nova) queens series:
  Confirmed

Bug description:
  You can use the following steps to reproduce(This problem also exists in the Ocata version)：
  1.nova pause 5e824732-c557-434f-9902-a6d9abce7bda
  2.nova list
  +--------------------------------------+-------+--------+------------+-------------+---------------------+
  | ID                                   | Name  | Status | Task State | Power State | Networks            |
  +--------------------------------------+-------+--------+------------+-------------+---------------------+
  | 5e824732-c557-434f-9902-a6d9abce7bda | test  | PAUSED | -          | Paused      | network=192.168.0.9 |
  +--------------------------------------+-------+--------+------------+-------------+---------------------+
  3.nova shelve 5e824732-c557-434f-9902-a6d9abce7bda

  Error log is as follows：
  oslo_concurrency.lockutils [None req-54849f29-2fd5-4ece-976b-bf4b2c5fd169 admin admin] Lock "compute_resources" acquired by "nova.compute.resource_tracker.update_usage" :: waited 0.000s {{(pid=8304) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273}}
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: DEBUG oslo_concurrency.lockutils [None req-54849f29-2fd5-4ece-976b-bf4b2c5fd169 admin admin] Lock "compute_resources" released by "nova.compute.resource_tracker.update_usage" :: held 0.038s {{(pid=8304) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285}}
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: INFO nova.compute.manager [None req-54849f29-2fd5-4ece-976b-bf4b2c5fd169 admin admin] [instance: 5e824732-c557-434f-9902-a6d9abce7bda] Successfully reverted task state from shelving on failure for instance.
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server [None req-54849f29-2fd5-4ece-976b-bf4b2c5fd169 admin admin] Exception during message handling: libvirtError: Requested operation is not valid: domain is not running
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 76, in wrapped
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 67, in wrapped
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 186, in decorated_function
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 156, in decorated_function
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/utils.py", line 944, in decorated_function
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 214, in decorated_function
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 202, in decorated_function
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 4678, in shelve_instance
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     do_shelve_instance()
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 4677, in do_shelve_instance
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     self._shelve_instance(context, instance, image_id, clean_shutdown)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 4711, in _shelve_instance
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     self._power_off_instance(context, instance, clean_shutdown)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 2318, in _power_off_instance
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     self.driver.power_off(instance, timeout, retry_interval)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2684, in power_off
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     self._clean_shutdown(instance, timeout, retry_interval)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2639, in _clean_shutdown
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     guest.shutdown()
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 589, in shutdown
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     self._domain.shutdown()
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2399, in shutdown
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainShutdown() failed', dom=self)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server libvirtError: Requested operation is not valid: domain is not running
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server
  Jan 26 10:13:16 localhost.localdomain nova-compute[8304]: DEBUG oslo_service.periodic_task [None req-54849f29-2fd5-4ece-976b-bf4b2c5fd169 admin admin] Running periodic task ComputeManager._poll_rebooting_instances {{(pid=8304) run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215}}

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1745529/+subscriptions
References

[Bug 1745529] [NEW] The pause status of the virtual machine to perform the shelve operation reported error
From: Brin Zhang, 2018-01-26