yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1745529] Re: The pause status of the virtual machine to perform the shelve operation reported error

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Tue, 27 Mar 2018 14:22:49 -0000
Reply-to: Bug 1745529 <1745529@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
   Importance: Undecided => Medium

** Summary changed:

- The pause status of the virtual machine to perform the shelve operation reported error
+ Failed to shelve paused instance with libvirt

** Summary changed:

- Failed to shelve paused instance with libvirt
+ Failed to shelve paused instance with libvirt: "libvirtError: Requested operation is not valid: domain is not running"

** Also affects: nova/queens
   Importance: Undecided
       Status: New

** Also affects: nova/pike
   Importance: Undecided
       Status: New

** Changed in: nova/pike
       Status: New => Confirmed

** Changed in: nova/queens
       Status: New => Confirmed

** Changed in: nova/pike
   Importance: Undecided => Medium

** Changed in: nova/queens
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1745529

Title:
  Failed to shelve paused instance with libvirt: "libvirtError:
  Requested operation is not valid: domain is not running"

Status in OpenStack Compute (nova):
  In Progress
Status in OpenStack Compute (nova) pike series:
  Confirmed
Status in OpenStack Compute (nova) queens series:
  Confirmed

Bug description:
  You can use the following steps to reproduce(This problem also exists in the Ocata version)：
  1.nova pause 5e824732-c557-434f-9902-a6d9abce7bda
  2.nova list
  +--------------------------------------+-------+--------+------------+-------------+---------------------+
  | ID                                   | Name  | Status | Task State | Power State | Networks            |
  +--------------------------------------+-------+--------+------------+-------------+---------------------+
  | 5e824732-c557-434f-9902-a6d9abce7bda | test  | PAUSED | -          | Paused      | network=192.168.0.9 |
  +--------------------------------------+-------+--------+------------+-------------+---------------------+
  3.nova shelve 5e824732-c557-434f-9902-a6d9abce7bda

  Error log is as follows：
  oslo_concurrency.lockutils [None req-54849f29-2fd5-4ece-976b-bf4b2c5fd169 admin admin] Lock "compute_resources" acquired by "nova.compute.resource_tracker.update_usage" :: waited 0.000s {{(pid=8304) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273}}
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: DEBUG oslo_concurrency.lockutils [None req-54849f29-2fd5-4ece-976b-bf4b2c5fd169 admin admin] Lock "compute_resources" released by "nova.compute.resource_tracker.update_usage" :: held 0.038s {{(pid=8304) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285}}
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: INFO nova.compute.manager [None req-54849f29-2fd5-4ece-976b-bf4b2c5fd169 admin admin] [instance: 5e824732-c557-434f-9902-a6d9abce7bda] Successfully reverted task state from shelving on failure for instance.
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server [None req-54849f29-2fd5-4ece-976b-bf4b2c5fd169 admin admin] Exception during message handling: libvirtError: Requested operation is not valid: domain is not running
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 76, in wrapped
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 67, in wrapped
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 186, in decorated_function
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 156, in decorated_function
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/utils.py", line 944, in decorated_function
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 214, in decorated_function
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 202, in decorated_function
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 4678, in shelve_instance
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     do_shelve_instance()
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 4677, in do_shelve_instance
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     self._shelve_instance(context, instance, image_id, clean_shutdown)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 4711, in _shelve_instance
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     self._power_off_instance(context, instance, clean_shutdown)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 2318, in _power_off_instance
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     self.driver.power_off(instance, timeout, retry_interval)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2684, in power_off
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     self._clean_shutdown(instance, timeout, retry_interval)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2639, in _clean_shutdown
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     guest.shutdown()
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 589, in shutdown
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     self._domain.shutdown()
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2399, in shutdown
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainShutdown() failed', dom=self)
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server libvirtError: Requested operation is not valid: domain is not running
  Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server
  Jan 26 10:13:16 localhost.localdomain nova-compute[8304]: DEBUG oslo_service.periodic_task [None req-54849f29-2fd5-4ece-976b-bf4b2c5fd169 admin admin] Running periodic task ComputeManager._poll_rebooting_instances {{(pid=8304) run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215}}

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1745529/+subscriptions
References

[Bug 1745529] [NEW] The pause status of the virtual machine to perform the shelve operation reported error
From: Brin Zhang, 2018-01-26