yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1509195] [NEW] error VM state when evacuate fail

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: hougangliu <liuhoug@xxxxxxxxxx>
Date: Fri, 23 Oct 2015 06:19:12 -0000
Reply-to: Bug 1509195 <1509195@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

when I evacuate a stopped VM, after the target host rebuilds the VM
failed with error message like below, the VM changed to ACTIVE(I think
the VM state should be ERROR or stopped):

2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 142, in inner
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     return func(*args, **kwargs)
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     payload)
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 72, in wrapped
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 352, in decorated_function
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance=instance)
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 325, in decorated_function
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 402, in decorated_function
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 380, in decorated_function
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 368, in decorated_function
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2726, in rebuild_instance
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     bdms, recreate, on_shared_storage, preserve_ephemeral)
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2765, in _do_rebuild_instance_with_claim
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     self._do_rebuild_instance(*args, **kwargs)
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2872, in _do_rebuild_instance
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     self._rebuild_default_impl(**kwargs)
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2646, in _rebuild_default_impl
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     block_device_info=new_block_device_info)
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2431, in spawn
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     block_device_info)
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 630, in get_disk_info
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     rescue)
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 537, in get_disk_mapping
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     disk_bus, cdrom_bus, root_device_name)
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 432, in get_root_info
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     if image_meta.disk_format == 'iso':
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 66, in getter
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     self.obj_load_attr(name)
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 555, in obj_load_attr
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     _("Cannot load '%s' in the base class") % attrname)
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher NotImplementedError: Cannot load 'disk_format' in the base class
2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher
2015-10-23 13:45:27.976 20194 INFO nova.compute.resource_tracker [req-23876311-4547-4500-9b9f-f1b786eb41ea - - - - -] Auditing locally available compute resources for node rhel8240
2015-10-23 13:45:30.040 20194 WARNING nova.compute.resource_tracker [req-23876311-4547-4500-9b9f-f1b786eb41ea - - - - -] [instance: 7f5d2ab2-523d-42c6-979d-7a0122573cf3] Instance not resizing, skipping migration.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1509195

Title:
  error VM state when evacuate fail

Status in OpenStack Compute (nova):
  New

Bug description:
  when I evacuate a stopped VM, after the target host rebuilds the VM
  failed with error message like below, the VM changed to ACTIVE(I think
  the VM state should be ERROR or stopped):

  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 142, in inner
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     return func(*args, **kwargs)
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     payload)
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 72, in wrapped
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 352, in decorated_function
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance=instance)
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 325, in decorated_function
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 402, in decorated_function
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 380, in decorated_function
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 368, in decorated_function
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2726, in rebuild_instance
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     bdms, recreate, on_shared_storage, preserve_ephemeral)
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2765, in _do_rebuild_instance_with_claim
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     self._do_rebuild_instance(*args, **kwargs)
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2872, in _do_rebuild_instance
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     self._rebuild_default_impl(**kwargs)
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2646, in _rebuild_default_impl
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     block_device_info=new_block_device_info)
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2431, in spawn
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     block_device_info)
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 630, in get_disk_info
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     rescue)
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 537, in get_disk_mapping
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     disk_bus, cdrom_bus, root_device_name)
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 432, in get_root_info
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     if image_meta.disk_format == 'iso':
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 66, in getter
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     self.obj_load_attr(name)
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 555, in obj_load_attr
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher     _("Cannot load '%s' in the base class") % attrname)
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher NotImplementedError: Cannot load 'disk_format' in the base class
  2015-10-23 13:44:58.535 20194 ERROR oslo_messaging.rpc.dispatcher
  2015-10-23 13:45:27.976 20194 INFO nova.compute.resource_tracker [req-23876311-4547-4500-9b9f-f1b786eb41ea - - - - -] Auditing locally available compute resources for node rhel8240
  2015-10-23 13:45:30.040 20194 WARNING nova.compute.resource_tracker [req-23876311-4547-4500-9b9f-f1b786eb41ea - - - - -] [instance: 7f5d2ab2-523d-42c6-979d-7a0122573cf3] Instance not resizing, skipping migration.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1509195/+subscriptions
Follow ups

[Bug 1509195] Re: VM is put back into ACTIVE state even though evacuate failed
From: Matt Riedemann, 2015-11-26