yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1492345] [NEW] Unable to perform nova-api operations on instance

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Ioana-Madalina Patrichi <1492345@xxxxxxxxxxxxxxxxxx>
Date: Fri, 04 Sep 2015 15:45:43 -0000
Reply-to: Bug 1492345 <1492345@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Openstack version: Kilo
Nova version:  1:2015.1.0-0ubuntu1.1

Every time I try to perform an operation on an instance, the compute
node reports the following error:

2015-09-04 15:35:48.529 42883 INFO nova.compute.manager [req-b750c91a-ea2a-425c-832d-906e3c452904 c39a72988ef2478a930e627caa7f706a 2ff06b822bab4d59a6f0bc81be34980f - - -] [instance: 6a97676c-f735-4a93-b787-6d9b4c367836] Rebooting instance
2015-09-04 15:35:48.847 42883 INFO nova.scheduler.client.report [req-b750c91a-ea2a-425c-832d-906e3c452904 c39a72988ef2478a930e627caa7f706a 2ff06b822bab4d59a6f0bc81be34980f - - -] Compute_service record updated for ('ncn11', 'ncn11.hpscto.local')
2015-09-04 15:35:48.848 42883 ERROR oslo_messaging.rpc.dispatcher [req-b750c91a-ea2a-425c-832d-906e3c452904 c39a72988ef2478a930e627caa7f706a 2ff06b822bab4d59a6f0bc81be34980f - - -] Exception during message handling: Cannot call obj_load_attr on orphaned Instance object
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6695, in reboot_instance
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     reboot_type)
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     payload)
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 327, in decorated_function
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance_uuid=instance_uuid)
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 298, in decorated_function
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 377, in decorated_function
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 355, in decorated_function
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 343, in decorated_function
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3106, in reboot_instance
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     self._notify_about_instance_usage(context, instance, "reboot.start")
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2078, in _notify_about_instance_usage
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     extra_usage_info=extra_usage_info, fault=fault)
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/utils.py", line 305, in notify_about_instance_usage
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     network_info, system_metadata, **extra_usage_info)
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/notifications.py", line 374, in info_from_instance
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     instance_type = instance.get_flavor()
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1007, in get_flavor
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1007, in get_flavor
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     return getattr(self, attr)
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 72, in getter
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     self.obj_load_attr(name)
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 997, in obj_load_attr
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     self._load_flavor()
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 941, in _load_flavor
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     self.flavor = instance.flavor
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 72, in getter
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     self.obj_load_attr(name)
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 978, in obj_load_attr
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     objtype=self.obj_name())
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object
2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1492345

Title:
  Unable to perform nova-api operations on instance

Status in OpenStack Compute (nova):
  New

Bug description:
  Openstack version: Kilo
  Nova version:  1:2015.1.0-0ubuntu1.1

  Every time I try to perform an operation on an instance, the compute
  node reports the following error:

  2015-09-04 15:35:48.529 42883 INFO nova.compute.manager [req-b750c91a-ea2a-425c-832d-906e3c452904 c39a72988ef2478a930e627caa7f706a 2ff06b822bab4d59a6f0bc81be34980f - - -] [instance: 6a97676c-f735-4a93-b787-6d9b4c367836] Rebooting instance
  2015-09-04 15:35:48.847 42883 INFO nova.scheduler.client.report [req-b750c91a-ea2a-425c-832d-906e3c452904 c39a72988ef2478a930e627caa7f706a 2ff06b822bab4d59a6f0bc81be34980f - - -] Compute_service record updated for ('ncn11', 'ncn11.hpscto.local')
  2015-09-04 15:35:48.848 42883 ERROR oslo_messaging.rpc.dispatcher [req-b750c91a-ea2a-425c-832d-906e3c452904 c39a72988ef2478a930e627caa7f706a 2ff06b822bab4d59a6f0bc81be34980f - - -] Exception during message handling: Cannot call obj_load_attr on orphaned Instance object
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6695, in reboot_instance
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     reboot_type)
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     payload)
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 327, in decorated_function
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance_uuid=instance_uuid)
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 298, in decorated_function
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 377, in decorated_function
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 355, in decorated_function
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 343, in decorated_function
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3106, in reboot_instance
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     self._notify_about_instance_usage(context, instance, "reboot.start")
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2078, in _notify_about_instance_usage
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     extra_usage_info=extra_usage_info, fault=fault)
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/utils.py", line 305, in notify_about_instance_usage
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     network_info, system_metadata, **extra_usage_info)
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/notifications.py", line 374, in info_from_instance
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     instance_type = instance.get_flavor()
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1007, in get_flavor
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1007, in get_flavor
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     return getattr(self, attr)
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 72, in getter
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     self.obj_load_attr(name)
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 997, in obj_load_attr
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     self._load_flavor()
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 941, in _load_flavor
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     self.flavor = instance.flavor
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 72, in getter
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     self.obj_load_attr(name)
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 978, in obj_load_attr
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher     objtype=self.obj_name())
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object
  2015-09-04 15:35:48.848 42883 TRACE oslo_messaging.rpc.dispatcher

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1492345/+subscriptions
Follow ups

[Bug 1492345] Re: Unable to perform nova-api operations on instance
From: Launchpad Bug Tracker, 2015-11-04