yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #80965
[Bug 1769025] Re: OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object error is thrown for ironic deploy in Pike
Is this still an issue? I'm inclined to mark this as invalid unless it
can be recreated on a newer release.
** Changed in: nova
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1769025
Title:
OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance
object error is thrown for ironic deploy in Pike
Status in OpenStack Compute (nova):
Invalid
Bug description:
On the Pike setup, nova boot for ironic fails consistently due to
following error:
2018-04-13 07:56:02.846 DEBUG oslo_concurrency.processutils [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] CMD "genisoimage -o /tmp/tmppY5Ikz -ldots -allow-low
ercase -allow-multidot -l -publisher OpenStack Nova 16.1.1 -quiet -J -r -V config-2 /tmp/tmpxFZUMX" returned: 0 in 0.019s from (pid=47123) execute /usr/local/lib/pytho
n2.7/dist-packages/oslo_concurrency/processutils.py:404
2018-04-13 07:56:02.863 INFO nova.virt.ironic.driver [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] Config drive for instance 3b1195b4-3106-4c5c-87f4-69d203d08
884 on baremetal node 4fdcff13-0a9c-489c-8f71-99cfddfadf96 created.
2018-04-13 07:56:03.656 ERROR oslo.service.loopingcall [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] Fixed interval looping call 'nova.virt.ironic.driver.Iron
icDriver._wait_for_active' failed: OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object
Traceback (most recent call last):
File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch
return getattr(target, method)(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
return fn(self, *args, **kwargs)
File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh
elif self[field] != current[field]:
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 762, in __getitem__
return getattr(self, name)
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter
self.obj_load_attr(name)
File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr
objtype=self.obj_name())
OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall Traceback (most recent call last):
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 137, in _run_loop
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall result = func(*self.args, **self.kw)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/virt/ironic/driver.py", line 473, in _wait_for_active
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall instance.refresh()
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 210, in wrapper
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall ctxt, self, fn.__name__, args, kwargs)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/conductor/rpcapi.py", line 245, in object_action
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall objmethod=objmethod, args=args, kwargs=kwargs)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall retry=self.retry)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123, in _send
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall timeout=timeout, retry=retry)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall retry=retry)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 557, in _send
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall raise result
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall Traceback (most recent call last):
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall return getattr(target, method)(*args, **kwargs)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall return fn(self, *args, **kwargs)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall elif self[field] != current[field]:
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 762, in __getitem__
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall return getattr(self, name)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall self.obj_load_attr(name)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall objtype=self.obj_name())
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.660 ERROR nova.virt.ironic.driver [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] Error deploying instance 3b1195b4-3106-4c5c-87f4-69d203d08884 on baremetal node 4fdcff13-0a9c-489c-8f71-99cfddfadf96.: OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object
Traceback (most recent call last):
File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch
return getattr(target, method)(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
return fn(self, *args, **kwargs)
File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh
elif self[field] != current[field]:
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 762, in __getitem__
return getattr(self, name)
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter
self.obj_load_attr(name)
File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr
objtype=self.obj_name())
OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.661 ERROR nova.compute.manager [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] Instance failed to spawn: OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object
Traceback (most recent call last):
File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch
return getattr(target, method)(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
return fn(self, *args, **kwargs)
File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh
elif self[field] != current[field]:
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 762, in __getitem__
return getattr(self, name)
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter
self.obj_load_attr(name)
File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr
objtype=self.obj_name())
OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] Traceback (most recent call last):
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/compute/manager.py", line 2200, in _build_resources
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] yield resources
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/compute/manager.py", line 2015, in _build_and_run_instance
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] block_device_info=block_device_info)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/virt/ironic/driver.py", line 1108, in spawn
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] 'node': node_uuid})
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] self.force_reraise()
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] six.reraise(self.type_, self.value, self.tb)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/virt/ironic/driver.py", line 1100, in spawn
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] timer.start(interval=CONF.ironic.api_retry_interval).wait()
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] return hubs.get_hub().switch()
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] return self.greenlet.switch()
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 137, in _run_loop
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] result = func(*self.args, **self.kw)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/virt/ironic/driver.py", line 473, in _wait_for_active
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] instance.refresh()
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 210, in wrapper
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] ctxt, self, fn.__name__, args, kwargs)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/conductor/rpcapi.py", line 245, in object_action
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] objmethod=objmethod, args=args, kwargs=kwargs)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] retry=self.retry)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123, in _send
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] timeout=timeout, retry=retry)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] retry=retry)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 557, in _send
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] raise result
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] Traceback (most recent call last):
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] return getattr(target, method)(*args, **kwargs)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] return fn(self, *args, **kwargs)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] elif self[field] != current[field]:
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 762, in __getitem__
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] return getattr(self, name)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] self.obj_load_attr(name)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] objtype=self.obj_name())
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.665 INFO nova.compute.manager [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] Terminating instance
The issue is due to the assignment at line https://github.com/openstack/nova/blob/stable/pike/nova/objects/instance.py#L824
After commenting out this line, the deploy works end to end.
This has been reproduced consistently on two Pike setups.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1769025/+subscriptions
References
