yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #72730
[Bug 1769025] [NEW] OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object error is thrown for ironic deploy in Pike
Public bug reported:
On the Pike setup, nova boot for ironic fails consistently due to
following error:
2018-04-13 07:56:02.846 DEBUG oslo_concurrency.processutils [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] CMD "genisoimage -o /tmp/tmppY5Ikz -ldots -allow-low
ercase -allow-multidot -l -publisher OpenStack Nova 16.1.1 -quiet -J -r -V config-2 /tmp/tmpxFZUMX" returned: 0 in 0.019s from (pid=47123) execute /usr/local/lib/pytho
n2.7/dist-packages/oslo_concurrency/processutils.py:404
2018-04-13 07:56:02.863 INFO nova.virt.ironic.driver [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] Config drive for instance 3b1195b4-3106-4c5c-87f4-69d203d08
884 on baremetal node 4fdcff13-0a9c-489c-8f71-99cfddfadf96 created.
2018-04-13 07:56:03.656 ERROR oslo.service.loopingcall [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] Fixed interval looping call 'nova.virt.ironic.driver.Iron
icDriver._wait_for_active' failed: OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object
Traceback (most recent call last):
File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch
return getattr(target, method)(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
return fn(self, *args, **kwargs)
File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh
elif self[field] != current[field]:
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 762, in __getitem__
return getattr(self, name)
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter
self.obj_load_attr(name)
File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr
objtype=self.obj_name())
OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall Traceback (most recent call last):
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 137, in _run_loop
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall result = func(*self.args, **self.kw)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/virt/ironic/driver.py", line 473, in _wait_for_active
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall instance.refresh()
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 210, in wrapper
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall ctxt, self, fn.__name__, args, kwargs)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/conductor/rpcapi.py", line 245, in object_action
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall objmethod=objmethod, args=args, kwargs=kwargs)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall retry=self.retry)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123, in _send
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall timeout=timeout, retry=retry)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall retry=retry)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 557, in _send
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall raise result
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall Traceback (most recent call last):
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall return getattr(target, method)(*args, **kwargs)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall return fn(self, *args, **kwargs)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall elif self[field] != current[field]:
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 762, in __getitem__
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall return getattr(self, name)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall self.obj_load_attr(name)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall objtype=self.obj_name())
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.660 ERROR nova.virt.ironic.driver [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] Error deploying instance 3b1195b4-3106-4c5c-87f4-69d203d08884 on baremetal node 4fdcff13-0a9c-489c-8f71-99cfddfadf96.: OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object
Traceback (most recent call last):
File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch
return getattr(target, method)(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
return fn(self, *args, **kwargs)
File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh
elif self[field] != current[field]:
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 762, in __getitem__
return getattr(self, name)
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter
self.obj_load_attr(name)
File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr
objtype=self.obj_name())
OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.661 ERROR nova.compute.manager [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] Instance failed to spawn: OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object
Traceback (most recent call last):
File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch
return getattr(target, method)(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
return fn(self, *args, **kwargs)
File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh
elif self[field] != current[field]:
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 762, in __getitem__
return getattr(self, name)
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter
self.obj_load_attr(name)
File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr
objtype=self.obj_name())
OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] Traceback (most recent call last):
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/compute/manager.py", line 2200, in _build_resources
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] yield resources
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/compute/manager.py", line 2015, in _build_and_run_instance
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] block_device_info=block_device_info)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/virt/ironic/driver.py", line 1108, in spawn
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] 'node': node_uuid})
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] self.force_reraise()
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] six.reraise(self.type_, self.value, self.tb)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/virt/ironic/driver.py", line 1100, in spawn
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] timer.start(interval=CONF.ironic.api_retry_interval).wait()
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] return hubs.get_hub().switch()
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] return self.greenlet.switch()
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 137, in _run_loop
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] result = func(*self.args, **self.kw)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/virt/ironic/driver.py", line 473, in _wait_for_active
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] instance.refresh()
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 210, in wrapper
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] ctxt, self, fn.__name__, args, kwargs)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/conductor/rpcapi.py", line 245, in object_action
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] objmethod=objmethod, args=args, kwargs=kwargs)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] retry=self.retry)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123, in _send
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] timeout=timeout, retry=retry)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] retry=retry)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 557, in _send
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] raise result
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] Traceback (most recent call last):
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] return getattr(target, method)(*args, **kwargs)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] return fn(self, *args, **kwargs)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] elif self[field] != current[field]:
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 762, in __getitem__
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] return getattr(self, name)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] self.obj_load_attr(name)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] objtype=self.obj_name())
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.665 INFO nova.compute.manager [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] Terminating instance
The issue is due to the assignment at line https://github.com/openstack/nova/blob/stable/pike/nova/objects/instance.py#L824
After commenting out this line, the deploy works end to end.
This has been reproduced consistently on two Pike setups.
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1769025
Title:
OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance
object error is thrown for ironic deploy in Pike
Status in OpenStack Compute (nova):
New
Bug description:
On the Pike setup, nova boot for ironic fails consistently due to
following error:
2018-04-13 07:56:02.846 DEBUG oslo_concurrency.processutils [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] CMD "genisoimage -o /tmp/tmppY5Ikz -ldots -allow-low
ercase -allow-multidot -l -publisher OpenStack Nova 16.1.1 -quiet -J -r -V config-2 /tmp/tmpxFZUMX" returned: 0 in 0.019s from (pid=47123) execute /usr/local/lib/pytho
n2.7/dist-packages/oslo_concurrency/processutils.py:404
2018-04-13 07:56:02.863 INFO nova.virt.ironic.driver [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] Config drive for instance 3b1195b4-3106-4c5c-87f4-69d203d08
884 on baremetal node 4fdcff13-0a9c-489c-8f71-99cfddfadf96 created.
2018-04-13 07:56:03.656 ERROR oslo.service.loopingcall [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] Fixed interval looping call 'nova.virt.ironic.driver.Iron
icDriver._wait_for_active' failed: OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object
Traceback (most recent call last):
File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch
return getattr(target, method)(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
return fn(self, *args, **kwargs)
File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh
elif self[field] != current[field]:
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 762, in __getitem__
return getattr(self, name)
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter
self.obj_load_attr(name)
File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr
objtype=self.obj_name())
OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall Traceback (most recent call last):
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 137, in _run_loop
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall result = func(*self.args, **self.kw)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/virt/ironic/driver.py", line 473, in _wait_for_active
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall instance.refresh()
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 210, in wrapper
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall ctxt, self, fn.__name__, args, kwargs)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/conductor/rpcapi.py", line 245, in object_action
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall objmethod=objmethod, args=args, kwargs=kwargs)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall retry=self.retry)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123, in _send
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall timeout=timeout, retry=retry)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall retry=retry)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 557, in _send
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall raise result
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall Traceback (most recent call last):
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall return getattr(target, method)(*args, **kwargs)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall return fn(self, *args, **kwargs)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall elif self[field] != current[field]:
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 762, in __getitem__
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall return getattr(self, name)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall self.obj_load_attr(name)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall objtype=self.obj_name())
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.660 ERROR nova.virt.ironic.driver [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] Error deploying instance 3b1195b4-3106-4c5c-87f4-69d203d08884 on baremetal node 4fdcff13-0a9c-489c-8f71-99cfddfadf96.: OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object
Traceback (most recent call last):
File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch
return getattr(target, method)(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
return fn(self, *args, **kwargs)
File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh
elif self[field] != current[field]:
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 762, in __getitem__
return getattr(self, name)
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter
self.obj_load_attr(name)
File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr
objtype=self.obj_name())
OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.661 ERROR nova.compute.manager [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] Instance failed to spawn: OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object
Traceback (most recent call last):
File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch
return getattr(target, method)(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
return fn(self, *args, **kwargs)
File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh
elif self[field] != current[field]:
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 762, in __getitem__
return getattr(self, name)
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter
self.obj_load_attr(name)
File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr
objtype=self.obj_name())
OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] Traceback (most recent call last):
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/compute/manager.py", line 2200, in _build_resources
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] yield resources
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/compute/manager.py", line 2015, in _build_and_run_instance
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] block_device_info=block_device_info)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/virt/ironic/driver.py", line 1108, in spawn
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] 'node': node_uuid})
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] self.force_reraise()
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] six.reraise(self.type_, self.value, self.tb)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/virt/ironic/driver.py", line 1100, in spawn
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] timer.start(interval=CONF.ironic.api_retry_interval).wait()
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] return hubs.get_hub().switch()
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] return self.greenlet.switch()
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 137, in _run_loop
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] result = func(*self.args, **self.kw)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/virt/ironic/driver.py", line 473, in _wait_for_active
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] instance.refresh()
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 210, in wrapper
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] ctxt, self, fn.__name__, args, kwargs)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/conductor/rpcapi.py", line 245, in object_action
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] objmethod=objmethod, args=args, kwargs=kwargs)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] retry=self.retry)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123, in _send
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] timeout=timeout, retry=retry)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] retry=retry)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 557, in _send
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] raise result
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] Traceback (most recent call last):
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] return getattr(target, method)(*args, **kwargs)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] return fn(self, *args, **kwargs)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] elif self[field] != current[field]:
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 762, in __getitem__
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] return getattr(self, name)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] self.obj_load_attr(name)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] objtype=self.obj_name())
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.665 INFO nova.compute.manager [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] Terminating instance
The issue is due to the assignment at line https://github.com/openstack/nova/blob/stable/pike/nova/objects/instance.py#L824
After commenting out this line, the deploy works end to end.
This has been reproduced consistently on two Pike setups.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1769025/+subscriptions
Follow ups
