yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #39288
[Bug 1500289] [NEW] Booting instance failed with Kilo stable compute node and liberty controller
Public bug reported:
When booting instance will get error as below in kilo compute node
2015-09-27 06:31:13.528 ERROR nova.compute.manager [req-aa4fc705-3d6e-4913-9b6b-49b0371d1b83
None None] [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] Instance failed to spawn
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] Traceback (most recent call last):
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] File "/opt/stack/nova/nova/compute/manager.py", line 2442, in _build_resources
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] yield resources
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] File "/opt/stack/nova/nova/compute/manager.py", line 2314, in _build_and_run_instance
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] block_device_info=block_device_info)
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2351, in spawn
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] write_to_disk=True)
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4172, in _get_guest_xml
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] context)
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3982, in _get_guest_config
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] File "/opt/stack/nova/nova/pci/manager.py", line 279, in get_instance_pci_devs
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] pci_devices = inst.pci_devices
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] File "/opt/stack/nova/nova/objects/base.py", line 72, in getter
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] self.obj_load_attr(name)
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] File "/opt/stack/nova/nova/objects/instance.py", line 1000, in obj_load_attr
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] self._load_generic(attrname)
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] File "/opt/stack/nova/nova/objects/instance.py", line 890, in _load_generic
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] reason='loading %s requires recursion' % attrname)
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] ObjectActionError: Object action obj_load_attr failed because: loading pci_devices requires recursion
This is due to https://review.openstack.org/#/c/202616/
The kilo node can't understand 1.2 PCIDeviceList.
Then when kilo compute node retry building instance request send to
liberty conductor will get error as below:
2015-09-27 02:21:23.340 ERROR oslo_messaging.rpc.dispatcher [req-f5e20f63-bdaf-417d-9843-b96c6ef3fae6 admin admin] Exception during message handling: 'instance_type_memory_mb'
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/conductor/manager.py", line 715, in build_instances
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher instances)
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/scheduler/utils.py", line 63, in build_request_spec
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher instance_type = flavors.extract_flavor(instance)
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/flavors.py", line 290, in extract_flavor
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher setattr(flavor, key, sys_meta[type_key])
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher KeyError: 'instance_type_memory_mb'
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher
This is due to https://github.com/openstack/nova/blob/master/nova/scheduler/utils.py#L59 didn't return True anymore, because the kilo node renturn InstanceV1, not InstanceV2.
** Affects: nova
Importance: Critical
Assignee: Alex Xu (xuhj)
Status: In Progress
** Tags: liberty-rc-potential
** Changed in: nova
Assignee: (unassigned) => Alex Xu (xuhj)
** Changed in: nova
Importance: Undecided => Critical
** Changed in: nova
Status: New => In Progress
** Tags added: liberty-rc-potential
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1500289
Title:
Booting instance failed with Kilo stable compute node and liberty
controller
Status in OpenStack Compute (nova):
In Progress
Bug description:
When booting instance will get error as below in kilo compute node
2015-09-27 06:31:13.528 ERROR nova.compute.manager [req-aa4fc705-3d6e-4913-9b6b-49b0371d1b83
None None] [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] Instance failed to spawn
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] Traceback (most recent call last):
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] File "/opt/stack/nova/nova/compute/manager.py", line 2442, in _build_resources
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] yield resources
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] File "/opt/stack/nova/nova/compute/manager.py", line 2314, in _build_and_run_instance
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] block_device_info=block_device_info)
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2351, in spawn
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] write_to_disk=True)
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4172, in _get_guest_xml
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] context)
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3982, in _get_guest_config
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] File "/opt/stack/nova/nova/pci/manager.py", line 279, in get_instance_pci_devs
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] pci_devices = inst.pci_devices
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] File "/opt/stack/nova/nova/objects/base.py", line 72, in getter
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] self.obj_load_attr(name)
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] File "/opt/stack/nova/nova/objects/instance.py", line 1000, in obj_load_attr
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] self._load_generic(attrname)
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] File "/opt/stack/nova/nova/objects/instance.py", line 890, in _load_generic
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] reason='loading %s requires recursion' % attrname)
2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] ObjectActionError: Object action obj_load_attr failed because: loading pci_devices requires recursion
This is due to https://review.openstack.org/#/c/202616/
The kilo node can't understand 1.2 PCIDeviceList.
Then when kilo compute node retry building instance request send to
liberty conductor will get error as below:
2015-09-27 02:21:23.340 ERROR oslo_messaging.rpc.dispatcher [req-f5e20f63-bdaf-417d-9843-b96c6ef3fae6 admin admin] Exception during message handling: 'instance_type_memory_mb'
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/conductor/manager.py", line 715, in build_instances
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher instances)
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/scheduler/utils.py", line 63, in build_request_spec
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher instance_type = flavors.extract_flavor(instance)
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/flavors.py", line 290, in extract_flavor
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher setattr(flavor, key, sys_meta[type_key])
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher KeyError: 'instance_type_memory_mb'
2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher
This is due to https://github.com/openstack/nova/blob/master/nova/scheduler/utils.py#L59 didn't return True anymore, because the kilo node renturn InstanceV1, not InstanceV2.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1500289/+subscriptions
Follow ups