yahoo-eng-team team mailing list archive

[Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>]

** Changed in: nova
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1500289

Title:
  Booting instance failed with Kilo stable compute node and liberty
  controller

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  When booting instance will get error as below in kilo compute node

  	2015-09-27 06:31:13.528 ERROR nova.compute.manager [req-aa4fc705-3d6e-4913-9b6b-49b0371d1b83
  	 None None] [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] Instance failed to spawn
  	2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] Traceback (most recent call last):
  	2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4]   File "/opt/stack/nova/nova/compute/manager.py", line 2442, in _build_resources
  	2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4]     yield resources
  	2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4]   File "/opt/stack/nova/nova/compute/manager.py", line 2314, in _build_and_run_instance
  	2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4]     block_device_info=block_device_info)
  	2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2351, in spawn
  	2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4]     write_to_disk=True)
  	2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4172, in _get_guest_xml
  	2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4]     context)
  	2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3982, in _get_guest_config
  	2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4]     pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
  	2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4]   File "/opt/stack/nova/nova/pci/manager.py", line 279, in get_instance_pci_devs
  	2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4]     pci_devices = inst.pci_devices
  	2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4]   File "/opt/stack/nova/nova/objects/base.py", line 72, in getter
  	2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4]     self.obj_load_attr(name)
  	2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4]   File "/opt/stack/nova/nova/objects/instance.py", line 1000, in obj_load_attr
  	2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4]     self._load_generic(attrname)
  	2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4]   File "/opt/stack/nova/nova/objects/instance.py", line 890, in _load_generic
  	2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4]     reason='loading %s requires recursion' % attrname)
  	2015-09-27 06:31:13.528 TRACE nova.compute.manager [instance: 712e49b9-baef-4e7c-9ece-63a6f9f074e4] ObjectActionError: Object action obj_load_attr failed because: loading pci_devices requires recursion

  
  This is due to https://review.openstack.org/#/c/202616/ 

  The kilo node can't understand 1.2 PCIDeviceList.


  Then when kilo compute node retry building instance request send to
  liberty conductor will get error as below:

  	2015-09-27 02:21:23.340 ERROR oslo_messaging.rpc.dispatcher [req-f5e20f63-bdaf-417d-9843-b96c6ef3fae6 admin admin] Exception during message handling: 'instance_type_memory_mb'
  	2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  	2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  	2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  	2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  	2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  	2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
  	2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  	2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/conductor/manager.py", line 715, in build_instances
  	2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher     instances)
  	2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/scheduler/utils.py", line 63, in build_request_spec
  	2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher     instance_type = flavors.extract_flavor(instance)
  	2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/flavors.py", line 290, in extract_flavor
  	2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher     setattr(flavor, key, sys_meta[type_key])
  	2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher KeyError: 'instance_type_memory_mb'
  	2015-09-27 02:21:23.340 TRACE oslo_messaging.rpc.dispatcher
  	
  	
  This is due to https://github.com/openstack/nova/blob/master/nova/scheduler/utils.py#L59 didn't return True anymore, because the kilo node renturn InstanceV1, not InstanceV2.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1500289/+subscriptions