yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1648887] [NEW] nova host-servers-migrate [hostname] fails

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Alexander Chuzhoy <sashac88@xxxxxxxxxxx>
Date: Fri, 09 Dec 2016 21:14:31 -0000
Reply-to: Bug 1648887 <1648887@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Tried to migrate all vms as part of upgrading controllers going from OSP9 to OSP10.
Successfully migrated all vms from computes of the same version.
Upgraded one compute node.
Then tried to migrate all vms from the other machine to the upgraded one with:
nova host-servers-migrate overcloud-compute-0.localdomain

This resulted in error:
Grepping for errors the /var/log/nova/nova-compute.log file on a target compute:

2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server [req-5bbd4371-dd07-443c-884f-02937064bb8f 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] Exception during message handling
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     LOG.warning(msg, e, instance=instance)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 145, in decorated_function
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     migration.instance_uuid, exc_info=True)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 122, in decorated_function
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3960, in finish_resize
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     self._set_instance_obj_error_state(context, instance)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     disk_info, image_meta)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     old_instance_type)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     block_device_info, power_on)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     block_device_info=block_device_info)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     context)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     return [device for device in pci_devices if
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not iterable
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server 
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [req-10021f30-ac47-495d-83b6-2fe54d4ab647 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] Setting instance vm_state to ERROR
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] Traceback (most recent call last):
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]     disk_info, image_meta)
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]     old_instance_type)
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]     self.force_reraise()
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]     six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]     block_device_info, power_on)
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]     block_device_info=block_device_info)
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]     context)
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]     pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]   File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]     return [device for device in pci_devices if
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] TypeError: 'NoneType' object is not iterable
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] 
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server [req-10021f30-ac47-495d-83b6-2fe54d4ab647 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] Exception during message handling
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     LOG.warning(msg, e, instance=instance)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 145, in decorated_function
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     migration.instance_uuid, exc_info=True)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 122, in decorated_function
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3960, in finish_resize
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     self._set_instance_obj_error_state(context, instance)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     disk_info, image_meta)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     old_instance_type)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     block_device_info, power_on)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     block_device_info=block_device_info)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     context)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     return [device for device in pci_devices if
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not iterable
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server 
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [req-7ecfc3ee-2b17-4582-9788-f81a90de21e8 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] [instance: e9797290-6333-485f-a1c4-965355b11874] Setting instance vm_state to ERROR
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] Traceback (most recent call last):
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]     disk_info, image_meta)
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]     old_instance_type)
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]     self.force_reraise()
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]     six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]     block_device_info, power_on)
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]     block_device_info=block_device_info)
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]     context)
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]     pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]   File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]     if pci_devices is None:
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] TypeError: 'NoneType' object is not iterable
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] 
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server [req-7ecfc3ee-2b17-4582-9788-f81a90de21e8 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] Exception during message handling
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     LOG.warning(msg, e, instance=instance)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 145, in decorated_function
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     migration.instance_uuid, exc_info=True)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 122, in decorated_function
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3960, in finish_resize
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     self._set_instance_obj_error_state(context, instance)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     disk_info, image_meta)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     old_instance_type)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     block_device_info, power_on)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     block_device_info=block_device_info)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     context)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     if pci_devices is None:
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not iterable
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server 
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [req-81456147-61d3-4d1e-ac49-6becc999f201 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] Setting instance vm_state to ERROR
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] Traceback (most recent call last):
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]     disk_info, image_meta)
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]     old_instance_type)
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]     self.force_reraise()
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]     six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]     block_device_info, power_on)
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]     block_device_info=block_device_info)
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]     context)
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]     pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]   File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]     if pci_devices is None:
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] TypeError: 'NoneType' object is not iterable
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] 
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server [req-81456147-61d3-4d1e-ac49-6becc999f201 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] Exception during message handling
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     LOG.warning(msg, e, instance=instance)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 145, in decorated_function
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     migration.instance_uuid, exc_info=True)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 122, in decorated_function
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3960, in finish_resize
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     self._set_instance_obj_error_state(context, instance)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     disk_info, image_meta)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     old_instance_type)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     block_device_info, power_on)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     block_device_info=block_device_info)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     context)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     if pci_devices is None:
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not iterable
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server

** Affects: nova
     Importance: Undecided
     Assignee: Dan Smith (danms)
         Status: In Progress

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1648887

Title:
  nova host-servers-migrate [hostname] fails

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  Tried to migrate all vms as part of upgrading controllers going from OSP9 to OSP10.
  Successfully migrated all vms from computes of the same version.
  Upgraded one compute node.
  Then tried to migrate all vms from the other machine to the upgraded one with:
  nova host-servers-migrate overcloud-compute-0.localdomain

  This resulted in error:
  Grepping for errors the /var/log/nova/nova-compute.log file on a target compute:

  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server [req-5bbd4371-dd07-443c-884f-02937064bb8f 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] Exception during message handling
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     LOG.warning(msg, e, instance=instance)
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 145, in decorated_function
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     migration.instance_uuid, exc_info=True)
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 122, in decorated_function
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3960, in finish_resize
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     self._set_instance_obj_error_state(context, instance)
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     disk_info, image_meta)
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     old_instance_type)
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     block_device_info, power_on)
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     block_device_info=block_device_info)
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     context)
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server     return [device for device in pci_devices if
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not iterable
  2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server 
  2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [req-10021f30-ac47-495d-83b6-2fe54d4ab647 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] Setting instance vm_state to ERROR
  2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] Traceback (most recent call last):
  2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
  2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]     disk_info, image_meta)
  2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
  2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]     old_instance_type)
  2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]     self.force_reraise()
  2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
  2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]     block_device_info, power_on)
  2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
  2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]     block_device_info=block_device_info)
  2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
  2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]     context)
  2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
  2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]     pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
  2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]   File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
  2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]     return [device for device in pci_devices if
  2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] TypeError: 'NoneType' object is not iterable
  2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] 
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server [req-10021f30-ac47-495d-83b6-2fe54d4ab647 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] Exception during message handling
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     LOG.warning(msg, e, instance=instance)
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 145, in decorated_function
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     migration.instance_uuid, exc_info=True)
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 122, in decorated_function
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3960, in finish_resize
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     self._set_instance_obj_error_state(context, instance)
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     disk_info, image_meta)
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     old_instance_type)
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     block_device_info, power_on)
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     block_device_info=block_device_info)
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     context)
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server     return [device for device in pci_devices if
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not iterable
  2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server 
  2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [req-7ecfc3ee-2b17-4582-9788-f81a90de21e8 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] [instance: e9797290-6333-485f-a1c4-965355b11874] Setting instance vm_state to ERROR
  2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] Traceback (most recent call last):
  2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
  2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]     disk_info, image_meta)
  2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
  2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]     old_instance_type)
  2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]     self.force_reraise()
  2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
  2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]     block_device_info, power_on)
  2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
  2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]     block_device_info=block_device_info)
  2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
  2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]     context)
  2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
  2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]     pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
  2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]   File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
  2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]     if pci_devices is None:
  2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] TypeError: 'NoneType' object is not iterable
  2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] 
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server [req-7ecfc3ee-2b17-4582-9788-f81a90de21e8 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] Exception during message handling
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     LOG.warning(msg, e, instance=instance)
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 145, in decorated_function
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     migration.instance_uuid, exc_info=True)
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 122, in decorated_function
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3960, in finish_resize
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     self._set_instance_obj_error_state(context, instance)
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     disk_info, image_meta)
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     old_instance_type)
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     block_device_info, power_on)
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     block_device_info=block_device_info)
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     context)
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server     if pci_devices is None:
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not iterable
  2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server 
  2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [req-81456147-61d3-4d1e-ac49-6becc999f201 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] Setting instance vm_state to ERROR
  2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] Traceback (most recent call last):
  2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
  2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]     disk_info, image_meta)
  2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
  2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]     old_instance_type)
  2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]     self.force_reraise()
  2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
  2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]     block_device_info, power_on)
  2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
  2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]     block_device_info=block_device_info)
  2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
  2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]     context)
  2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
  2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]     pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
  2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]   File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
  2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]     if pci_devices is None:
  2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] TypeError: 'NoneType' object is not iterable
  2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] 
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server [req-81456147-61d3-4d1e-ac49-6becc999f201 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] Exception during message handling
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     LOG.warning(msg, e, instance=instance)
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 145, in decorated_function
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     migration.instance_uuid, exc_info=True)
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 122, in decorated_function
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3960, in finish_resize
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     self._set_instance_obj_error_state(context, instance)
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     disk_info, image_meta)
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     old_instance_type)
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     block_device_info, power_on)
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     block_device_info=block_device_info)
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     context)
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server     if pci_devices is None:
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not iterable
  2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1648887/+subscriptions
Follow ups

[Bug 1648887] Re: nova host-servers-migrate [hostname] fails
From: Lee Yarwood, 2020-11-06
[Bug 1648887] Re: nova host-servers-migrate [hostname] fails
From: Matt Riedemann, 2016-12-12
[Bug 1648887] Re: nova host-servers-migrate [hostname] fails
From: OpenStack Infra, 2016-12-12