yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1512907] [NEW] leak of vswitch port if delete an instance while resizing

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Chris Friesen <chris.friesen@xxxxxxxxxxxxx>
Date: Wed, 04 Nov 2015 00:18:31 -0000
Reply-to: Bug 1512907 <1512907@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I've been testing with a modified version of stable/kilo, but I believe
the bug is present in upstream stable/kilo.

When using nova with neutron, if I boot an instance, then trigger a
resize, and then delete the instance at just the right point during the
resize it ends up causing a vswitch port to be "leaked".

So far I've been able to show that if I issue the "nova delete" command
while the resize operation is anywhere in
nova.compute.manager.ComputeManager._finish_resize() up to the point
where we set "instance.vm_state = vm_states.RESIZED" then I end up
"leaking" a vswitch port.  (By "leaking" I mean that it stays allocated
even after the instance that it was allocated for is deleted.)  I've
been testing this by calling pdb.set_trace() to pause the resize while
the nova delete runs, then letting the resize continue.  Yes, this
exaggerates the timing issues, but it shouldn't introduce any new races
if the code is correct.

I think the problem occurs because the deletion path can't confirm the
migration/resize because it hasn't gotten to the proper state yet.  The
resize code takes various exceptions depending on the exact timing of
when the deletion happens, but it doesn't trigger a revert of the resize
and it doesn't clean up the vswitch port on the source host.  See sample
log below.

I'm not sure what the proper fix is for this case.  It seems that until
we set "instance.vm_state = vm_states.RESIZED" it should be up to the
resize code to clean up all resources if the instance gets deleted while
a resize is in progress.


Sample log on source compute node.  This is with a pause right at the beginning of _finish_resize().

(Pdb) c
2015-11-03 23:28:37.968 17000 INFO nova.compute.resource_tracker [req-2d9812a5-eadf-4eb2-96a3-d46e496b292d - - - - -] Auditing locally available compute resources for node compute-1
2015-11-03 23:28:38.511 17000 INFO nova.network.neutronv2.api [req-46519d0e-6e80-40e1-bbb3-dc39184eb046 41f42dfc41f9428fb143623f0a83d2fa 726f4a1ce23f4f12acb9139dcfcdb313 - - -] [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] Port dad525cf-75af-47a7-a57c-cc6fa26a6cf2 from network info_cache is no longer associated with instance in Neutron. Removing from network info_cache.
2015-11-03 23:28:38.677 17000 ERROR nova.compute.manager [req-46519d0e-6e80-40e1-bbb3-dc39184eb046 41f42dfc41f9428fb143623f0a83d2fa 726f4a1ce23f4f12acb9139dcfcdb313 - - -] [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] Setting instance vm_state to ERROR
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] Traceback (most recent call last):
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 4350, in finish_resize
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]     disk_info, image)
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 4247, in _finish_resize
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]     resize_instance = False
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7866, in macs_for_instance
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/pci/manager.py", line 380, in get_instance_pci_devs
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/objects/base.py", line 72, in getter
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/objects/instance.py", line 1022, in obj_load_attr
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/objects/instance.py", line 904, in _load_generic
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/objects/base.py", line 161, in wrapper
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/conductor/rpcapi.py", line 335, in object_class_action
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "/usr/lib64/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]     retry=self.retry)
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "/usr/lib64/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]     timeout=timeout, retry=retry)
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "/usr/lib64/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]     retry=retry)
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "/usr/lib64/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 341, in _send
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]     raise result
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] InstanceNotFound_Remote: Instance 21a4ff59-9057-4bc2-8b5e-185374c2d557 could not be found.
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] Traceback (most recent call last):
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] 
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/conductor/manager.py", line 426, in _object_dispatch
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] 
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/objects/base.py", line 163, in wrapper
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] 
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/objects/instance.py", line 581, in get_by_uuid
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] 
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/db/api.py", line 664, in instance_get_by_uuid
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] 
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 233, in wrapper
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] 
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1757, in instance_get_by_uuid
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] 
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1769, in _instance_get_by_uuid
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] 
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] InstanceNotFound: Instance 21a4ff59-9057-4bc2-8b5e-185374c2d557 could not be found.
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] 
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] 
2015-11-03 23:28:39.053 17000 ERROR oslo_messaging.rpc.dispatcher [req-46519d0e-6e80-40e1-bbb3-dc39184eb046 41f42dfc41f9428fb143623f0a83d2fa 726f4a1ce23f4f12acb9139dcfcdb313 - - -] Exception during message handling: Instance 21a4ff59-9057-4bc2-8b5e-185374c2d557 could not be found.
Traceback (most recent call last):

  File "./usr/lib64/python2.7/site-packages/nova/conductor/manager.py",
line 426, in _object_dispatch

  File "./usr/lib64/python2.7/site-packages/nova/objects/base.py", line
163, in wrapper

  File "./usr/lib64/python2.7/site-packages/nova/objects/instance.py",
line 581, in get_by_uuid

  File "./usr/lib64/python2.7/site-packages/nova/db/api.py", line 664,
in instance_get_by_uuid

  File "./usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/api.py",
line 233, in wrapper

  File "./usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/api.py",
line 1757, in instance_get_by_uuid

  File "./usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/api.py",
line 1769, in _instance_get_by_uuid

InstanceNotFound: Instance 21a4ff59-9057-4bc2-8b5e-185374c2d557 could not be found.
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 7263, in finish_resize
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     reservations, migration)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 345, in decorated_function
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance_uuid=instance_uuid)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 316, in decorated_function
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 395, in decorated_function
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 304, in decorated_function
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     migration.instance_uuid, exc_info=True)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 280, in decorated_function
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 361, in decorated_function
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 4362, in finish_resize
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     self._set_instance_error_state(context, instance)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 4350, in finish_resize
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     disk_info, image)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 4247, in _finish_resize
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     resize_instance = False
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7866, in macs_for_instance
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/pci/manager.py", line 380, in get_instance_pci_devs
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/objects/base.py", line 72, in getter
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/objects/instance.py", line 1022, in obj_load_attr
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/objects/instance.py", line 904, in _load_generic
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/objects/base.py", line 161, in wrapper
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/conductor/rpcapi.py", line 335, in object_class_action
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     retry=self.retry)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     timeout=timeout, retry=retry)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     retry=retry)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 341, in _send
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     raise result
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher InstanceNotFound_Remote: Instance 21a4ff59-9057-4bc2-8b5e-185374c2d557 could not be found.
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher 
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/conductor/manager.py", line 426, in _object_dispatch
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher 
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/objects/base.py", line 163, in wrapper
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher 
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/objects/instance.py", line 581, in get_by_uuid
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher 
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/db/api.py", line 664, in instance_get_by_uuid
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher 
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 233, in wrapper
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher 
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1757, in instance_get_by_uuid
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher 
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1769, in _instance_get_by_uuid
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher 
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher InstanceNotFound: Instance 21a4ff59-9057-4bc2-8b5e-185374c2d557 could not be found.
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher 
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher 
2015-11-03 23:28:39.165 17000 INFO nova.scheduler.client.report [req-2d9812a5-eadf-4eb2-96a3-d46e496b292d - - - - -] Compute_service record updated for ('compute-1', 'compute-1')

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: libvirt resize

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1512907

Title:
  leak of vswitch port if delete an instance while resizing

Status in OpenStack Compute (nova):
  New

Bug description:
  I've been testing with a modified version of stable/kilo, but I
  believe the bug is present in upstream stable/kilo.

  When using nova with neutron, if I boot an instance, then trigger a
  resize, and then delete the instance at just the right point during
  the resize it ends up causing a vswitch port to be "leaked".

  So far I've been able to show that if I issue the "nova delete"
  command while the resize operation is anywhere in
  nova.compute.manager.ComputeManager._finish_resize() up to the point
  where we set "instance.vm_state = vm_states.RESIZED" then I end up
  "leaking" a vswitch port.  (By "leaking" I mean that it stays
  allocated even after the instance that it was allocated for is
  deleted.)  I've been testing this by calling pdb.set_trace() to pause
  the resize while the nova delete runs, then letting the resize
  continue.  Yes, this exaggerates the timing issues, but it shouldn't
  introduce any new races if the code is correct.

  I think the problem occurs because the deletion path can't confirm the
  migration/resize because it hasn't gotten to the proper state yet.
  The resize code takes various exceptions depending on the exact timing
  of when the deletion happens, but it doesn't trigger a revert of the
  resize and it doesn't clean up the vswitch port on the source host.
  See sample log below.

  I'm not sure what the proper fix is for this case.  It seems that
  until we set "instance.vm_state = vm_states.RESIZED" it should be up
  to the resize code to clean up all resources if the instance gets
  deleted while a resize is in progress.

  
  Sample log on source compute node.  This is with a pause right at the beginning of _finish_resize().

  (Pdb) c
  2015-11-03 23:28:37.968 17000 INFO nova.compute.resource_tracker [req-2d9812a5-eadf-4eb2-96a3-d46e496b292d - - - - -] Auditing locally available compute resources for node compute-1
  2015-11-03 23:28:38.511 17000 INFO nova.network.neutronv2.api [req-46519d0e-6e80-40e1-bbb3-dc39184eb046 41f42dfc41f9428fb143623f0a83d2fa 726f4a1ce23f4f12acb9139dcfcdb313 - - -] [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] Port dad525cf-75af-47a7-a57c-cc6fa26a6cf2 from network info_cache is no longer associated with instance in Neutron. Removing from network info_cache.
  2015-11-03 23:28:38.677 17000 ERROR nova.compute.manager [req-46519d0e-6e80-40e1-bbb3-dc39184eb046 41f42dfc41f9428fb143623f0a83d2fa 726f4a1ce23f4f12acb9139dcfcdb313 - - -] [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] Setting instance vm_state to ERROR
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] Traceback (most recent call last):
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 4350, in finish_resize
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]     disk_info, image)
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 4247, in _finish_resize
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]     resize_instance = False
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7866, in macs_for_instance
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/pci/manager.py", line 380, in get_instance_pci_devs
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/objects/base.py", line 72, in getter
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/objects/instance.py", line 1022, in obj_load_attr
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/objects/instance.py", line 904, in _load_generic
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/objects/base.py", line 161, in wrapper
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/conductor/rpcapi.py", line 335, in object_class_action
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "/usr/lib64/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]     retry=self.retry)
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "/usr/lib64/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]     timeout=timeout, retry=retry)
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "/usr/lib64/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]     retry=retry)
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "/usr/lib64/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 341, in _send
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]     raise result
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] InstanceNotFound_Remote: Instance 21a4ff59-9057-4bc2-8b5e-185374c2d557 could not be found.
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] Traceback (most recent call last):
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] 
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/conductor/manager.py", line 426, in _object_dispatch
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] 
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/objects/base.py", line 163, in wrapper
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] 
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/objects/instance.py", line 581, in get_by_uuid
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] 
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/db/api.py", line 664, in instance_get_by_uuid
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] 
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 233, in wrapper
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] 
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1757, in instance_get_by_uuid
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] 
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]   File "./usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1769, in _instance_get_by_uuid
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] 
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] InstanceNotFound: Instance 21a4ff59-9057-4bc2-8b5e-185374c2d557 could not be found.
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] 
  2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] 
  2015-11-03 23:28:39.053 17000 ERROR oslo_messaging.rpc.dispatcher [req-46519d0e-6e80-40e1-bbb3-dc39184eb046 41f42dfc41f9428fb143623f0a83d2fa 726f4a1ce23f4f12acb9139dcfcdb313 - - -] Exception during message handling: Instance 21a4ff59-9057-4bc2-8b5e-185374c2d557 could not be found.
  Traceback (most recent call last):

    File "./usr/lib64/python2.7/site-
  packages/nova/conductor/manager.py", line 426, in _object_dispatch

    File "./usr/lib64/python2.7/site-packages/nova/objects/base.py",
  line 163, in wrapper

    File "./usr/lib64/python2.7/site-packages/nova/objects/instance.py",
  line 581, in get_by_uuid

    File "./usr/lib64/python2.7/site-packages/nova/db/api.py", line 664,
  in instance_get_by_uuid

    File "./usr/lib64/python2.7/site-
  packages/nova/db/sqlalchemy/api.py", line 233, in wrapper

    File "./usr/lib64/python2.7/site-
  packages/nova/db/sqlalchemy/api.py", line 1757, in
  instance_get_by_uuid

    File "./usr/lib64/python2.7/site-
  packages/nova/db/sqlalchemy/api.py", line 1769, in
  _instance_get_by_uuid

  InstanceNotFound: Instance 21a4ff59-9057-4bc2-8b5e-185374c2d557 could not be found.
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 7263, in finish_resize
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     reservations, migration)
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/exception.py", line 88, in wrapped
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/exception.py", line 71, in wrapped
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 345, in decorated_function
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance_uuid=instance_uuid)
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 316, in decorated_function
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 395, in decorated_function
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 304, in decorated_function
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     migration.instance_uuid, exc_info=True)
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 280, in decorated_function
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 361, in decorated_function
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 4362, in finish_resize
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     self._set_instance_error_state(context, instance)
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 4350, in finish_resize
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     disk_info, image)
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 4247, in _finish_resize
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     resize_instance = False
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7866, in macs_for_instance
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/pci/manager.py", line 380, in get_instance_pci_devs
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/objects/base.py", line 72, in getter
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/objects/instance.py", line 1022, in obj_load_attr
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/objects/instance.py", line 904, in _load_generic
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/objects/base.py", line 161, in wrapper
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/conductor/rpcapi.py", line 335, in object_class_action
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     retry=self.retry)
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     timeout=timeout, retry=retry)
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     retry=retry)
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 341, in _send
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher     raise result
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher InstanceNotFound_Remote: Instance 21a4ff59-9057-4bc2-8b5e-185374c2d557 could not be found.
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher 
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/conductor/manager.py", line 426, in _object_dispatch
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher 
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/objects/base.py", line 163, in wrapper
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher 
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/objects/instance.py", line 581, in get_by_uuid
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher 
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/db/api.py", line 664, in instance_get_by_uuid
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher 
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 233, in wrapper
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher 
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1757, in instance_get_by_uuid
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher 
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher   File "./usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1769, in _instance_get_by_uuid
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher 
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher InstanceNotFound: Instance 21a4ff59-9057-4bc2-8b5e-185374c2d557 could not be found.
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher 
  2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher 
  2015-11-03 23:28:39.165 17000 INFO nova.scheduler.client.report [req-2d9812a5-eadf-4eb2-96a3-d46e496b292d - - - - -] Compute_service record updated for ('compute-1', 'compute-1')

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1512907/+subscriptions