yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1762876] [NEW] KeyError during move operation functional tests

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Wed, 11 Apr 2018 00:43:57 -0000
Reply-to: Bug 1762876 <1762876@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I noticed this in a stable/pike functional test job run:

http://logs.openstack.org/46/560146/2/check/nova-tox-functional/4a9d1fd
/job-output.txt.gz#_2018-04-10_21_37_20_943583

2018-04-10 21:37:20.944928 | ubuntu-xenial | Captured stderr:
2018-04-10 21:37:20.944966 | ubuntu-xenial | ~~~~~~~~~~~~~~~~
2018-04-10 21:37:20.945029 | ubuntu-xenial |     Traceback (most recent call last):
2018-04-10 21:37:20.945231 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 457, in fire_timers
2018-04-10 21:37:20.945268 | ubuntu-xenial |         timer()
2018-04-10 21:37:20.945467 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 58, in __call__
2018-04-10 21:37:20.945513 | ubuntu-xenial |         cb(*args, **kw)
2018-04-10 21:37:20.945598 | ubuntu-xenial |       File "nova/utils.py", line 1030, in context_wrapper
2018-04-10 21:37:20.945650 | ubuntu-xenial |         func(*args, **kwargs)
2018-04-10 21:37:20.945756 | ubuntu-xenial |       File "nova/compute/manager.py", line 5620, in dispatch_live_migration
2018-04-10 21:37:20.945839 | ubuntu-xenial |         self._do_live_migration(*args, **kwargs)
2018-04-10 21:37:20.945939 | ubuntu-xenial |       File "nova/compute/manager.py", line 5599, in _do_live_migration
2018-04-10 21:37:20.945993 | ubuntu-xenial |         clean_task_state=True)
2018-04-10 21:37:20.946194 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-04-10 21:37:20.946246 | ubuntu-xenial |         self.force_reraise()
2018-04-10 21:37:20.946452 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-04-10 21:37:20.946532 | ubuntu-xenial |         six.reraise(self.type_, self.value, self.tb)
2018-04-10 21:37:20.946679 | ubuntu-xenial |       File "nova/compute/manager.py", line 5588, in _do_live_migration
2018-04-10 21:37:20.946764 | ubuntu-xenial |         block_migration, migrate_data)
2018-04-10 21:37:20.946856 | ubuntu-xenial |       File "nova/virt/fake.py", line 497, in live_migration
2018-04-10 21:37:20.946901 | ubuntu-xenial |         migrate_data)
2018-04-10 21:37:20.947003 | ubuntu-xenial |       File "nova/exception_wrapper.py", line 76, in wrapped
2018-04-10 21:37:20.947069 | ubuntu-xenial |         function_name, call_dict, binary)
2018-04-10 21:37:20.947270 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-04-10 21:37:20.947322 | ubuntu-xenial |         self.force_reraise()
2018-04-10 21:37:20.947536 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-04-10 21:37:20.947619 | ubuntu-xenial |         six.reraise(self.type_, self.value, self.tb)
2018-04-10 21:37:20.947707 | ubuntu-xenial |       File "nova/exception_wrapper.py", line 67, in wrapped
2018-04-10 21:37:20.947779 | ubuntu-xenial |         return f(self, context, *args, **kw)
2018-04-10 21:37:20.947878 | ubuntu-xenial |       File "nova/compute/manager.py", line 218, in decorated_function
2018-04-10 21:37:20.947950 | ubuntu-xenial |         kwargs['instance'], e, sys.exc_info())
2018-04-10 21:37:20.948160 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-04-10 21:37:20.948214 | ubuntu-xenial |         self.force_reraise()
2018-04-10 21:37:20.948433 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-04-10 21:37:20.948510 | ubuntu-xenial |         six.reraise(self.type_, self.value, self.tb)
2018-04-10 21:37:20.948607 | ubuntu-xenial |       File "nova/compute/manager.py", line 206, in decorated_function
2018-04-10 21:37:20.948687 | ubuntu-xenial |         return function(self, context, *args, **kwargs)
2018-04-10 21:37:20.948802 | ubuntu-xenial |       File "nova/compute/manager.py", line 5824, in _post_live_migration
2018-04-10 21:37:20.948856 | ubuntu-xenial |         instance, source_node)
2018-04-10 21:37:20.949000 | ubuntu-xenial |       File "nova/compute/resource_tracker.py", line 1294, in delete_allocation_for_migrated_instance
2018-04-10 21:37:20.949111 | ubuntu-xenial |         self._delete_allocation_for_moved_instance(instance, node, 'migrated')
2018-04-10 21:37:20.949244 | ubuntu-xenial |       File "nova/compute/resource_tracker.py", line 1302, in _delete_allocation_for_moved_instance
2018-04-10 21:37:20.949319 | ubuntu-xenial |         cn_uuid = self.compute_nodes[node].uuid
2018-04-10 21:37:20.949363 | ubuntu-xenial |     KeyError: u'host3'

But I see it's also happening on master:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22cn_uuid%20%3D%20self.compute_nodes%5Bnode%5D.uuid%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d

** Affects: nova
     Importance: Medium
         Status: Confirmed


** Tags: functional testing

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1762876

Title:
  KeyError during move operation functional tests

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  I noticed this in a stable/pike functional test job run:

  http://logs.openstack.org/46/560146/2/check/nova-tox-
  functional/4a9d1fd/job-output.txt.gz#_2018-04-10_21_37_20_943583

  2018-04-10 21:37:20.944928 | ubuntu-xenial | Captured stderr:
  2018-04-10 21:37:20.944966 | ubuntu-xenial | ~~~~~~~~~~~~~~~~
  2018-04-10 21:37:20.945029 | ubuntu-xenial |     Traceback (most recent call last):
  2018-04-10 21:37:20.945231 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 457, in fire_timers
  2018-04-10 21:37:20.945268 | ubuntu-xenial |         timer()
  2018-04-10 21:37:20.945467 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 58, in __call__
  2018-04-10 21:37:20.945513 | ubuntu-xenial |         cb(*args, **kw)
  2018-04-10 21:37:20.945598 | ubuntu-xenial |       File "nova/utils.py", line 1030, in context_wrapper
  2018-04-10 21:37:20.945650 | ubuntu-xenial |         func(*args, **kwargs)
  2018-04-10 21:37:20.945756 | ubuntu-xenial |       File "nova/compute/manager.py", line 5620, in dispatch_live_migration
  2018-04-10 21:37:20.945839 | ubuntu-xenial |         self._do_live_migration(*args, **kwargs)
  2018-04-10 21:37:20.945939 | ubuntu-xenial |       File "nova/compute/manager.py", line 5599, in _do_live_migration
  2018-04-10 21:37:20.945993 | ubuntu-xenial |         clean_task_state=True)
  2018-04-10 21:37:20.946194 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-04-10 21:37:20.946246 | ubuntu-xenial |         self.force_reraise()
  2018-04-10 21:37:20.946452 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-04-10 21:37:20.946532 | ubuntu-xenial |         six.reraise(self.type_, self.value, self.tb)
  2018-04-10 21:37:20.946679 | ubuntu-xenial |       File "nova/compute/manager.py", line 5588, in _do_live_migration
  2018-04-10 21:37:20.946764 | ubuntu-xenial |         block_migration, migrate_data)
  2018-04-10 21:37:20.946856 | ubuntu-xenial |       File "nova/virt/fake.py", line 497, in live_migration
  2018-04-10 21:37:20.946901 | ubuntu-xenial |         migrate_data)
  2018-04-10 21:37:20.947003 | ubuntu-xenial |       File "nova/exception_wrapper.py", line 76, in wrapped
  2018-04-10 21:37:20.947069 | ubuntu-xenial |         function_name, call_dict, binary)
  2018-04-10 21:37:20.947270 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-04-10 21:37:20.947322 | ubuntu-xenial |         self.force_reraise()
  2018-04-10 21:37:20.947536 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-04-10 21:37:20.947619 | ubuntu-xenial |         six.reraise(self.type_, self.value, self.tb)
  2018-04-10 21:37:20.947707 | ubuntu-xenial |       File "nova/exception_wrapper.py", line 67, in wrapped
  2018-04-10 21:37:20.947779 | ubuntu-xenial |         return f(self, context, *args, **kw)
  2018-04-10 21:37:20.947878 | ubuntu-xenial |       File "nova/compute/manager.py", line 218, in decorated_function
  2018-04-10 21:37:20.947950 | ubuntu-xenial |         kwargs['instance'], e, sys.exc_info())
  2018-04-10 21:37:20.948160 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-04-10 21:37:20.948214 | ubuntu-xenial |         self.force_reraise()
  2018-04-10 21:37:20.948433 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-04-10 21:37:20.948510 | ubuntu-xenial |         six.reraise(self.type_, self.value, self.tb)
  2018-04-10 21:37:20.948607 | ubuntu-xenial |       File "nova/compute/manager.py", line 206, in decorated_function
  2018-04-10 21:37:20.948687 | ubuntu-xenial |         return function(self, context, *args, **kwargs)
  2018-04-10 21:37:20.948802 | ubuntu-xenial |       File "nova/compute/manager.py", line 5824, in _post_live_migration
  2018-04-10 21:37:20.948856 | ubuntu-xenial |         instance, source_node)
  2018-04-10 21:37:20.949000 | ubuntu-xenial |       File "nova/compute/resource_tracker.py", line 1294, in delete_allocation_for_migrated_instance
  2018-04-10 21:37:20.949111 | ubuntu-xenial |         self._delete_allocation_for_moved_instance(instance, node, 'migrated')
  2018-04-10 21:37:20.949244 | ubuntu-xenial |       File "nova/compute/resource_tracker.py", line 1302, in _delete_allocation_for_moved_instance
  2018-04-10 21:37:20.949319 | ubuntu-xenial |         cn_uuid = self.compute_nodes[node].uuid
  2018-04-10 21:37:20.949363 | ubuntu-xenial |     KeyError: u'host3'

  But I see it's also happening on master:

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22cn_uuid%20%3D%20self.compute_nodes%5Bnode%5D.uuid%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1762876/+subscriptions
Follow ups

[Bug 1762876] Re: test_resize_with_reschedule_then_live_migrate intermittently failing; migration is not yet complete
From: OpenStack Infra, 2018-04-12