yahoo-eng-team team mailing list archive

[OpenStack Infra <1762876@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/560454
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4afa8c2b97929faaf8bbf2bbaa72235fab7d3d13
Submitter: Zuul
Branch:    master

commit 4afa8c2b97929faaf8bbf2bbaa72235fab7d3d13
Author: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date:   Wed Apr 11 10:43:34 2018 -0400

    Fix race fail in test_resize_with_reschedule_then_live_migrate
    
    The assertion in the test that the migration status is 'completed'
    is flawed in that it assumes when the instance status is 'ACTIVE'
    the migration is completed, which isn't True, since the instance
    status gets changed before the migration is completed, but they are
    very close in time so there is a race, which is how this test slipped
    by. This fixes the issue by polling the migration status until it
    is actually completed or we timeout.
    
    Change-Id: I61f745667f4c003d7e3ca6f2f9a99194930ac892
    Closes-Bug: #1762876


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1762876

Title:
  test_resize_with_reschedule_then_live_migrate intermittently failing;
  migration is not yet complete

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  I noticed this in a stable/pike functional test job run:

  http://logs.openstack.org/46/560146/2/check/nova-tox-
  functional/4a9d1fd/job-output.txt.gz#_2018-04-10_21_37_20_943583

  2018-04-10 21:37:20.944928 | ubuntu-xenial | Captured stderr:
  2018-04-10 21:37:20.944966 | ubuntu-xenial | ~~~~~~~~~~~~~~~~
  2018-04-10 21:37:20.945029 | ubuntu-xenial |     Traceback (most recent call last):
  2018-04-10 21:37:20.945231 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 457, in fire_timers
  2018-04-10 21:37:20.945268 | ubuntu-xenial |         timer()
  2018-04-10 21:37:20.945467 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 58, in __call__
  2018-04-10 21:37:20.945513 | ubuntu-xenial |         cb(*args, **kw)
  2018-04-10 21:37:20.945598 | ubuntu-xenial |       File "nova/utils.py", line 1030, in context_wrapper
  2018-04-10 21:37:20.945650 | ubuntu-xenial |         func(*args, **kwargs)
  2018-04-10 21:37:20.945756 | ubuntu-xenial |       File "nova/compute/manager.py", line 5620, in dispatch_live_migration
  2018-04-10 21:37:20.945839 | ubuntu-xenial |         self._do_live_migration(*args, **kwargs)
  2018-04-10 21:37:20.945939 | ubuntu-xenial |       File "nova/compute/manager.py", line 5599, in _do_live_migration
  2018-04-10 21:37:20.945993 | ubuntu-xenial |         clean_task_state=True)
  2018-04-10 21:37:20.946194 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-04-10 21:37:20.946246 | ubuntu-xenial |         self.force_reraise()
  2018-04-10 21:37:20.946452 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-04-10 21:37:20.946532 | ubuntu-xenial |         six.reraise(self.type_, self.value, self.tb)
  2018-04-10 21:37:20.946679 | ubuntu-xenial |       File "nova/compute/manager.py", line 5588, in _do_live_migration
  2018-04-10 21:37:20.946764 | ubuntu-xenial |         block_migration, migrate_data)
  2018-04-10 21:37:20.946856 | ubuntu-xenial |       File "nova/virt/fake.py", line 497, in live_migration
  2018-04-10 21:37:20.946901 | ubuntu-xenial |         migrate_data)
  2018-04-10 21:37:20.947003 | ubuntu-xenial |       File "nova/exception_wrapper.py", line 76, in wrapped
  2018-04-10 21:37:20.947069 | ubuntu-xenial |         function_name, call_dict, binary)
  2018-04-10 21:37:20.947270 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-04-10 21:37:20.947322 | ubuntu-xenial |         self.force_reraise()
  2018-04-10 21:37:20.947536 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-04-10 21:37:20.947619 | ubuntu-xenial |         six.reraise(self.type_, self.value, self.tb)
  2018-04-10 21:37:20.947707 | ubuntu-xenial |       File "nova/exception_wrapper.py", line 67, in wrapped
  2018-04-10 21:37:20.947779 | ubuntu-xenial |         return f(self, context, *args, **kw)
  2018-04-10 21:37:20.947878 | ubuntu-xenial |       File "nova/compute/manager.py", line 218, in decorated_function
  2018-04-10 21:37:20.947950 | ubuntu-xenial |         kwargs['instance'], e, sys.exc_info())
  2018-04-10 21:37:20.948160 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-04-10 21:37:20.948214 | ubuntu-xenial |         self.force_reraise()
  2018-04-10 21:37:20.948433 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-04-10 21:37:20.948510 | ubuntu-xenial |         six.reraise(self.type_, self.value, self.tb)
  2018-04-10 21:37:20.948607 | ubuntu-xenial |       File "nova/compute/manager.py", line 206, in decorated_function
  2018-04-10 21:37:20.948687 | ubuntu-xenial |         return function(self, context, *args, **kwargs)
  2018-04-10 21:37:20.948802 | ubuntu-xenial |       File "nova/compute/manager.py", line 5824, in _post_live_migration
  2018-04-10 21:37:20.948856 | ubuntu-xenial |         instance, source_node)
  2018-04-10 21:37:20.949000 | ubuntu-xenial |       File "nova/compute/resource_tracker.py", line 1294, in delete_allocation_for_migrated_instance
  2018-04-10 21:37:20.949111 | ubuntu-xenial |         self._delete_allocation_for_moved_instance(instance, node, 'migrated')
  2018-04-10 21:37:20.949244 | ubuntu-xenial |       File "nova/compute/resource_tracker.py", line 1302, in _delete_allocation_for_moved_instance
  2018-04-10 21:37:20.949319 | ubuntu-xenial |         cn_uuid = self.compute_nodes[node].uuid
  2018-04-10 21:37:20.949363 | ubuntu-xenial |     KeyError: u'host3'

  But I see it's also happening on master:

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22cn_uuid%20%3D%20self.compute_nodes%5Bnode%5D.uuid%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1762876/+subscriptions