yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1881455] [NEW] migrate server reporting list index of out bound

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: norman shen <1881455@xxxxxxxxxxxxxxxxxx>
Date: Sun, 31 May 2020 03:07:38 -0000
Reply-to: Bug 1881455 <1881455@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
============

When resize to local host enabled and do a cold migration sometimes
fails with

1. migrating to same host failed
2. and then a list index out of bound error


Steps to reproduce
===================

deploy two compute nodes and make workload imbalance, for example compute01 has more allocations
than compute02. Then migrate server on compute02.

Expected result
================

cold migration succeeded

actual result
==============

sometimes failed

log
======

8084-4fa8-a3c4-2874555fb27c held by migration 0a8a29a5-7f9c-4af3-85a1-ea62ee5658c3 for instance
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [req-ee48014a-51c1-4e82-9ef3-e3b68a9a34e4 5f0b0ff35b914c84b24efb363965530d 0606e9bf4e9c4334b6cb9a5012c60fb8 - default default] [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] Error: Unable to migrate instance (
8189fa53-3e8a-42e3-a735-1d91b9ff0c3b) to current host (compute02).: UnableToMigrateToSelf: Unable to migrate instance (8189fa53-3e8a-42e3-a735-1d91b9ff0c3b) to current host (compute02).
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] Traceback (most recent call last):
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4555, in prep_resize
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]     node, migration, clean_shutdown)
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4499, in _prep_resize
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]     instance_id=instance.uuid, host=self.host)
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] UnableToMigrateToSelf: Unable to migrate instance (8189fa53-3e8a-42e3-a735-1d91b9ff0c3b) to current host (compute02).
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] 
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [req-ee48014a-51c1-4e82-9ef3-e3b68a9a34e4 5f0b0ff35b914c84b24efb363965530d 0606e9bf4e9c4334b6cb9a5012c60fb8 - default default] [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] Error: Unable to migrate instance (
8189fa53-3e8a-42e3-a735-1d91b9ff0c3b) to current host (compute02).: UnableToMigrateToSelf: Unable to migrate instance (8189fa53-3e8a-42e3-a735-1d91b9ff0c3b) to current host (compute02).
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] Traceback (most recent call last):
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4555, in prep_resize
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]     node, migration, clean_shutdown)
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4499, in _prep_resize
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]     instance_id=instance.uuid, host=self.host)
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] UnableToMigrateToSelf: Unable to migrate instance (8189fa53-3e8a-42e3-a735-1d91b9ff0c3b) to current host (compute02).
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] 
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [req-ee48014a-51c1-4e82-9ef3-e3b68a9a34e4 5f0b0ff35b914c84b24efb363965530d 0606e9bf4e9c4334b6cb9a5012c60fb8 - default default] [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] Setting instance vm_state to ERROR:
 IndexError: list index out of range
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] Traceback (most recent call last):
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 8333, in _error_out_instance_on_exception
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]     yield
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4576, in prep_resize
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]     filter_properties, host_list)
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4637, in _reschedule_resize_or_reraise
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]     tb=','.join(traceback.format_exception(*exc_info)))
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]   File "/var/lib/openstack/lib/python2.7/site-packages/nova/rpc.py", line 231, in wrapped
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]     return f(*args, **kwargs)
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/utils.py", line 442, in notify_about_instance_action
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]     fault, priority = _get_fault_and_priority_from_exc_and_tb(exception, tb)
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/utils.py", line 422, in _get_fault_and_priority_from_exc_and_tb
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]     exception, tb)
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]   File "/var/lib/openstack/lib/python2.7/site-packages/nova/notifications/objects/exception.py", line 45, in from_exc_and_traceback
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]     trace = inspect.trace()[-1]
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] IndexError: list index out of range
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1881455

Title:
  migrate server reporting list index of out bound

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ============

  When resize to local host enabled and do a cold migration sometimes
  fails with

  1. migrating to same host failed
  2. and then a list index out of bound error

  
  Steps to reproduce
  ===================

  deploy two compute nodes and make workload imbalance, for example compute01 has more allocations
  than compute02. Then migrate server on compute02.

  Expected result
  ================

  cold migration succeeded

  actual result
  ==============

  sometimes failed

  log
  ======

  8084-4fa8-a3c4-2874555fb27c held by migration 0a8a29a5-7f9c-4af3-85a1-ea62ee5658c3 for instance
  2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [req-ee48014a-51c1-4e82-9ef3-e3b68a9a34e4 5f0b0ff35b914c84b24efb363965530d 0606e9bf4e9c4334b6cb9a5012c60fb8 - default default] [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] Error: Unable to migrate instance (
  8189fa53-3e8a-42e3-a735-1d91b9ff0c3b) to current host (compute02).: UnableToMigrateToSelf: Unable to migrate instance (8189fa53-3e8a-42e3-a735-1d91b9ff0c3b) to current host (compute02).
  2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] Traceback (most recent call last):
  2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4555, in prep_resize
  2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]     node, migration, clean_shutdown)
  2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4499, in _prep_resize
  2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]     instance_id=instance.uuid, host=self.host)
  2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] UnableToMigrateToSelf: Unable to migrate instance (8189fa53-3e8a-42e3-a735-1d91b9ff0c3b) to current host (compute02).
  2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] 
  2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [req-ee48014a-51c1-4e82-9ef3-e3b68a9a34e4 5f0b0ff35b914c84b24efb363965530d 0606e9bf4e9c4334b6cb9a5012c60fb8 - default default] [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] Error: Unable to migrate instance (
  8189fa53-3e8a-42e3-a735-1d91b9ff0c3b) to current host (compute02).: UnableToMigrateToSelf: Unable to migrate instance (8189fa53-3e8a-42e3-a735-1d91b9ff0c3b) to current host (compute02).
  2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] Traceback (most recent call last):
  2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4555, in prep_resize
  2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]     node, migration, clean_shutdown)
  2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4499, in _prep_resize
  2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]     instance_id=instance.uuid, host=self.host)
  2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] UnableToMigrateToSelf: Unable to migrate instance (8189fa53-3e8a-42e3-a735-1d91b9ff0c3b) to current host (compute02).
  2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] 
  2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [req-ee48014a-51c1-4e82-9ef3-e3b68a9a34e4 5f0b0ff35b914c84b24efb363965530d 0606e9bf4e9c4334b6cb9a5012c60fb8 - default default] [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] Setting instance vm_state to ERROR:
   IndexError: list index out of range
  2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] Traceback (most recent call last):
  2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 8333, in _error_out_instance_on_exception
  2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]     yield
  2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4576, in prep_resize
  2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]     filter_properties, host_list)
  2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4637, in _reschedule_resize_or_reraise
  2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]     tb=','.join(traceback.format_exception(*exc_info)))
  2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]   File "/var/lib/openstack/lib/python2.7/site-packages/nova/rpc.py", line 231, in wrapped
  2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]     return f(*args, **kwargs)
  2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/utils.py", line 442, in notify_about_instance_action
  2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]     fault, priority = _get_fault_and_priority_from_exc_and_tb(exception, tb)
  2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/utils.py", line 422, in _get_fault_and_priority_from_exc_and_tb
  2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]     exception, tb)
  2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]   File "/var/lib/openstack/lib/python2.7/site-packages/nova/notifications/objects/exception.py", line 45, in from_exc_and_traceback
  2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]     trace = inspect.trace()[-1]
  2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] IndexError: list index out of range
  2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1881455/+subscriptions
Follow ups

[Bug 1881455] Related fix merged to nova (master)
From: OpenStack Infra, 2020-06-10