yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #82805
[Bug 1881455] [NEW] migrate server reporting list index of out bound
Public bug reported:
Description
============
When resize to local host enabled and do a cold migration sometimes
fails with
1. migrating to same host failed
2. and then a list index out of bound error
Steps to reproduce
===================
deploy two compute nodes and make workload imbalance, for example compute01 has more allocations
than compute02. Then migrate server on compute02.
Expected result
================
cold migration succeeded
actual result
==============
sometimes failed
log
======
8084-4fa8-a3c4-2874555fb27c held by migration 0a8a29a5-7f9c-4af3-85a1-ea62ee5658c3 for instance
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [req-ee48014a-51c1-4e82-9ef3-e3b68a9a34e4 5f0b0ff35b914c84b24efb363965530d 0606e9bf4e9c4334b6cb9a5012c60fb8 - default default] [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] Error: Unable to migrate instance (
8189fa53-3e8a-42e3-a735-1d91b9ff0c3b) to current host (compute02).: UnableToMigrateToSelf: Unable to migrate instance (8189fa53-3e8a-42e3-a735-1d91b9ff0c3b) to current host (compute02).
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] Traceback (most recent call last):
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4555, in prep_resize
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] node, migration, clean_shutdown)
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4499, in _prep_resize
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] instance_id=instance.uuid, host=self.host)
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] UnableToMigrateToSelf: Unable to migrate instance (8189fa53-3e8a-42e3-a735-1d91b9ff0c3b) to current host (compute02).
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [req-ee48014a-51c1-4e82-9ef3-e3b68a9a34e4 5f0b0ff35b914c84b24efb363965530d 0606e9bf4e9c4334b6cb9a5012c60fb8 - default default] [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] Error: Unable to migrate instance (
8189fa53-3e8a-42e3-a735-1d91b9ff0c3b) to current host (compute02).: UnableToMigrateToSelf: Unable to migrate instance (8189fa53-3e8a-42e3-a735-1d91b9ff0c3b) to current host (compute02).
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] Traceback (most recent call last):
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4555, in prep_resize
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] node, migration, clean_shutdown)
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4499, in _prep_resize
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] instance_id=instance.uuid, host=self.host)
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] UnableToMigrateToSelf: Unable to migrate instance (8189fa53-3e8a-42e3-a735-1d91b9ff0c3b) to current host (compute02).
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [req-ee48014a-51c1-4e82-9ef3-e3b68a9a34e4 5f0b0ff35b914c84b24efb363965530d 0606e9bf4e9c4334b6cb9a5012c60fb8 - default default] [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] Setting instance vm_state to ERROR:
IndexError: list index out of range
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] Traceback (most recent call last):
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 8333, in _error_out_instance_on_exception
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] yield
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4576, in prep_resize
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] filter_properties, host_list)
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4637, in _reschedule_resize_or_reraise
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] tb=','.join(traceback.format_exception(*exc_info)))
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] File "/var/lib/openstack/lib/python2.7/site-packages/nova/rpc.py", line 231, in wrapped
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] return f(*args, **kwargs)
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/utils.py", line 442, in notify_about_instance_action
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] fault, priority = _get_fault_and_priority_from_exc_and_tb(exception, tb)
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/utils.py", line 422, in _get_fault_and_priority_from_exc_and_tb
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] exception, tb)
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] File "/var/lib/openstack/lib/python2.7/site-packages/nova/notifications/objects/exception.py", line 45, in from_exc_and_traceback
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] trace = inspect.trace()[-1]
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] IndexError: list index out of range
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1881455
Title:
migrate server reporting list index of out bound
Status in OpenStack Compute (nova):
New
Bug description:
Description
============
When resize to local host enabled and do a cold migration sometimes
fails with
1. migrating to same host failed
2. and then a list index out of bound error
Steps to reproduce
===================
deploy two compute nodes and make workload imbalance, for example compute01 has more allocations
than compute02. Then migrate server on compute02.
Expected result
================
cold migration succeeded
actual result
==============
sometimes failed
log
======
8084-4fa8-a3c4-2874555fb27c held by migration 0a8a29a5-7f9c-4af3-85a1-ea62ee5658c3 for instance
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [req-ee48014a-51c1-4e82-9ef3-e3b68a9a34e4 5f0b0ff35b914c84b24efb363965530d 0606e9bf4e9c4334b6cb9a5012c60fb8 - default default] [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] Error: Unable to migrate instance (
8189fa53-3e8a-42e3-a735-1d91b9ff0c3b) to current host (compute02).: UnableToMigrateToSelf: Unable to migrate instance (8189fa53-3e8a-42e3-a735-1d91b9ff0c3b) to current host (compute02).
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] Traceback (most recent call last):
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4555, in prep_resize
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] node, migration, clean_shutdown)
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4499, in _prep_resize
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] instance_id=instance.uuid, host=self.host)
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] UnableToMigrateToSelf: Unable to migrate instance (8189fa53-3e8a-42e3-a735-1d91b9ff0c3b) to current host (compute02).
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [req-ee48014a-51c1-4e82-9ef3-e3b68a9a34e4 5f0b0ff35b914c84b24efb363965530d 0606e9bf4e9c4334b6cb9a5012c60fb8 - default default] [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] Error: Unable to migrate instance (
8189fa53-3e8a-42e3-a735-1d91b9ff0c3b) to current host (compute02).: UnableToMigrateToSelf: Unable to migrate instance (8189fa53-3e8a-42e3-a735-1d91b9ff0c3b) to current host (compute02).
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] Traceback (most recent call last):
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4555, in prep_resize
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] node, migration, clean_shutdown)
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4499, in _prep_resize
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] instance_id=instance.uuid, host=self.host)
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] UnableToMigrateToSelf: Unable to migrate instance (8189fa53-3e8a-42e3-a735-1d91b9ff0c3b) to current host (compute02).
2020-05-31 02:55:51.649 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [req-ee48014a-51c1-4e82-9ef3-e3b68a9a34e4 5f0b0ff35b914c84b24efb363965530d 0606e9bf4e9c4334b6cb9a5012c60fb8 - default default] [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] Setting instance vm_state to ERROR:
IndexError: list index out of range
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] Traceback (most recent call last):
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 8333, in _error_out_instance_on_exception
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] yield
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4576, in prep_resize
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] filter_properties, host_list)
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4637, in _reschedule_resize_or_reraise
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] tb=','.join(traceback.format_exception(*exc_info)))
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] File "/var/lib/openstack/lib/python2.7/site-packages/nova/rpc.py", line 231, in wrapped
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] return f(*args, **kwargs)
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/utils.py", line 442, in notify_about_instance_action
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] fault, priority = _get_fault_and_priority_from_exc_and_tb(exception, tb)
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/utils.py", line 422, in _get_fault_and_priority_from_exc_and_tb
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] exception, tb)
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] File "/var/lib/openstack/lib/python2.7/site-packages/nova/notifications/objects/exception.py", line 45, in from_exc_and_traceback
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] trace = inspect.trace()[-1]
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b] IndexError: list index out of range
2020-05-31 02:55:51.740 2419133 ERROR nova.compute.manager [instance: 8189fa53-3e8a-42e3-a735-1d91b9ff0c3b]
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1881455/+subscriptions
Follow ups