← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #73358

[Bug 1777540] [NEW] "IndexError: list index out of range" in ExceptionPayload.from_exception during resize failure

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

I noticed this from the nova-compute logs attached to bug 1777157:

2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [req-69050681-be73-48c5-be25-993f05b0b47e e511323a68a740f597880a1b056e3918 28f4e0797d544d758700e5be42aa82c6 - default default] [instance: daa5b10b-0655-4178-9413-66697004d7f8] Setting instance vm_state to ERROR: IndexError: list index out of range
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] Traceback (most recent call last):
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 7445, in _error_out_instance_on_exception
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]     yield
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 4162, in prep_resize
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]     filter_properties, host_list)
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 4221, in _reschedule_resize_or_reraise
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]     exception=exc_info[1])
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/rpc.py", line 225, in wrapped
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]     return f(*args, **kwargs)
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/utils.py", line 367, in notify_about_instance_action
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]     fault, priority = _get_fault_and_priority_from_exc(exception)
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/utils.py", line 348, in _get_fault_and_priority_from_exc
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]     exception)
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/notifications/objects/exception.py", line 42, in from_exception
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]     trace = inspect.trace()[-1]
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] IndexError: list index out of range
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] 

This is based on queens, and this is the problem:

https://github.com/openstack/nova/blob/530c24596a74a6115f2e944176c46e4f3d199811/nova/compute/manager.py#L4216

When we call that, we're not in an except block, so this fails later:

https://github.com/openstack/nova/blob/530c24596a74a6115f2e944176c46e4f3d199811/nova/notifications/objects/exception.py#L42

Although, looking higher up the stack, we are in an except block here:

https://github.com/openstack/nova/blob/530c24596a74a6115f2e944176c46e4f3d199811/nova/compute/manager.py#L4155

So this is a bit weird...

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: notifications

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1777540

Title:
  "IndexError: list index out of range" in
  ExceptionPayload.from_exception during resize failure

Status in OpenStack Compute (nova):
  New

Bug description:
  I noticed this from the nova-compute logs attached to bug 1777157:

  2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [req-69050681-be73-48c5-be25-993f05b0b47e e511323a68a740f597880a1b056e3918 28f4e0797d544d758700e5be42aa82c6 - default default] [instance: daa5b10b-0655-4178-9413-66697004d7f8] Setting instance vm_state to ERROR: IndexError: list index out of range
  2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] Traceback (most recent call last):
  2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 7445, in _error_out_instance_on_exception
  2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]     yield
  2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 4162, in prep_resize
  2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]     filter_properties, host_list)
  2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 4221, in _reschedule_resize_or_reraise
  2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]     exception=exc_info[1])
  2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/rpc.py", line 225, in wrapped
  2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]     return f(*args, **kwargs)
  2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/utils.py", line 367, in notify_about_instance_action
  2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]     fault, priority = _get_fault_and_priority_from_exc(exception)
  2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/utils.py", line 348, in _get_fault_and_priority_from_exc
  2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]     exception)
  2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/notifications/objects/exception.py", line 42, in from_exception
  2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]     trace = inspect.trace()[-1]
  2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] IndexError: list index out of range
  2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] 

  This is based on queens, and this is the problem:

  https://github.com/openstack/nova/blob/530c24596a74a6115f2e944176c46e4f3d199811/nova/compute/manager.py#L4216

  When we call that, we're not in an except block, so this fails later:

  https://github.com/openstack/nova/blob/530c24596a74a6115f2e944176c46e4f3d199811/nova/notifications/objects/exception.py#L42

  Although, looking higher up the stack, we are in an except block here:

  https://github.com/openstack/nova/blob/530c24596a74a6115f2e944176c46e4f3d199811/nova/compute/manager.py#L4155

  So this is a bit weird...

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1777540/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next