yahoo-eng-team team mailing list archive

[melanie witt <1719730@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Ran into this while hacking on something locally and running the server
groups functional tests:


==============================
Failed 1 tests - output below:
==============================

nova.tests.functional.test_server_group.ServerGroupTestV215.test_rebuild_with_anti_affinity
-------------------------------------------------------------------------------------------

Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
19:45:29,525 ERROR [nova.scheduler.utils] Error from last host: host2 (node host2): ['Traceback (most recent call last):\n', '  File "nova/compute/manager.py", line 1831, in _do_build_and_run_instance\n    filter_properties)\n', '  File "nova/compute/manager.py", line 2061, in _build_and_run_instance\n    instance_uuid=instance.uuid, reason=six.text_type(e))\n', 'RescheduledException: Build of instance c249e39f-0d38-40ce-860d-6c72cdeba436 was re-scheduled: Build of instance c249e39f-0d38-40ce-860d-6c72cdeba436 was re-scheduled: Anti-affinity instance group policy was violated.\n']
19:45:29,526 WARNING [nova.scheduler.utils] Failed to compute_task_build_instances: 'NoneType' object is not iterable
19:45:29,527 WARNING [nova.scheduler.utils] Setting instance to ERROR state.


Two instances are being booted simultaneously and both land on the same host, so the second one will fail the late affinity check and raise a RescheduledException to be rescheduled to another host. But conductor fails to do that because the 'group_members' key doesn't exist in filter_properties and an attempt to make a list out of it fails [1].

In the past, code [2] was added 'group_members' to filter_properties to
handle affinity and a more recent change removed most of it but missed
'group_members' [3]. So nothing is ever setting
filter_properties['group_members'] but RequestSpec.from_primitives()
expects it to be there and blows up trying to make a list from None.


[1] https://github.com/openstack/nova/blob/ad6d339/nova/objects/request_spec.py#L205 
[2] https://review.openstack.org/#/c/148277
[3] https://review.openstack.org/#/c/469037

** Affects: nova
     Importance: High
         Status: Confirmed

** Affects: nova/pike
     Importance: High
         Status: Confirmed


** Tags: affinity requestspec reschedule server-groups

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1719730

Title:
  Reschedule after the late affinity check fails with "'NoneType' object
  is not iterable"

Status in OpenStack Compute (nova):
  Confirmed
Status in OpenStack Compute (nova) pike series:
  Confirmed

Bug description:
  Ran into this while hacking on something locally and running the
  server groups functional tests:

  
  ==============================
  Failed 1 tests - output below:
  ==============================

  nova.tests.functional.test_server_group.ServerGroupTestV215.test_rebuild_with_anti_affinity
  -------------------------------------------------------------------------------------------

  Captured pythonlogging:
  ~~~~~~~~~~~~~~~~~~~~~~~
  19:45:29,525 ERROR [nova.scheduler.utils] Error from last host: host2 (node host2): ['Traceback (most recent call last):\n', '  File "nova/compute/manager.py", line 1831, in _do_build_and_run_instance\n    filter_properties)\n', '  File "nova/compute/manager.py", line 2061, in _build_and_run_instance\n    instance_uuid=instance.uuid, reason=six.text_type(e))\n', 'RescheduledException: Build of instance c249e39f-0d38-40ce-860d-6c72cdeba436 was re-scheduled: Build of instance c249e39f-0d38-40ce-860d-6c72cdeba436 was re-scheduled: Anti-affinity instance group policy was violated.\n']
  19:45:29,526 WARNING [nova.scheduler.utils] Failed to compute_task_build_instances: 'NoneType' object is not iterable
  19:45:29,527 WARNING [nova.scheduler.utils] Setting instance to ERROR state.

  
  Two instances are being booted simultaneously and both land on the same host, so the second one will fail the late affinity check and raise a RescheduledException to be rescheduled to another host. But conductor fails to do that because the 'group_members' key doesn't exist in filter_properties and an attempt to make a list out of it fails [1].

  In the past, code [2] was added 'group_members' to filter_properties
  to handle affinity and a more recent change removed most of it but
  missed 'group_members' [3]. So nothing is ever setting
  filter_properties['group_members'] but RequestSpec.from_primitives()
  expects it to be there and blows up trying to make a list from None.

  
  [1] https://github.com/openstack/nova/blob/ad6d339/nova/objects/request_spec.py#L205 
  [2] https://review.openstack.org/#/c/148277
  [3] https://review.openstack.org/#/c/469037

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1719730/+subscriptions