yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem.os@xxxxxxxxx>]

Bug 1675676 has the same failure but from a different scenario, and goes
back to at least Newton.

** Also affects: nova/newton
   Importance: Undecided
       Status: New

** Also affects: nova/ocata
   Importance: Undecided
       Status: New

** Changed in: nova/ocata
     Assignee: (unassigned) => Matt Riedemann (mriedem)

** Changed in: nova/ocata
       Status: New => In Progress

** Changed in: nova/newton
       Status: New => Confirmed

** Changed in: nova/ocata
   Importance: Undecided => High

** Changed in: nova/newton
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1719730

Title:
  Reschedule after the late affinity check fails with "'NoneType' object
  is not iterable"

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) newton series:
  Confirmed
Status in OpenStack Compute (nova) ocata series:
  In Progress
Status in OpenStack Compute (nova) pike series:
  Fix Committed

Bug description:
  Ran into this while hacking on something locally and running the
  server groups functional tests:

  
  ==============================
  Failed 1 tests - output below:
  ==============================

  nova.tests.functional.test_server_group.ServerGroupTestV215.test_rebuild_with_anti_affinity
  -------------------------------------------------------------------------------------------

  Captured pythonlogging:
  ~~~~~~~~~~~~~~~~~~~~~~~
  19:45:29,525 ERROR [nova.scheduler.utils] Error from last host: host2 (node host2): ['Traceback (most recent call last):\n', '  File "nova/compute/manager.py", line 1831, in _do_build_and_run_instance\n    filter_properties)\n', '  File "nova/compute/manager.py", line 2061, in _build_and_run_instance\n    instance_uuid=instance.uuid, reason=six.text_type(e))\n', 'RescheduledException: Build of instance c249e39f-0d38-40ce-860d-6c72cdeba436 was re-scheduled: Build of instance c249e39f-0d38-40ce-860d-6c72cdeba436 was re-scheduled: Anti-affinity instance group policy was violated.\n']
  19:45:29,526 WARNING [nova.scheduler.utils] Failed to compute_task_build_instances: 'NoneType' object is not iterable
  19:45:29,527 WARNING [nova.scheduler.utils] Setting instance to ERROR state.

  
  Two instances are being booted simultaneously and both land on the same host, so the second one will fail the late affinity check and raise a RescheduledException to be rescheduled to another host. But conductor fails to do that because the 'group_members' key doesn't exist in filter_properties and an attempt to make a list out of it fails [1].

  In the past, code [2] was added 'group_members' to filter_properties
  to handle affinity and a more recent change removed most of it but
  missed 'group_members' [3]. So nothing is ever setting
  filter_properties['group_members'] but RequestSpec.from_primitives()
  expects it to be there and blows up trying to make a list from None.

  
  [1] https://github.com/openstack/nova/blob/ad6d339/nova/objects/request_spec.py#L205 
  [2] https://review.openstack.org/#/c/148277
  [3] https://review.openstack.org/#/c/469037

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1719730/+subscriptions