yahoo-eng-team team mailing list archive

[OpenStack Infra <1719730@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/507938
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9d6632a67d91fb3c5145c14ac38011e919d6d8c0
Submitter: Jenkins
Branch:    master

commit 9d6632a67d91fb3c5145c14ac38011e919d6d8c0
Author: melanie witt <melwittt@xxxxxxxxx>
Date:   Wed Sep 27 17:27:56 2017 +0000

    Set group_members when converting to legacy request spec
    
    In Pike we converted the affinity filter code to use the RequestSpec
    object instead of legacy dicts. The filter used to populate server
    group info in the filter_properties and the conversion removed that.
    However, in the conductor, we are still converting RequestSpec back
    and forth between object and primitive, and there is a mismatch
    between the keys being set/get in filter_properties. So during a
    reschedule with a server group, we hit an exception
    "'NoneType' object is not iterable" in the RequestSpec.from_primitives
    method and the reschedule fails.
    
    This adds 'group_members' to the _to_legacy_group_info method to set
    the key.
    
    Closes-Bug: #1719730
    
    Change-Id: Icb418f2be575bb2ba82756fdeb67b24a28950746


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1719730

Title:
  Reschedule after the late affinity check fails with "'NoneType' object
  is not iterable"

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) pike series:
  Confirmed

Bug description:
  Ran into this while hacking on something locally and running the
  server groups functional tests:

  
  ==============================
  Failed 1 tests - output below:
  ==============================

  nova.tests.functional.test_server_group.ServerGroupTestV215.test_rebuild_with_anti_affinity
  -------------------------------------------------------------------------------------------

  Captured pythonlogging:
  ~~~~~~~~~~~~~~~~~~~~~~~
  19:45:29,525 ERROR [nova.scheduler.utils] Error from last host: host2 (node host2): ['Traceback (most recent call last):\n', '  File "nova/compute/manager.py", line 1831, in _do_build_and_run_instance\n    filter_properties)\n', '  File "nova/compute/manager.py", line 2061, in _build_and_run_instance\n    instance_uuid=instance.uuid, reason=six.text_type(e))\n', 'RescheduledException: Build of instance c249e39f-0d38-40ce-860d-6c72cdeba436 was re-scheduled: Build of instance c249e39f-0d38-40ce-860d-6c72cdeba436 was re-scheduled: Anti-affinity instance group policy was violated.\n']
  19:45:29,526 WARNING [nova.scheduler.utils] Failed to compute_task_build_instances: 'NoneType' object is not iterable
  19:45:29,527 WARNING [nova.scheduler.utils] Setting instance to ERROR state.

  
  Two instances are being booted simultaneously and both land on the same host, so the second one will fail the late affinity check and raise a RescheduledException to be rescheduled to another host. But conductor fails to do that because the 'group_members' key doesn't exist in filter_properties and an attempt to make a list out of it fails [1].

  In the past, code [2] was added 'group_members' to filter_properties
  to handle affinity and a more recent change removed most of it but
  missed 'group_members' [3]. So nothing is ever setting
  filter_properties['group_members'] but RequestSpec.from_primitives()
  expects it to be there and blows up trying to make a list from None.

  
  [1] https://github.com/openstack/nova/blob/ad6d339/nova/objects/request_spec.py#L205 
  [2] https://review.openstack.org/#/c/148277
  [3] https://review.openstack.org/#/c/469037

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1719730/+subscriptions