yahoo-eng-team team mailing list archive

[OpenStack Infra <1870096@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/713863
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5ab9ef11e27014ce8b43e1bac76903fed70d0fbf
Submitter: Zuul
Branch:    master

commit 5ab9ef11e27014ce8b43e1bac76903fed70d0fbf
Author: Johannes Kulik <johannes.kulik@xxxxxxx>
Date:   Thu Mar 19 12:51:25 2020 +0100

    Don't recompute weighers' minval/maxval attributes
    
    Changing the minval/maxval attribute to the minimum/maxium of every
    weigher run changes the outcome of future runs. We noticed it in the
    SoftAffinityWeigher, where a previous run with a host hosting a lot of
    instances for a server-group would make a later run use that maximum.
    This resulted in the weight being lower than 1 for a host hosting all
    instances of another server-group, if the number of instances of that
    server-group on that host is less than a previous server-group's
    instances on any host.
    
    Previously, there were two places that computed the maxval/minval - once
    in normalize() and once in weigh_objects() - but only the one in
    weigh_objects() saved the values to the weigher.
    
    The code now uses the maxval/minval as defined by the weigher and keeps
    the weights inside the maxval-minval range. There's also only one place
    to compute the minval/maxval now, if the weigher did not set a value:
    normalize().
    
    Closes-Bug: 1870096
    
    Change-Id: I60a90dabcd21b4e049e218c7c55fa075bb7ff933


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1870096

Title:
  soft-affinity weight not normalized base on server group's maximum

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) pike series:
  New
Status in OpenStack Compute (nova) queens series:
  New
Status in OpenStack Compute (nova) rocky series:
  New
Status in OpenStack Compute (nova) stein series:
  New
Status in OpenStack Compute (nova) train series:
  New

Bug description:
  Description
  ===========

  When using soft-affinity to schedule instances on the same host, the
  weight is unexpectedly low if a server was previously scheduled to any
  server-group with more members on a host. This low weight can then be
  easily outweighed by differences in resources (e.g. RAM/CPU).

  Steps to reproduce
  ==================

  Do not restart nova-scheduler in the process or the bug doesn't
  appear. You need to change the ServerGroupSoftAffinityWeigher to
  actually log the weights it computes to see the problem.

  * Create a server-group with soft-affinity (let's call it A)
  * Create 6 servers in server-group A, one after the other so they end up on the same host.
  * Create another server-group with soft-affinity (B)
  * Create 1 server in server-group B
  * Create 1 server in server-group B and look at the scheduler's weights assigned to the hosts by the ServerGroupSoftAffinityWeigher.

  Expected result
  ===============

  The weight assigned to the host by the ServerGroupSoftAffinityWeigher
  should be 1, as the maximum number of instances for server-group B is
  on that host (the one we created there before).

  Actual result
  =============
  The weight assigned to the host by the ServerGroupSoftAffinityWeigher is 0.2, as the maximum number of instances ever encountered on a host is 5.

  Environment
  ===========

  We noticed this on a queens version of nova a year ago. Can't give the
  exact commit anymore, but the code still looks broken in current
  master.

  I've opened a review-request for fixing this bug here:
  https://review.opendev.org/#/c/713863/

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1870096/+subscriptions