yahoo-eng-team team mailing list archive

[sean mooney <1893121@xxxxxxxxxxxxxxxxxx>]

this was fixed by
https://github.com/openstack/nova/commit/d13412648d011994a146dac1e7214ead3b82b31b

** Changed in: nova
       Status: Confirmed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1893121

Title:
  nova does not balance vm across numa node or prefer numa node with pci
  device when one is requested

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  the current implementation of numa has evolved over the years to
  support pci affinity policyes and numa affinity for other deivces like
  pmem.

  when numa was first intoduced the recomenation was to match the
  virtual numa toplogy of a guest to the numa toplogy of the host for
  best performance.

  in such a configuration the guest cpus and memory are evenly distubuted across the host numa nodes
  meanin that the memroy contoler and phsyicall cpus are consumed evenly.
  i.e. all vms do not use the cores form oh host numa node.

  if you create a vm with only hw:numa_nodes set and no other numa
  requests however due to how we currently iterage over host numa cells
  in a deterministic order the all vms will be placeed on numa node 0.

  if other vms also request numa resource like pinned cpus hw:cpu_policy=dedicated or explict pages size hw:mem_page_size<small|large|any|###> then the consumption of those resource will eventually
  cause those vms to loadblance onto the other numa nodes.

  as a reuslt the current behavior is to fill the first numa node before
  ever using resouces form the rest for numa vms using cpu pinnign or
  hugepages but numa vms that only request hw:numa_nodes wont be
  loadblanced.

  in both case this is suboptimal as it resulting in lower utilisation of the host hardware as
  the second and subsequent numa nodes will not be used untill the first numa node is full when using pinning and huge pages and will never be used for numa instance that dont request other numa resources.

  in a similar vain

  https://specs.openstack.org/openstack/nova-specs/specs/pike/implemented/reserve-numa-with-pci.html
  partly implemented a preferential sorting of host with pci devices.

  if the vm  did not request a pci device we weighter host with
  pcidevice lowere vs host without them.

  https://github.com/openstack/nova/blob/20459e3e88cb8382d450c7fdb042e2016d5560c5/nova/virt/hardware.py#L2268-L2275

  
  a full implemation would have on the selected host prefer putting the vms on numa nodes that had a pci device.

  as a result if a host has 2 numa nodes and the vm request a pci deice and 1 numa node.
  if the vm will fit on the first numa node (node0) and has the prefered police for pci affintiy we wont check or use the second numa node (node1).

  
  the fix for this is triviail add an else clause

   # If PCI device(s) are not required, prefer host cells that don't have
      # devices attached. Presence of a given numa_node in a PCI pool is
      # indicative of a PCI device being associated with that node
      if not pci_requests and pci_stats:
          # TODO(stephenfin): pci_stats can't be None here but mypy can't figure
          # that out for some reason
          host_cells = sorted(host_cells, key=lambda cell: cell.id in [
              pool['numa_node'] for pool in pci_stats.pools])  # type: ignore

  becomes

   # If PCI device(s) are not required, prefer host cells that don't have
      # devices attached. Presence of a given numa_node in a PCI pool is
      # indicative of a PCI device being associated with that node
      if not pci_requests and pci_stats:
          # TODO(stephenfin): pci_stats can't be None here but mypy can't figure
          # that out for some reason
          host_cells = sorted(host_cells, key=lambda cell: cell.id in [
              pool['numa_node'] for pool in pci_stats.pools])  # type: ignore
       else:
          host_cells = sorted(host_cells, key=lambda cell: cell.id in [
              pool['numa_node'] for pool in pci_stats.pools], reverse=True)  # type: ignore
   
  or more compactly

  
   # If PCI device(s) are not required, prefer host cells that don't have
      # devices attached. Presence of a given numa_node in a PCI pool is
      # indicative of a PCI device being associated with that node
      reverse =  pci_requests and pci_stats:
      # TODO(stephenfin): pci_stats can't be None here but mypy can't figure
      # that out for some reason
      host_cells = sorted(host_cells, key=lambda cell: cell.id in [
              pool['numa_node'] for pool in pci_stats.pools], reverse=reverse)  # type: ignore
      

  since python support stable sort orders complex sort can be achcive by
  multiple stables sorts

  https://docs.python.org/3/howto/sorting.html#sort-stability-and-
  complex-sorts

  so we can also adress the numa blanceing issue by 
  first sorting by instance per numa node
  then sorting by free memory per numa node
  then by cpus per numa node and finally 
  by pci device per numa node.

  this will allow nova to evenly distubtue vms optimally per numa node
  and also fully support the preference aspect of the preferred sriov
  numa affinity policy which currenlty only select a host that is
  capable of provideing numa affintiy but does not actully pferfer the
  numa node when we boot the vm.

  this bug applies to all currently supported release of nova.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1893121/+subscriptions