yahoo-eng-team team mailing list archive

[OpenStack Infra <1860555@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/nova/+/926407
Committed: https://opendev.org/openstack/nova/commit/f8b98390dc99f6cb0101c88223eb840e0d1c7124
Submitter: "Zuul (22348)"
Branch:    master

commit f8b98390dc99f6cb0101c88223eb840e0d1c7124
Author: Balazs Gibizer <gibi@xxxxxxxxxx>
Date:   Thu Aug 15 13:06:39 2024 +0200

    Fix PCI passthrough cleanup on reschedule
    
    The resource tracker Claim object works on a copy of the instance object
    got from the compute manager. But the PCI claim logic does not use the
    copy but use the original instance object. However the abort claim logic
    including the abort PCI claim logic worked on the copy only. Therefore the
    claimed PCI devices are visible to the compute manager in the
    instance.pci_decives list even after the claim is aborted.
    
    There was another bug in the PCIDevice object where the instance object
    wasn't passed to the free() function and therefore the
    instance.pci_devices list wasn't updated when the device was freed.
    
    Closes-Bug: #1860555
    Change-Id: Iff343d4d78996cd17a6a584fefa7071c81311673


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1860555

Title:
  PCI passthrough reschedule race condition

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Steps to reproduce
  ------------------

  Create multiple instances concurrently using a flavor with a PCI
  passthrough request (--property
  "pci_passthrough:alias"="<alias>:<count>"), and a scheduler hint with
  some anti-affinity constraint.

  Expected result
  ---------------

  The instances are created successfully, and each have the expected
  number of PCI devices attached.

  Actual result
  -------------

  Sometimes, instances may fail during creation, or may be created with
  more PCI devices than requested.

  Environment
  -----------

  Nova 18.2.2 (rocky), CentOS 7, libvirt, deployed by kolla-ansible.

  Analysis
  --------

  If an instance with PCI passthrough devices is rescheduled (e.g. due to
  affinity violation), the instance can end up with extra PCI devices attached.
  If the devices selected on the original and subsequent compute nodes have the
  same address, the instance will fail to create, with the following error:

  libvirtError: internal error: Device 0000:89:00.0 is already in use

  However, if the devices are different, and all available on the first and
  second compute nodes, the VM may end up with additional hostdevs.

  On investigation, when the node is rescheduled, the instance object passed to
  the conductor RPC API contains the PCI devices that should have been freed.
  This is because the claim object holds a clone of the instance that is used to
  perform the abort on failure [1][2], and the PCI devices removed from its list are not
  reflected in the original object. There is a secondary issue that the PCI
  manager was not passing through the instance to the PCI object's free() method
  in all cases [3], resulting in the PCI device not being removed from the
  instance.pci_devices list.

  I have two alternative fixes for this issue, but they will need a
  little time to work their way out of an organisation. Essentially:

  1. pass the original instance (not the clone) to the abort function in the Claim.
  2. refresh the instance from DB when rescheduling

  The former is a more general solution, but I don't know the reasons
  for using a clone in the first place. The second works for
  reschedules, but may leave a hole for resize or migration. I haven't
  reproduced the issue in those cases but it seems possible that it
  would be present.

  [1] https://opendev.org/openstack/nova/src/branch/master/nova/compute/claims.py#L64
  [2] https://opendev.org/openstack/nova/src/branch/master/nova/compute/claims.py#L83
  [3] https://opendev.org/openstack/nova/src/branch/master/nova/pci/manager.py#L309

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1860555/+subscriptions