yahoo-eng-team team mailing list archive

[OpenStack Infra <1986838@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/nova/+/853611
Committed: https://opendev.org/openstack/nova/commit/2b447b7236f95752d00ebcee8c32cfef4850cf5d
Submitter: "Zuul (22348)"
Branch:    master

commit 2b447b7236f95752d00ebcee8c32cfef4850cf5d
Author: Balazs Gibizer <gibi@xxxxxxxxxx>
Date:   Wed Aug 17 18:19:15 2022 +0200

    Trigger reschedule if PCI consumption fail on compute
    
    The PciPassthroughFilter logic checks each InstancePCIRequest
    individually against the available PCI pools of a given host and given
    boot request. So it is possible that the scheduler accepts a host that
    has a single PCI device available even if two devices are requested for
    a single instance via two separate PCI aliases. Then the PCI claim on
    the compute detects this but does not stop the boot just logs an ERROR.
    This results in the instance booted without any PCI device.
    
    This patch does two things:
    1) changes the PCI claim to fail with an exception and trigger a
       re-schedule instead of just logging an ERROR.
    2) change the PciDeviceStats.support_requests that is called during
       scheduling to not just filter pools for individual requests but also
       consume the request from the pool within the scope of a single boot
       request.
    
    The fix in #2) would not be enough alone as two parallel scheduling
    request could race for a single device on the same host. #1) is the
    ultimate place where we consume devices under a compute global lock so
    we need the fix there too.
    
    Closes-Bug: #1986838
    Change-Id: Iea477be57ae4e95dfc03acc9368f31d4be895343


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1986838

Title:
  Booting with two identical PCI aliases on a host with a single
  matching dev succeeds but the instance will have no PCI allocations

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Detected during reading the code.

  Reproduction
  1) configure a host with a single PCI passthrough device
  2) configure two PCI aliases (a1, a2) with different names but each matching the above device
  3) boot an instance with 'pci_passthrough:alias': 'a1:1,a2:1' flavor extra_spec.

  Expected result
  The instance fails to schedule

  Actual result
  The instance schedules to the host but has no PCI allocations
  The nova scheduler logs:
  Selected host: compute1 failed to consume from instance. Error: PCI device request [InstancePCIRequest(alias_name='a1',count=1,is_new=<?>,numa_policy='legacy',request_id=None,requester_id=<?>,spec=[{product_id='1533',vendor_id='8086'}]), InstancePCIRequest(alias_name='a2',count=1,is_new=<?>,numa_policy='legacy',request_id=None,requester_id=<?>,spec=[{product_id='1533',vendor_id='8086'}])] failed

  The nova compute logs:
  Failed to allocate PCI devices for instance. Unassigning devices back to pools. This should not happen, since the scheduler should have accurate information, and allocation during claims is controlled via a hold on the compute node semaphore.

  I think the root cause of the fault is that the
  PciDeviceStats.support_requests() [1] call matches each
  InstancePCIRequest object independently to the available PCI pools and
  does not update the status of the pools locally.

  I will push a functional reproduction test shortly.

  [1]
  https://github.com/openstack/nova/blob/69bc4c38d1c5b98fcbbe8b16a7dfeb654e3b8173/nova/pci/stats.py#L645

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1986838/+subscriptions