yahoo-eng-team team mailing list archive

[OpenStack Infra <2098892@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/nova/+/942334
Committed: https://opendev.org/openstack/nova/commit/2c07aa06458a2e70d79f8b3c68960bdccd11ab57
Submitter: "Zuul (22348)"
Branch:    master

commit 2c07aa06458a2e70d79f8b3c68960bdccd11ab57
Author: melanie witt <melwittt@xxxxxxxxx>
Date:   Thu Feb 20 04:09:44 2025 +0000

    libvirt: Fix regression of listDevices() return type
    
    This a partial revert of change
    I60d6f04d374e9ede5895a43b7a75e955b0fea3c5 which added tpool.Proxy
    wrapping to the listDevices() and listAllDevices() methods.
    
    The regression was caught during downstream testing with vGPUs and the
    update_available_resource() periodic task was failing with:
    
      TypeError: virNodeDeviceLookupByName() argument 2 must be str or
        None, not Proxy
    
    It turns out that while the listAllDevices() method returns a list of
    virNodeDevice objects [1], the listDevices() method returns a list of
    string names [2] and is generated from the corresponding function in C
    [3].
    
    The error was not caught by unit or functional testing because those
    test environments intentionally do not import the libvirt Python
    module -- so mocked code in the LibvirtFixture runs instead. Also, the
    update_available_resource() method has a 'except Exception:' at the end
    which logs an error but does not re-raise. So it would not cause a
    functional test to fail.
    
    This reverts the change that caused the regression, updates potentially
    confusing docstrings, adds type annotations to the methods that use
    listDevices(), and moves the nodeDeviceLookupByName type checking into
    the LibvirtFixture.
    
    Closes-Bug: #2098892
    
    [1] https://github.com/libvirt/libvirt-python/blob/408815a/libvirt-override-virConnect.py#L520-L524
    [2] https://github.com/libvirt/libvirt-python/blob/408815a/libvirt-override-api.xml#L448-L453
    [3] https://libvirt.org/html/libvirt-libvirt-nodedev.html#virNodeListDevices
    
    Change-Id: Ib5befdd3c13367daa208ff969f66cba693ae2c76


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2098892

Title:
  "TypeError: virNodeDeviceLookupByName() argument 2 must be str or
  None, not Proxy" from update_available_resource with mdevs

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) 2024.1 series:
  New
Status in OpenStack Compute (nova) 2024.2 series:
  New
Status in OpenStack Compute (nova) antelope series:
  New
Status in OpenStack Compute (nova) bobcat series:
  New

Bug description:
  The patch meant to fix https://bugs.launchpad.net/nova/+bug/2091033
  unfortunately has broken the nova-compute update_available_resource
  periodic task when mdevs are involved with a traceback like shown
  below.

  We are working on a partial revert that will be backported to the same
  branches where the original patch was merged.

  
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager Traceback (most recent call last):
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 10530, in _update_available_resource_for_node
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager self.rt.update_available_resource(context, nodename,
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/compute/resource_tracker.py", line 889, in update_available_resource
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/oslo_concurrency/lockutils.py", line 414, in inner
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager return f(*args, **kwargs)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/compute/resource_tracker.py", line 994, in _update_available_resource
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager self._update(context, cn, startup=startup)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/compute/resource_tracker.py", line 1303, in _update
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/retrying.py", line 49, in wrapped_f
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/retrying.py", line 206, in call
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager return attempt.get(self._wrap_exception)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/retrying.py", line 247, in get
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2])
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/six.py", line 709, in reraise
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager raise value
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/retrying.py", line 200, in call
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/compute/resource_tracker.py", line 1216, in _update_to_placement
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager self.driver.update_provider_tree(prov_tree, nodename)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 9120, in update_provider_tree
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager self._update_provider_tree_for_vgpu(
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 9529, in _update_provider_tree_for_vgpu
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager inventories_dict = self._get_gpu_inventories()
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 8199, in _get_gpu_inventories
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager count_per_dev = self._count_mdev_capable_devices(enabled_mdev_types)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 8150, in _count_mdev_capable_devices
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager mdev_capable_devices = self._get_mdev_capable_devices(
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 8392, in _get_mdev_capable_devices
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager device = self._get_mdev_capabilities_for_dev(name, types)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 8361, in _get_mdev_capabilities_for_dev
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager virtdev = self._host.device_lookup_by_name(devname)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/host.py", line 1253, in device_lookup_by_name
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager return self.get_connection().nodeDeviceLookupByName(name)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 193, in doit
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager result = proxy_call(self._autowrap, f, *args, **kwargs)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 151, in proxy_call
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager rv = execute(f, *args, **kwargs)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 132, in execute
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager six.reraise(c, e, tb)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/six.py", line 709, in reraise
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager raise value
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 86, in tworker
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager rv = meth(*args, **kwargs)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib64/python3.9/site-packages/libvirt.py", line 5201, in nodeDeviceLookupByName
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager ret = libvirtmod.virNodeDeviceLookupByName(self._o, name)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager TypeError: virNodeDeviceLookupByName() argument 2 must be str or None, not Proxy

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2098892/+subscriptions