yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2098892] Re: "TypeError: virNodeDeviceLookupByName() argument 2 must be str or None, not Proxy" from update_available_resource with mdevs

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Edward Hope-Morley <2098892@xxxxxxxxxxxxxxxxxx>
Date: Thu, 10 Jul 2025 11:26:38 -0000
Reply-to: Bug 2098892 <2098892@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
@tjaalton apologies I have updated this bug with the sru information.
For sru submission/debdiff please see bug 2091033 where I have submitted
a single debdiff that contains both patches since they depend on each
other and that way I could test them together.

** Description changed:

+ [Impact]
+ 
+ The fix in bug 2091033 led to a regression that is fixed by this patch.
+ In order to backport the former we also need this patch so both are
+ being proposed for SRU at the same time.
+ 
+ [Test Plan]
+ 
+  * Deploy Openstack Yoga and ensure that "TypeError: virNodeDeviceLookupByName() argument 2 must be str or None, not Proxy" does not appear in /var/log/nova/nova-compute.log
+  * The default candence for the resource tracker is 60s so need to wait at least that long for this to occur.
+  * Once you see the log "Auditing locally available compute resources for" the resource tracker has started and it finishes with "Hypervisor/Node resource view:"
+ 
+ [Regression Potential]
+ 
+ This patch addresses a regression identified in the original version of
+ the fix for bug 2091033 and is not expected to cause any regressions in
+ itself.
+ 
+ --------------------------------------------------------------------------
+ 
  The patch meant to fix https://bugs.launchpad.net/nova/+bug/2091033
  unfortunately has broken the nova-compute update_available_resource
  periodic task when mdevs are involved with a traceback like shown below.
  
  We are working on a partial revert that will be backported to the same
  branches where the original patch was merged.
- 
  
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager Traceback (most recent call last):
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 10530, in _update_available_resource_for_node
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager self.rt.update_available_resource(context, nodename,
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/compute/resource_tracker.py", line 889, in update_available_resource
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/oslo_concurrency/lockutils.py", line 414, in inner
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager return f(*args, **kwargs)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/compute/resource_tracker.py", line 994, in _update_available_resource
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager self._update(context, cn, startup=startup)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/compute/resource_tracker.py", line 1303, in _update
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/retrying.py", line 49, in wrapped_f
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/retrying.py", line 206, in call
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager return attempt.get(self._wrap_exception)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/retrying.py", line 247, in get
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2])
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/six.py", line 709, in reraise
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager raise value
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/retrying.py", line 200, in call
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/compute/resource_tracker.py", line 1216, in _update_to_placement
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager self.driver.update_provider_tree(prov_tree, nodename)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 9120, in update_provider_tree
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager self._update_provider_tree_for_vgpu(
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 9529, in _update_provider_tree_for_vgpu
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager inventories_dict = self._get_gpu_inventories()
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 8199, in _get_gpu_inventories
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager count_per_dev = self._count_mdev_capable_devices(enabled_mdev_types)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 8150, in _count_mdev_capable_devices
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager mdev_capable_devices = self._get_mdev_capable_devices(
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 8392, in _get_mdev_capable_devices
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager device = self._get_mdev_capabilities_for_dev(name, types)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 8361, in _get_mdev_capabilities_for_dev
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager virtdev = self._host.device_lookup_by_name(devname)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/host.py", line 1253, in device_lookup_by_name
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager return self.get_connection().nodeDeviceLookupByName(name)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 193, in doit
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager result = proxy_call(self._autowrap, f, *args, **kwargs)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 151, in proxy_call
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager rv = execute(f, *args, **kwargs)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 132, in execute
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager six.reraise(c, e, tb)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/six.py", line 709, in reraise
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager raise value
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 86, in tworker
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager rv = meth(*args, **kwargs)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib64/python3.9/site-packages/libvirt.py", line 5201, in nodeDeviceLookupByName
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager ret = libvirtmod.virNodeDeviceLookupByName(self._o, name)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager TypeError: virNodeDeviceLookupByName() argument 2 must be str or None, not Proxy

** Also affects: nova (Ubuntu)
   Importance: Undecided
       Status: New

** Also affects: nova (Ubuntu Noble)
   Importance: Undecided
       Status: New

** Also affects: nova (Ubuntu Jammy)
   Importance: Undecided
       Status: New

** Also affects: cloud-archive
   Importance: Undecided
       Status: New

** Also affects: cloud-archive/caracal
   Importance: Undecided
       Status: New

** Also affects: cloud-archive/yoga
   Importance: Undecided
       Status: New

** Also affects: cloud-archive/dalmatian
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2098892

Title:
  "TypeError: virNodeDeviceLookupByName() argument 2 must be str or
  None, not Proxy" from update_available_resource with mdevs

Status in Ubuntu Cloud Archive:
  New
Status in Ubuntu Cloud Archive caracal series:
  New
Status in Ubuntu Cloud Archive dalmatian series:
  New
Status in Ubuntu Cloud Archive yoga series:
  New
Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) 2024.1 series:
  Fix Committed
Status in OpenStack Compute (nova) 2024.2 series:
  Fix Committed
Status in OpenStack Compute (nova) antelope series:
  In Progress
Status in OpenStack Compute (nova) bobcat series:
  Fix Released
Status in nova package in Ubuntu:
  New
Status in nova source package in Jammy:
  New
Status in nova source package in Noble:
  New

Bug description:
  [Impact]

  The fix in bug 2091033 led to a regression that is fixed by this
  patch. In order to backport the former we also need this patch so both
  are being proposed for SRU at the same time.

  [Test Plan]

   * Deploy Openstack Yoga and ensure that "TypeError: virNodeDeviceLookupByName() argument 2 must be str or None, not Proxy" does not appear in /var/log/nova/nova-compute.log
   * The default candence for the resource tracker is 60s so need to wait at least that long for this to occur.
   * Once you see the log "Auditing locally available compute resources for" the resource tracker has started and it finishes with "Hypervisor/Node resource view:"

  [Regression Potential]

  This patch addresses a regression identified in the original version
  of the fix for bug 2091033 and is not expected to cause any
  regressions in itself.

  --------------------------------------------------------------------------

  The patch meant to fix https://bugs.launchpad.net/nova/+bug/2091033
  unfortunately has broken the nova-compute update_available_resource
  periodic task when mdevs are involved with a traceback like shown
  below.

  We are working on a partial revert that will be backported to the same
  branches where the original patch was merged.

  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager Traceback (most recent call last):
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 10530, in _update_available_resource_for_node
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager self.rt.update_available_resource(context, nodename,
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/compute/resource_tracker.py", line 889, in update_available_resource
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/oslo_concurrency/lockutils.py", line 414, in inner
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager return f(*args, **kwargs)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/compute/resource_tracker.py", line 994, in _update_available_resource
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager self._update(context, cn, startup=startup)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/compute/resource_tracker.py", line 1303, in _update
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/retrying.py", line 49, in wrapped_f
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/retrying.py", line 206, in call
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager return attempt.get(self._wrap_exception)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/retrying.py", line 247, in get
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2])
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/six.py", line 709, in reraise
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager raise value
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/retrying.py", line 200, in call
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/compute/resource_tracker.py", line 1216, in _update_to_placement
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager self.driver.update_provider_tree(prov_tree, nodename)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 9120, in update_provider_tree
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager self._update_provider_tree_for_vgpu(
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 9529, in _update_provider_tree_for_vgpu
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager inventories_dict = self._get_gpu_inventories()
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 8199, in _get_gpu_inventories
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager count_per_dev = self._count_mdev_capable_devices(enabled_mdev_types)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 8150, in _count_mdev_capable_devices
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager mdev_capable_devices = self._get_mdev_capable_devices(
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 8392, in _get_mdev_capable_devices
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager device = self._get_mdev_capabilities_for_dev(name, types)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 8361, in _get_mdev_capabilities_for_dev
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager virtdev = self._host.device_lookup_by_name(devname)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/host.py", line 1253, in device_lookup_by_name
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager return self.get_connection().nodeDeviceLookupByName(name)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 193, in doit
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager result = proxy_call(self._autowrap, f, *args, **kwargs)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 151, in proxy_call
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager rv = execute(f, *args, **kwargs)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 132, in execute
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager six.reraise(c, e, tb)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/six.py", line 709, in reraise
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager raise value
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 86, in tworker
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager rv = meth(*args, **kwargs)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager File "/usr/lib64/python3.9/site-packages/libvirt.py", line 5201, in nodeDeviceLookupByName
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager ret = libvirtmod.virNodeDeviceLookupByName(self._o, name)
  2025-02-18 20:43:33.462 2 ERROR nova.compute.manager TypeError: virNodeDeviceLookupByName() argument 2 must be str or None, not Proxy

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2098892/+subscriptions
References

[Bug 2098892] [NEW] "TypeError: virNodeDeviceLookupByName() argument 2 must be str or None, not Proxy" from update_available_resource with mdevs
From: melanie witt, 2025-02-19