yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #90532
[Bug 1951656] Re: Nova fails to parse new libvirt mediated device name format
Reviewed: https://review.opendev.org/c/openstack/nova/+/838976
Committed: https://opendev.org/openstack/nova/commit/a28b907c4f0dbba6e141a8fbea807e6cb0438977
Submitter: "Zuul (22348)"
Branch: master
commit a28b907c4f0dbba6e141a8fbea807e6cb0438977
Author: Billy Olsen <billy.olsen@xxxxxxxxx>
Date: Thu Apr 21 19:42:27 2022 -0700
Handle mdev devices in libvirt 7.7+
Libvirt 7.7 changed the mdev device naming to include the parent PCI
device when listing node devices. The domain, however, will still only
see the UUID and not see the parent PCI device. Changing the parsing to
simply drop the PCI identifier is not enough as the device cannot be
found when attempting to lookup the new ID.
Modify the Libvirt Driver's _get_mediated_device_information to tolerate
different formats of the mdev name. This first uses the legacy behavior
by trying to lookup the device name that is passed in (typically
mdev_<uuid> format) and if that is not found, iterates the list of mdev
node devices until the right UUID is found and selects that one.
Note that the lookup of the mdev device by UUID are needed in order
to keep the ability to recreate assigned mediated devices on a reboot of
the compute node.
Additionally, the libvirt utils parsing method mdev_name2uuid, has
been updated to tolerate both mdev_<uuid> and mdev_<uuid>_<pciid>
formats.
Closes-Bug: 1951656
Change-Id: Ifed0fa16053228990a6a8df8d4c666521db7e329
** Changed in: nova
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1951656
Title:
Nova fails to parse new libvirt mediated device name format
Status in OpenStack Compute (nova):
Fix Released
Bug description:
The name format of mediated devices in libvirt was recently changed from `mdev_<uuid>` to `mdev_<uuid>_<parent>`, e.g.:
Old: `mdev_a12c7bf8_fcf4_4c3b_a256_604cda8e62d5`
New: `mdev_a12c7bf8_fcf4_4c3b_a256_604cda8e62d5_0000_c1_00_0`
This results in the following error:
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager [req-570c7e8f-0540-49fb-b2b0-8c2ac932e4dc - - - - -] Error updating resources for node: ValueError: badly formed hexadecimal UUID string
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager Traceback (most recent call last):
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 9993, in _update_available_resource_for_node
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager startup=startup)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 895, in update_available_resource
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 360, in inner
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager return f(*args, **kwargs)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 975, in _update_available_resource
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager self._update(context, cn, startup=startup)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1227, in _update
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/retrying.py", line 49, in wrapped_f
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/retrying.py", line 206, in call
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager return attempt.get(self._wrap_exception)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/retrying.py", line 247, in get
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2])
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/usr/local/lib/python3.6/site-packages/six.py", line 719, in reraise
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager raise value
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/retrying.py", line 200, in call
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1163, in _update_to_placement
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager self.driver.update_provider_tree(prov_tree, nodename)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 8355, in update_provider_tree
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager provider_tree, nodename, allocations=allocations)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 8757, in _update_provider_tree_for_vgpu
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager inventories_dict = self._get_gpu_inventories()
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7597, in _get_gpu_inventories
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager count_per_parent = self._count_mediated_devices(enabled_mdev_types)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7538, in _count_mediated_devices
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager mediated_devices = self._get_mediated_devices(types=enabled_mdev_types)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7788, in _get_mediated_devices
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager device = self._get_mediated_device_information(name)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7769, in _get_mediated_device_information
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager "uuid": libvirt_utils.mdev_name2uuid(cfgdev.name),
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/utils.py", line 583, in mdev_name2uuid
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager return str(uuid.UUID(mdev_name[5:].replace('_', '-')))
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/usr/lib64/python3.6/uuid.py", line 140, in __init__
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager raise ValueError('badly formed hexadecimal UUID string')
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager ValueError: badly formed hexadecimal UUID string
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager
Nova is assuming everything after `mdev_` is the UUID, but this string
now includes the parent PCI address.
Relevant commit in libvirt:
https://github.com/libvirt/libvirt/commit/3bd8181bc5548a0ce81107cbfb480dfdcba5679d
Environment: qemu/kvm, latest openstack xena (kolla)
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1951656/+subscriptions
References
