yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1951656] [NEW] Nova fails to parse new libvirt mediated device name format

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Joe Kralicky <1951656@xxxxxxxxxxxxxxxxxx>
Date: Fri, 19 Nov 2021 23:32:57 -0000
Reply-to: Bug 1951656 <1951656@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

The name format of mediated devices in libvirt was recently changed from `mdev_<uuid>` to `mdev_<uuid>_<parent>`, e.g.: 
Old: `mdev_a12c7bf8_fcf4_4c3b_a256_604cda8e62d5`
New: `mdev_a12c7bf8_fcf4_4c3b_a256_604cda8e62d5_0000_c1_00_0`


This results in the following error:

2021-11-19 22:51:45.952 7 ERROR nova.compute.manager [req-570c7e8f-0540-49fb-b2b0-8c2ac932e4dc - - - - -] Error updating resources for node: ValueError: badly formed hexadecimal UUID string 
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager Traceback (most recent call last):                                                                                                                 
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 9993, in _update_available_resource_for_node                
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     startup=startup)                                                                                                                               
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 895, in update_available_resource                  
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     self._update_available_resource(context, resources, startup=startup)                                                                           
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 360, in inner                                         
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     return f(*args, **kwargs)                                                                                                                      
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 975, in _update_available_resource                 
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     self._update(context, cn, startup=startup)                                                                                                     
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1227, in _update                                   
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     self._update_to_placement(context, compute_node, startup)                                                                                      
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/retrying.py", line 49, in wrapped_f                                                        
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     return Retrying(*dargs, **dkw).call(f, *args, **kw)                                                                                            
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/retrying.py", line 206, in call                                                            
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     return attempt.get(self._wrap_exception)                                                                                                       
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/retrying.py", line 247, in get                                                             
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     six.reraise(self.value[0], self.value[1], self.value[2])                                                                                       
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/usr/local/lib/python3.6/site-packages/six.py", line 719, in reraise                                                                       
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     raise value                                                                                                                                    
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/retrying.py", line 200, in call                                                            
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)                                                                                  
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1163, in _update_to_placement                      
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     self.driver.update_provider_tree(prov_tree, nodename)                                                                                          
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 8355, in update_provider_tree                           
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     provider_tree, nodename, allocations=allocations)                                                                                              
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 8757, in _update_provider_tree_for_vgpu                 
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     inventories_dict = self._get_gpu_inventories()                                                                                                 
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7597, in _get_gpu_inventories                           
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     count_per_parent = self._count_mediated_devices(enabled_mdev_types)                                                                            
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7538, in _count_mediated_devices                        
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     mediated_devices = self._get_mediated_devices(types=enabled_mdev_types)                                                                        
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7788, in _get_mediated_devices                          
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     device = self._get_mediated_device_information(name)                                                                                           
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7769, in _get_mediated_device_information               
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     "uuid": libvirt_utils.mdev_name2uuid(cfgdev.name),                                                                                             
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/utils.py", line 583, in mdev_name2uuid                                   
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     return str(uuid.UUID(mdev_name[5:].replace('_', '-')))                                                                                         
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/usr/lib64/python3.6/uuid.py", line 140, in __init__                                                                                       
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     raise ValueError('badly formed hexadecimal UUID string')                                                                                       
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager ValueError: badly formed hexadecimal UUID string                                                                                                   
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager                                                                                                                                                    

Nova is assuming everything after `mdev_` is the UUID, but this string
now includes the parent PCI address.

Relevant commit in libvirt:
https://github.com/libvirt/libvirt/commit/3bd8181bc5548a0ce81107cbfb480dfdcba5679d

Environment: qemu/kvm, latest openstack xena (kolla)

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1951656

Title:
  Nova fails to parse new libvirt mediated device name format

Status in OpenStack Compute (nova):
  New

Bug description:
  The name format of mediated devices in libvirt was recently changed from `mdev_<uuid>` to `mdev_<uuid>_<parent>`, e.g.: 
  Old: `mdev_a12c7bf8_fcf4_4c3b_a256_604cda8e62d5`
  New: `mdev_a12c7bf8_fcf4_4c3b_a256_604cda8e62d5_0000_c1_00_0`

  
  This results in the following error:

  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager [req-570c7e8f-0540-49fb-b2b0-8c2ac932e4dc - - - - -] Error updating resources for node: ValueError: badly formed hexadecimal UUID string 
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager Traceback (most recent call last):                                                                                                                 
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 9993, in _update_available_resource_for_node                
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     startup=startup)                                                                                                                               
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 895, in update_available_resource                  
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     self._update_available_resource(context, resources, startup=startup)                                                                           
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 360, in inner                                         
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     return f(*args, **kwargs)                                                                                                                      
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 975, in _update_available_resource                 
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     self._update(context, cn, startup=startup)                                                                                                     
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1227, in _update                                   
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     self._update_to_placement(context, compute_node, startup)                                                                                      
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/retrying.py", line 49, in wrapped_f                                                        
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     return Retrying(*dargs, **dkw).call(f, *args, **kw)                                                                                            
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/retrying.py", line 206, in call                                                            
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     return attempt.get(self._wrap_exception)                                                                                                       
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/retrying.py", line 247, in get                                                             
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     six.reraise(self.value[0], self.value[1], self.value[2])                                                                                       
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/usr/local/lib/python3.6/site-packages/six.py", line 719, in reraise                                                                       
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     raise value                                                                                                                                    
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/retrying.py", line 200, in call                                                            
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)                                                                                  
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1163, in _update_to_placement                      
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     self.driver.update_provider_tree(prov_tree, nodename)                                                                                          
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 8355, in update_provider_tree                           
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     provider_tree, nodename, allocations=allocations)                                                                                              
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 8757, in _update_provider_tree_for_vgpu                 
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     inventories_dict = self._get_gpu_inventories()                                                                                                 
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7597, in _get_gpu_inventories                           
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     count_per_parent = self._count_mediated_devices(enabled_mdev_types)                                                                            
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7538, in _count_mediated_devices                        
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     mediated_devices = self._get_mediated_devices(types=enabled_mdev_types)                                                                        
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7788, in _get_mediated_devices                          
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     device = self._get_mediated_device_information(name)                                                                                           
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7769, in _get_mediated_device_information               
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     "uuid": libvirt_utils.mdev_name2uuid(cfgdev.name),                                                                                             
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/utils.py", line 583, in mdev_name2uuid                                   
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     return str(uuid.UUID(mdev_name[5:].replace('_', '-')))                                                                                         
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager   File "/usr/lib64/python3.6/uuid.py", line 140, in __init__                                                                                       
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager     raise ValueError('badly formed hexadecimal UUID string')                                                                                       
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager ValueError: badly formed hexadecimal UUID string                                                                                                   
  2021-11-19 22:51:45.952 7 ERROR nova.compute.manager                                                                                                                                                    

  Nova is assuming everything after `mdev_` is the UUID, but this string
  now includes the parent PCI address.

  Relevant commit in libvirt:
  https://github.com/libvirt/libvirt/commit/3bd8181bc5548a0ce81107cbfb480dfdcba5679d

  Environment: qemu/kvm, latest openstack xena (kolla)

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1951656/+subscriptions
Follow ups

[Bug 1951656] Re: Nova fails to parse new libvirt mediated device name format
From: OpenStack Infra, 2022-11-29