← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1981813] Re: Compute service fails to restart if the vnic_type of a bound port changed from direct to macvtap (CVE-2022-37394)

 

Reviewed:  https://review.opendev.org/c/openstack/nova/+/850003
Committed: https://opendev.org/openstack/nova/commit/e43bf900dc8ca66578603bed333c56b215b1876e
Submitter: "Zuul (22348)"
Branch:    master

commit e43bf900dc8ca66578603bed333c56b215b1876e
Author: Balazs Gibizer <gibi@xxxxxxxxxx>
Date:   Fri Jul 15 13:48:46 2022 +0200

    Gracefully ERROR in _init_instance if vnic_type changed
    
    If the vnic_type of a bound port changes from "direct" to "macvtap" and
    then the compute service is restarted then during _init_instance nova
    tries to plug the vif of the changed port. However as it now has macvtap
    vnic_type nova tries to look up the netdev of the parent VF. Still that
    VF is consumed by the instance so there is no such netdev on the host
    OS. This error killed the compute service at startup due to unhandled
    exception. This patch adds the exception handler, logs an ERROR and
    continue initializing other instances on the host.
    
    Also this patch adds a detailed ERROR log when nova detects that the
    vnic_type changed during _heal_instance_info_cache periodic.
    
    Closes-Bug: #1981813
    Change-Id: I1719f8eda04e8d15a3b01f0612977164c4e55e85


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1981813

Title:
  Compute service fails to restart if the vnic_type of a bound port
  changed from direct to macvtap (CVE-2022-37394)

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Security Advisory:
  In Progress

Bug description:
  We have a downstream bug report with the following reproduction steps:

  1) create a neutron port with vnic_type "direct"
  2) create an instance with that port
  3) after the instance is created successfully change the vnic_type of the bound port from "direct" to "macvtap". This is accepted by Neutron
  4) wait until the nova instance info caches is healed by the periodic task in nova-compute
  5) restart the nova-compute service.

  Actual behavior
  ---------------
  The nova-compute service fails to start with PciDeviceNotFoundById exception pointing to the PCI address of the VF the port is bound to on the host.

  Expected behavior
  -----------------
  The nova-compute service should start up successfully.

  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service Traceback (most recent call last):
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/pci/utils.py", line 167, in get_ifname_by_pci_address
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     dev_info = os.listdir(dev_path)
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service FileNotFoundError: [Errno 2] No such file or directory: '/sys/bus/pci/devices/0000:19:0a.7/net'
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service 
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service During handling of the above exception, another exception occurred:
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service 
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service Traceback (most recent call last):
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/usr/local/lib/python3.10/site-packages/oslo_service/service.py", line 806, in run_service
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     service.start()
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/service.py", line 159, in start
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     self.manager.init_host()
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/compute/manager.py", line 1536, in init_host
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     self._init_instance(context, instance)
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/compute/manager.py", line 1230, in _init_instance
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     self.driver.plug_vifs(instance, net_info)
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1386, in plug_vifs
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     self.vif_driver.plug(instance, vif)
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 730, in plug
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     self.plug_hw_veb(instance, vif)
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 628, in plug_hw_veb
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     set_vf_interface_vlan(
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 99, in set_vf_interface_vlan
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     vf_ifname = pci_utils.get_ifname_by_pci_address(pci_addr)
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/pci/utils.py", line 170, in get_ifname_by_pci_address
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     raise exception.PciDeviceNotFoundById(id=pci_addr)
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service nova.exception.PciDeviceNotFoundById: PCI device 0000:19:0a.7 not found
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1981813/+subscriptions



References