yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #89658
[Bug 1981813] Re: Compute service fails to restart if the vnic_type of a bound port changed from direct to macvtap (CVE-2022-37394)
Reviewed: https://review.opendev.org/c/openstack/nova/+/850003
Committed: https://opendev.org/openstack/nova/commit/e43bf900dc8ca66578603bed333c56b215b1876e
Submitter: "Zuul (22348)"
Branch: master
commit e43bf900dc8ca66578603bed333c56b215b1876e
Author: Balazs Gibizer <gibi@xxxxxxxxxx>
Date: Fri Jul 15 13:48:46 2022 +0200
Gracefully ERROR in _init_instance if vnic_type changed
If the vnic_type of a bound port changes from "direct" to "macvtap" and
then the compute service is restarted then during _init_instance nova
tries to plug the vif of the changed port. However as it now has macvtap
vnic_type nova tries to look up the netdev of the parent VF. Still that
VF is consumed by the instance so there is no such netdev on the host
OS. This error killed the compute service at startup due to unhandled
exception. This patch adds the exception handler, logs an ERROR and
continue initializing other instances on the host.
Also this patch adds a detailed ERROR log when nova detects that the
vnic_type changed during _heal_instance_info_cache periodic.
Closes-Bug: #1981813
Change-Id: I1719f8eda04e8d15a3b01f0612977164c4e55e85
** Changed in: nova
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1981813
Title:
Compute service fails to restart if the vnic_type of a bound port
changed from direct to macvtap (CVE-2022-37394)
Status in OpenStack Compute (nova):
Fix Released
Status in OpenStack Security Advisory:
In Progress
Bug description:
We have a downstream bug report with the following reproduction steps:
1) create a neutron port with vnic_type "direct"
2) create an instance with that port
3) after the instance is created successfully change the vnic_type of the bound port from "direct" to "macvtap". This is accepted by Neutron
4) wait until the nova instance info caches is healed by the periodic task in nova-compute
5) restart the nova-compute service.
Actual behavior
---------------
The nova-compute service fails to start with PciDeviceNotFoundById exception pointing to the PCI address of the VF the port is bound to on the host.
Expected behavior
-----------------
The nova-compute service should start up successfully.
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service Traceback (most recent call last):
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service File "/opt/stack/nova/nova/pci/utils.py", line 167, in get_ifname_by_pci_address
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service dev_info = os.listdir(dev_path)
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service FileNotFoundError: [Errno 2] No such file or directory: '/sys/bus/pci/devices/0000:19:0a.7/net'
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service During handling of the above exception, another exception occurred:
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service Traceback (most recent call last):
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service File "/usr/local/lib/python3.10/site-packages/oslo_service/service.py", line 806, in run_service
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service service.start()
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service File "/opt/stack/nova/nova/service.py", line 159, in start
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service self.manager.init_host()
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/manager.py", line 1536, in init_host
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service self._init_instance(context, instance)
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/manager.py", line 1230, in _init_instance
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service self.driver.plug_vifs(instance, net_info)
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1386, in plug_vifs
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service self.vif_driver.plug(instance, vif)
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 730, in plug
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service self.plug_hw_veb(instance, vif)
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 628, in plug_hw_veb
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service set_vf_interface_vlan(
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 99, in set_vf_interface_vlan
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service vf_ifname = pci_utils.get_ifname_by_pci_address(pci_addr)
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service File "/opt/stack/nova/nova/pci/utils.py", line 170, in get_ifname_by_pci_address
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service raise exception.PciDeviceNotFoundById(id=pci_addr)
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service nova.exception.PciDeviceNotFoundById: PCI device 0000:19:0a.7 not found
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1981813/+subscriptions
References