← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #89300

[Bug 1981813] [NEW] Compute service fails to restart if the vnic_type of a bound port changed from direct to macvtap

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

We have a downstream bug report with the following reproduction steps:

1) create a neutron port with vnic_type "direct"
2) create an instance with that port
3) after the instance is created successfully change the vnic_type of the bound port from "direct" to "macvtap". This is accepted by Neutron
4) wait until the nova instance info caches is healed by the periodic task in nova-compute
5) restart the nova-compute service.

Actual behavior
---------------
The nova-compute service fails to start with PciDeviceNotFoundById exception pointing to the PCI address of the VF the port is bound to on the host.

Expected behavior
-----------------
The nova-compute service should start up successfully.

Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service Traceback (most recent call last):
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/pci/utils.py", line 167, in get_ifname_by_pci_address
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     dev_info = os.listdir(dev_path)
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service FileNotFoundError: [Errno 2] No such file or directory: '/sys/bus/pci/devices/0000:19:0a.7/net'
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service 
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service During handling of the above exception, another exception occurred:
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service 
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service Traceback (most recent call last):
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/usr/local/lib/python3.10/site-packages/oslo_service/service.py", line 806, in run_service
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     service.start()
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/service.py", line 159, in start
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     self.manager.init_host()
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/compute/manager.py", line 1536, in init_host
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     self._init_instance(context, instance)
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/compute/manager.py", line 1230, in _init_instance
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     self.driver.plug_vifs(instance, net_info)
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1386, in plug_vifs
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     self.vif_driver.plug(instance, vif)
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 730, in plug
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     self.plug_hw_veb(instance, vif)
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 628, in plug_hw_veb
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     set_vf_interface_vlan(
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 99, in set_vf_interface_vlan
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     vf_ifname = pci_utils.get_ifname_by_pci_address(pci_addr)
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/pci/utils.py", line 170, in get_ifname_by_pci_address
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     raise exception.PciDeviceNotFoundById(id=pci_addr)
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service nova.exception.PciDeviceNotFoundById: PCI device 0000:19:0a.7 not found
Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service

** Affects: nova
     Importance: Undecided
     Assignee: Balazs Gibizer (balazs-gibizer)
         Status: New


** Tags: compute neutron pci

** Changed in: nova
     Assignee: (unassigned) => Balazs Gibizer (balazs-gibizer)

** Tags added: neutron pci

** Tags added: compute

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1981813

Title:
  Compute service fails to restart if the vnic_type of a bound port
  changed from direct to macvtap

Status in OpenStack Compute (nova):
  New

Bug description:
  We have a downstream bug report with the following reproduction steps:

  1) create a neutron port with vnic_type "direct"
  2) create an instance with that port
  3) after the instance is created successfully change the vnic_type of the bound port from "direct" to "macvtap". This is accepted by Neutron
  4) wait until the nova instance info caches is healed by the periodic task in nova-compute
  5) restart the nova-compute service.

  Actual behavior
  ---------------
  The nova-compute service fails to start with PciDeviceNotFoundById exception pointing to the PCI address of the VF the port is bound to on the host.

  Expected behavior
  -----------------
  The nova-compute service should start up successfully.

  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service Traceback (most recent call last):
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/pci/utils.py", line 167, in get_ifname_by_pci_address
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     dev_info = os.listdir(dev_path)
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service FileNotFoundError: [Errno 2] No such file or directory: '/sys/bus/pci/devices/0000:19:0a.7/net'
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service 
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service During handling of the above exception, another exception occurred:
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service 
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service Traceback (most recent call last):
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/usr/local/lib/python3.10/site-packages/oslo_service/service.py", line 806, in run_service
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     service.start()
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/service.py", line 159, in start
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     self.manager.init_host()
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/compute/manager.py", line 1536, in init_host
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     self._init_instance(context, instance)
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/compute/manager.py", line 1230, in _init_instance
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     self.driver.plug_vifs(instance, net_info)
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1386, in plug_vifs
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     self.vif_driver.plug(instance, vif)
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 730, in plug
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     self.plug_hw_veb(instance, vif)
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 628, in plug_hw_veb
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     set_vf_interface_vlan(
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 99, in set_vf_interface_vlan
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     vf_ifname = pci_utils.get_ifname_by_pci_address(pci_addr)
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service   File "/opt/stack/nova/nova/pci/utils.py", line 170, in get_ifname_by_pci_address
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service     raise exception.PciDeviceNotFoundById(id=pci_addr)
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service nova.exception.PciDeviceNotFoundById: PCI device 0000:19:0a.7 not found
  Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1981813/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next