yahoo-eng-team team mailing list archive

[Brian Haley <1977485@xxxxxxxxxxxxxxxxxx>]

Thanks for the info, I'll re-assign to Nova component based on that.

** Project changed: neutron => nova

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1977485

Title:
  Neutron deletes port in use and nova errors out when cleaning the VM
  XML

Status in OpenStack Compute (nova):
  Incomplete

Bug description:
  We recently upgraded to OpenStack XENA on a Kolla deployment and we
  hit this issue

  Neutron is able to delete a port in use and Nova errors out when
  trying to clean up the VM XML (we are talking about Windows VMs on KVM
  hypervisor).

  We need to move the ports from some VMs to a different subnet. We tried the following procedure:
  remove the port of a VM
  create a new port on the new subnet
  attach the new port to the VM

  Result:
  We have a VM with no network connectivity.
  The port gets deleted from OpenStack, and from OVS as well, but in the VM XML we see this:
  <interface type='bridge'>
        <mac address='00:16:3c:7b:2c:1c'/>
        <source bridge='br-int'/>
        <virtualport type='openvswitch'>
          <parameters interfaceid='fb15ad83-bf28-455d-a1b1-14158203b4bf'/>
        </virtualport>
        <target dev='tapfb15ad83-bf'/>
        <model type='virtio'/>
        <mtu size='1500'/>
        <alias name='net0'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
      </interface>
      <interface type='ethernet'>
        <mac address='00:16:3c:7b:2c:1c'/>
        <target dev='tapba91ea48-36'/>
        <model type='virtio'/>
        <mtu size='1500'/>
        <alias name='net1'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
      </interface>

  The bridge part represents the old interface (the interface of the old deleted port) and the ethernet part is the new port.
  We also tried to use:
  virsh detach-interface in order to remove the stale interface from the XML, the command said it was completed successfully but the interface is still there.
  We noticed that rebooting the VM cleans the XML file and the connectivity is back (this is not the desired solution)

  In the logs we see:
  When we delete the old port:
  2022-05-31 12:13:29.935 7 INFO nova.compute.manager [req-1bf9e960-fc99-453f-8bf9-cd6a76c12feb 9879764509c84ca58d054fc3b9575df6 24783cb241264363ad1b8808ba21c131 - default default] [instance: b6c60a66-e571-4d50-984a-101dcb29f6aa] Neutron deleted interface fb15ad83-bf28-455d-a1b1-14158203b4bf; detaching it from the instance and deleting it from the info cache
  2022-05-31 12:13:30.076 7 WARNING nova.virt.libvirt.driver [req-1bf9e960-fc99-453f-8bf9-cd6a76c12feb 9879764509c84ca58d054fc3b9575df6 24783cb241264363ad1b8808ba21c131 - default default] [instance: b6c60a66-e571-4d50-984a-101dcb29f6aa] Detaching interface 00:16:3c:7b:2c:1c failed because the device is no longer found on the guest.: nova.exception.DeviceNotFound: Device 'tapfb15ad83-bf' not found.
  2022-05-31 12:13:30.740 7 INFO os_vif [req-1bf9e960-fc99-453f-8bf9-cd6a76c12feb 9879764509c84ca58d054fc3b9575df6 24783cb241264363ad1b8808ba21c131 - default default] Successfully unplugged vif VIFOpenVSwitch(active=True,address=00:16:3c:7b:2c:1c,bridge_name='br-int',has_traffic_filtering=True,id=fb15ad83-bf28-455d-a1b1-14158203b4bf,network=Network(b03631f6-6fa7-4ff3-97e6-0a3bd077fac3),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=True,vif_name='tapfb15ad83-bf')

  When we attach the new port:
  2022-05-31 12:20:16.427 7 WARNING nova.compute.manager [req-f42820d6-1c70-428a-9c2d-305737838bfc 9879764509c84ca58d054fc3b9575df6 24783cb241264363ad1b8808ba21c131 - default default] [instance: b6c60a66-e571-4d50-984a-101dcb29f6aa] Received unexpected event network-vif-plugged-ba91ea48-3676-4934-87ba-1ad4cf80b1bc for instance with vm_state active and task_state None.
  2022-05-31 12:20:19.188 7 WARNING nova.compute.manager [req-305324c7-c25b-44a7-96cd-a8cc84284727 9879764509c84ca58d054fc3b9575df6 24783cb241264363ad1b8808ba21c131 - default default] [instance: b6c60a66-e571-4d50-984a-101dcb29f6aa] Received unexpected event network-vif-plugged-ba91ea48-3676-4934-87ba-1ad4cf80b1bc for instance with vm_state active and task_state None.

  
  We found that the following workaround works:
  we use virsh detach-interface while the old port of the VM exists (before we delete it)
  then we delete the old port 
  after that, we attach the new port
  This works as expected and the VM has network connectivity.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1977485/+subscriptions