yahoo-eng-team team mailing list archive

[OpenStack Infra <1607714@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/349014
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a3b3e8d8314b0cedc2604be509f0f4d523a35ed5
Submitter: Jenkins
Branch:    master

commit a3b3e8d8314b0cedc2604be509f0f4d523a35ed5
Author: Matt Riedemann <mriedem@xxxxxxxxxx>
Date:   Thu Feb 9 15:54:41 2017 -0500

    libvirt: wait for interface detach from the guest
    
    The test_reassign_port_between_servers test in Tempest creates
    a port in neutron and two servers. It attaches the port to the
    first server and then quickly detaches it and waits for the
    port.device_id to be unbound from the server. Then it repeats
    that for the second server.
    
    The interface detach from the guest is asynchronous and happens
    before nova unbinds the port, so there is a race where the port's
    device_id is unset but the interface is still on the first guest
    when we try to attach to the second guest, which fails.
    
    This is a latent bug, but apparently has been tickled by the
    move to our neutron CI jobs to use ubuntu xenial nodes.
    
    The fix is to add a detach and retry loop on the interface detach
    on the guest so that we can wait until the interface is gone
    from the guest before nova unbinds the port in neutron, which is
    what the Tempest test is waiting for. Then the device should be
    available for attaching to the second guest.
    
    This is similar to what we do with detaching volumes.
    
    Closes-Bug: #1607714
    
    Change-Id: Ic04aad8923ea2edf1d16e32c208cd41fdf898834


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1607714

Title:
  test_reassign_port_between_servers failing with tap device is busy
  errors in neutron xenial jobs since 7/28

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) newton series:
  Confirmed
Status in OpenStack Compute (nova) ocata series:
  In Progress

Bug description:
  Recent failures showing up in the tempest tests for
  'test_reassign_port_between_servers'.

  From n-cpu.log (http://logs.openstack.org/43/298443/3/check/gate-
  tempest-dsvm-neutron-full-ubuntu-
  xenial/5acdcca/logs/screen-n-cpu.txt.gz#_2016-07-29_07_25_47_948):

  
  2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [req-572c16fd-696d-44ab-a633-49e6625b8f9c tempest-AttachInterfacesTestJSON-1497713114 tempest-AttachInterfacesTestJSON-1497713114] [instance: 73323063-7cc3-4645-9a68-662bf80d9e42] attaching network adapter failed.
  2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42] Traceback (most recent call last):
  2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1389, in attach_interface
  2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42]     guest.attach_device(cfg, persistent=True, live=live)
  2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42]   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 295, in attach_device
  2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42]     self._domain.attachDeviceFlags(device_xml, flags=flags)
  2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
  2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
  2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42]     rv = execute(f, *args, **kwargs)
  2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
  2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42]     six.reraise(c, e, tb)
  2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
  2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42]     rv = meth(*args, **kwargs)
  2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42]   File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 560, in attachDeviceFlags
  2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
  2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42] libvirtError: Unable to create tap device tap9dd515f0-ac: Device or resource busy
  2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1607714/+subscriptions