yahoo-eng-team team mailing list archive

[Sivasathurappan Radhakrishnan <siva.radhakrishnan@xxxxxxxxx>]

This bug lacks the necessary information to effectively reproduce and
fix it, therefore it has been closed. Feel free to reopen the bug by
providing the requested information and set the bug status back to
''New'

** Changed in: nova
       Status: Incomplete => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1560472

Title:
  nova interface-attach command removes pre-existing neutron ports from
  the environment if it fails to attach to an instance _even_ where
  '--port-id' has been specified

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Problem description:
  The nova interface-attach command removes pre-existing neutron ports from the environment if it fails to attach to an instance _even_ where '--port-id' has been specified. This behaviour was introduced by fixing bug #1338551 [1].

  Steps to reproduce:
  1) create a new neutron port
    $ neutron port-create --name <port-name> <network-name>
  2) boot an instance (make sure to specify a keypair and check sec groups for ssh connectivity to the instance)
    $ nova boot ...
  3) [OPTIONAL] add/remove the port several times over to prove the funtionality is working OK.
    $ nova interface-attach --port-id <port-id> <instance-name>
    $ nova interface-detach <instance-name> <port-id>
  4) simulate a kernel crash on the instance, as this should cause a scenario where an interface attach will fail (ssh connectivity is assumed for this step)
    $ ssh <instance-ip> "sudo kill -11 1" # OR execute 'echo c > /proc/sysrq-trigger' while connected to the instance
    4a. Verify the kernel has actually crashed
        $ nova console-log <instance-name>
  5) try to attach the port while the instance is still crashed # **note** if the port hasn't been attached before (i.e. you skipped step 3, it may succeed initially then it will fail on subsequent attach attempts). Also, at this point it should not matter if the port is still attached to the instance.
    $ nova interface-attach --port-id <port-id> <instance-name>

  Errors observed:
  $ nova interface-attach --port-id <port-id> <instance-name>
  ERROR: Failed to attach interface (HTTP 500) (Request-ID: req-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)

  Expected results:
  The port should still exist after failure in this scenario.

  Actual results:
  'neutron port-list' will no longer show the port. It has been removed.
  The port is removed from the environment and therefore is no longer available.

  Snippet from /var/log/nova/nova-compute.log

  [instance: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] attaching network adapter failed.
   Traceback (most recent call last):
     File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1263, in attach_interface
       virt_dom.attachDeviceFlags(cfg.to_xml(), flags)
     File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
       result = proxy_call(self._autowrap, f, *args, **kwargs)
     File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
       rv = execute(f, *args, **kwargs)
     File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
       six.reraise(c, e, tb)
     File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
       rv = meth(*args, **kwargs)
     File "/usr/lib/python2.7/dist-packages/libvirt.py", line 513, in attachDeviceFlags
       if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
   libvirtError: Unable to create tap device tapxxxxxxxx-xx: Device or resource busy
   attach interface failed , try to deallocate port xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, reason: Failed to attach network adapter device to xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
   Exception during message handling: Failed to attach network adapter device to xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

  Full error:
  http://pastebin.ubuntu.com/15471511/

  $ sudo apt-cache policy nova-compute
  nova-compute:
    Installed: 1:2015.1.2-0ubuntu2~cloud0

  Ubuntu 14.04.4 LTS

  Why does this matter:
  As specified in [1], where a port has been attached using --net-id option it is automatically created before attaching to the VM. Therefore, it is the correct behaviour to cleanup after a failure to attach.
  Where "--port-id" has been specified, it should not be assumed that it was auto created, it has been specifically created and therefore may have pre-existed the VM, this means the port should be re-usable if desired and therefore should not be cleaned up in the case of attach failure. When the port has been pre-created and '--port-id' is specified in the interface-attach command, if the action fails to attach it should be handled without being removed from the environment and exist for re-assignment to another instance or for retry to the original instance once it has recovered from it's failure.

  This behaviour is confirmed on both Kilo and Liberty.

  Related bugs:
  [1] https://bugs.launchpad.net/nova/+bug/1338551

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1560472/+subscriptions