yahoo-eng-team team mailing list archive

[Alexandre arents <1934742@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Description
===========

It seems that nova may leak network interface in guest
if a port deletion is run in the middle of the a port attachment

in compute manager, attach_interface run atomically
the following tasks:
-update port in neutron(Binding)
-...
-driver.attach_interface()
-update net_info_cache
-...

When a Bound port is deleted, nova receive an event
"network-vif-deleted" and process it by running
def _process_instance_vif_deleted_event()
 ....
 driver.detach_interface()

if this event processing is done just after port binding
and before driver.attach_interface() of an
ongoing interface attachment of the same port,
nova will attach the deleted orphan interface to guest

Probably, the this event processing must be synchronized
with compute manager method attach_interface/detach_interface.


Steps to reproduce
==================

on master devstack:

$openstack server create --flavor m1.small --image cirros-0.5.2-x86_64-disk \
--nic net-id=private myvm
$openstack port create --network private myport

# For ease of reproduction add a pause just before
driver.attach_interface():

nova/compute/manager.py:
def attach_interface()
 try:
   time.sleep(8)
   self.driver.attach_interface(context, ...)

$sudo service devstack@n-cpu restart

$openstack server add port myvm myport &
$sleep 4 ; openstack port delete  myport
[1]+  Exit 1                  openstack server add port myvm myport
Port id 3d47bceb-34ef-4002-8e33-30957127a87f could not be found. (HTTP 404) (Request-ID: req-6c056ad3-1e61-4102-9e5e-48cdd4dffc43)

$ nova interface-list alex
+------------+--------------------------------------+--------------------------------------+-----------------------------------------------+-------------------+-----+
| Port State | Port ID                              | Net ID                               | IP addresses                                  | MAC Addr          | Tag |
+------------+--------------------------------------+--------------------------------------+-----------------------------------------------+-------------------+-----+
| ACTIVE     | 0fe9365b-5747-4532-be50-e6362b10b645 | d8f03257-d1e2-4488-bc42-0e189481a6c7 | 10.0.0.49,fde5:2b4:b028:0:f816:3eff:feb8:f14c | fa:16:3e:b8:f1:4c | -   |
+------------+--------------------------------------+--------------------------------------+-----------------------------------------------+-------------------+-----+

$ virsh domiflist instance-00000001
 Interface        Type     Source   Model    MAC
--------------------
 tap0fe9365b-57   bridge   br-int   virtio   fa:16:3e:b8:f1:4c
 tapdcbbae72-0b   bridge   br-int   virtio   fa:16:3e:95:91:25


Expected result
===============
interface should not be attached to guest

Actual result
=============
zombie interface is attached to guest

** Affects: nova
     Importance: Undecided
     Assignee: Alexandre arents (aarents)
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1934742

Title:
  nova may leak net interface in guest if port under attaching/deleting

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========

  It seems that nova may leak network interface in guest
  if a port deletion is run in the middle of the a port attachment

  in compute manager, attach_interface run atomically
  the following tasks:
  -update port in neutron(Binding)
  -...
  -driver.attach_interface()
  -update net_info_cache
  -...

  When a Bound port is deleted, nova receive an event
  "network-vif-deleted" and process it by running
  def _process_instance_vif_deleted_event()
   ....
   driver.detach_interface()

  if this event processing is done just after port binding
  and before driver.attach_interface() of an
  ongoing interface attachment of the same port,
  nova will attach the deleted orphan interface to guest

  Probably, the this event processing must be synchronized
  with compute manager method attach_interface/detach_interface.

  
  Steps to reproduce
  ==================

  on master devstack:

  $openstack server create --flavor m1.small --image cirros-0.5.2-x86_64-disk \
  --nic net-id=private myvm
  $openstack port create --network private myport

  # For ease of reproduction add a pause just before
  driver.attach_interface():

  nova/compute/manager.py:
  def attach_interface()
   try:
     time.sleep(8)
     self.driver.attach_interface(context, ...)

  $sudo service devstack@n-cpu restart

  $openstack server add port myvm myport &
  $sleep 4 ; openstack port delete  myport
  [1]+  Exit 1                  openstack server add port myvm myport
  Port id 3d47bceb-34ef-4002-8e33-30957127a87f could not be found. (HTTP 404) (Request-ID: req-6c056ad3-1e61-4102-9e5e-48cdd4dffc43)

  $ nova interface-list alex
  +------------+--------------------------------------+--------------------------------------+-----------------------------------------------+-------------------+-----+
  | Port State | Port ID                              | Net ID                               | IP addresses                                  | MAC Addr          | Tag |
  +------------+--------------------------------------+--------------------------------------+-----------------------------------------------+-------------------+-----+
  | ACTIVE     | 0fe9365b-5747-4532-be50-e6362b10b645 | d8f03257-d1e2-4488-bc42-0e189481a6c7 | 10.0.0.49,fde5:2b4:b028:0:f816:3eff:feb8:f14c | fa:16:3e:b8:f1:4c | -   |
  +------------+--------------------------------------+--------------------------------------+-----------------------------------------------+-------------------+-----+

  $ virsh domiflist instance-00000001
   Interface        Type     Source   Model    MAC
  --------------------
   tap0fe9365b-57   bridge   br-int   virtio   fa:16:3e:b8:f1:4c
   tapdcbbae72-0b   bridge   br-int   virtio   fa:16:3e:95:91:25

  
  Expected result
  ===============
  interface should not be attached to guest

  Actual result
  =============
  zombie interface is attached to guest

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1934742/+subscriptions