yahoo-eng-team team mailing list archive

[Arnaud Morin <arnaud.morin@xxxxxxxxx>]

Public bug reported:

Description
===========
When nova start an instance, it asks neutron to create a port and then update the instance info cache based on information from neutron.
If, in the middle of the spawning, the instance is getting deleted, the terminate_instance function is called with an instance object that DOES NOT contain any network info.
As a result, nova is deleting the instance but is never unplugging the interface.

Step to reproduce
=================
I am booting an instance and immediately deleting it thanks to a command like:
$ openstack server create --key-name fake --image ubuntu1810 --flavor c2-7 --net Ext-Net arnaudubuntu1810-3 ; nova delete arnaudubuntu1810-3


- [1] build_and_run_instance is executed, with a semaphore, thus, locking the instance. When booting, nova will fill the network_info cache, by calling [2] update_instance_cache_with_nw_info.
- [3] terminate_instance is executed few seconds later, but is waiting for the semaphore to be released. At this time, the instance network_info cache may not be filled, depending if the [2] update_instance_cache_with_nw_info has already been executed or not.
- If we follow the code, we end up at _shutdown_instance [4], which is doing a call to [5] get_network_info, which is returning a NetworkInfo object that contains no interface.
- At the end, nova is calling _unplug_vifs [6] which is doing nothing (no vif)


Note that I am running OpenStack Newton release, but the code involved
seems identical on master.


[1] https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1837
[2] https://github.com/openstack/nova/blob/master/nova/network/base_api.py#L34
[2] https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2765
[4] https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2559
[5] https://github.com/openstack/nova/blob/master/nova/objects/instance.py#L1252
[6] https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L919

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1830081

Title:
  Nova unplug interface race condition when deleting an instance

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  When nova start an instance, it asks neutron to create a port and then update the instance info cache based on information from neutron.
  If, in the middle of the spawning, the instance is getting deleted, the terminate_instance function is called with an instance object that DOES NOT contain any network info.
  As a result, nova is deleting the instance but is never unplugging the interface.

  Step to reproduce
  =================
  I am booting an instance and immediately deleting it thanks to a command like:
  $ openstack server create --key-name fake --image ubuntu1810 --flavor c2-7 --net Ext-Net arnaudubuntu1810-3 ; nova delete arnaudubuntu1810-3

  
  - [1] build_and_run_instance is executed, with a semaphore, thus, locking the instance. When booting, nova will fill the network_info cache, by calling [2] update_instance_cache_with_nw_info.
  - [3] terminate_instance is executed few seconds later, but is waiting for the semaphore to be released. At this time, the instance network_info cache may not be filled, depending if the [2] update_instance_cache_with_nw_info has already been executed or not.
  - If we follow the code, we end up at _shutdown_instance [4], which is doing a call to [5] get_network_info, which is returning a NetworkInfo object that contains no interface.
  - At the end, nova is calling _unplug_vifs [6] which is doing nothing (no vif)


  Note that I am running OpenStack Newton release, but the code involved
  seems identical on master.


  [1] https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1837
  [2] https://github.com/openstack/nova/blob/master/nova/network/base_api.py#L34
  [2] https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2765
  [4] https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2559
  [5] https://github.com/openstack/nova/blob/master/nova/objects/instance.py#L1252
  [6] https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L919

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1830081/+subscriptions