yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem@xxxxxxxxxx>]

** Also affects: nova/liberty
   Importance: Undecided
       Status: New

** Changed in: nova/liberty
     Assignee: (unassigned) => Mark Goddard (mgoddard)

** Changed in: nova/liberty
       Status: New => In Progress

** Changed in: nova/liberty
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1501735

Title:
  Network interface allocation corrupts instance info cache

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) liberty series:
  In Progress

Bug description:
  Allocation of network interfaces for an instance can result in
  corruption of the instance info cache in Nova. The result is that the
  cache may contain duplicate entries for network interfaces. This can
  cause failure to boot nodes, as seen with the Libvirt driver.

  Seen on Ubuntu / devstack / commit
  b0013d93ffeaed53bc28d9558def26bdb7041ed7.

  The issue can be reproduced using an instance with a large number of
  interfaces, for example using the heat stack in the attached YAML file
  heat-stack-many-interfaces.yaml. For improved reproducibility, add a
  short sleep in nova.network.neutronv2.api.API.allocate_for_instance,
  just before the call to self.get_instance_nw_info.

  This issue was found by SecurityFun23 when testing the fix for bug
  #1467581.

  The problem appears to be that in
  nova.network.neutronv2.api.API.allocate_for_instance, after the
  Neutron API calls to create/update ports, but before the instance info
  cache is  updated in get_instance_nw_info, it is possible for another
  request to refresh the instance info cache. This will cause the
  new/updated ports to be added to the cache as they are discovered in
  Neutron. Then, the original request resumes, and unconditionally adds
  the new interfaces to the cache. This results in duplicate entries.
  The most likely candidate for another request is probably Neutron
  network-change notifications, which are triggered by the port
  update/create operation. The allocation of multiple interfaces is more
  likely to make the problem to occur, as Neutron API requests are made
  serially for each of the ports, allowing time for the notifications to
  arrive.

  The perceived problem in a more visual form:

  Request:
  - Allocate interfaces for an instance (nova.network.neutronv2.api.API.allocate_for_instance)
  - n x Neutron API port create/updates
  ------------------------------
  Notification:
  - External event notification from Neutron - network-changed (nova.compute.manager.ComputeManager.external_instance_event)
  - Refresh instance network cache (network_api.get_instance_nw_info)
  - Query ports for device in Neutron
  - Add new ports to instance info cache
  -------------------------------
  Request:
  - Refresh instance network cache with new interfaces (get_instance_nw_info)
  - Unconditionally add duplicate interfaces to cache.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1501735/+subscriptions