yahoo-eng-team team mailing list archive

[OpenStack Infra <1510979@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/243477
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=08d24b733ee9f4da44bfbb8d6d3914924a41ccdc
Submitter: Jenkins
Branch:    master

commit 08d24b733ee9f4da44bfbb8d6d3914924a41ccdc
Author: Wen Zhi Yu <yuywz@xxxxxxxxxx>
Date:   Tue Nov 10 17:16:36 2015 +0800

    Clean up network resources when reschedule fails
    
    During the instance boot (spawn/run) process, neutron ports are
    allocated for the instance if necessary. If the instance fails
    to spawn (say as a result of a compute host failure), the default
    behaviour is to reschedule the instance and leave its networking
    resources in-tact for potential reuse on the rescheduled host.
    All is good if the instance is successfully rescheduled, but if
    the reschedule fails (say no more applicable hosts) the allocated
    ports are left as-is and effectively orphaned.
    
    This commit add code to clean up allocated network resources
    when the reschedule fails.
    
    Change-Id: Ic670dd4dc192603c2faecf18e14ef59ebca9e420
    Closes-Bug: #1510979


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1510979

Title:
  Instance reschedule failure leaves orphaned neutron ports

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  During the instance boot (spawn/run) process, neutron ports are
  allocated for the instance if necessary. If the instance fails to
  spawn (say as a result of a compute host failure), the default
  behavior is to reschedule the instance and leave it's networking
  resources in-tact for potential reuse on the rescheduled host (as per
  deallocate_networks_on_reschedule() [1] which returns False for most
  compute drivers).

  All is good if the instance is successfully rescheduled, but if the
  reschedule fails (say no more applicable hosts) the allocated ports
  are left as-is and effectively orphaned.

  There are some related defects ([2] and [3]), but they don't quite
  touch on the particular behavior described herein.

  Obviously there are a number of ways to address this issue, but the
  most obvious is perhaps nova should be aware of the reschedule failure
  and deallocate any resources which may have been left in-tact for the
  reschedule.

  I'm running devstack all-in-one setup from openstack master branches.

  nova --version
  2.32.0
  neutron --version
  3.1.0

  The easiest way to repo is to use an all-in-one devstack (only 1
  compute host) simulate a host spawn failure by editing the spwan()
  method of your compute driver to raise an exception at the end of the
  method and simply try to boot a server. In this setup there's only 1
  host so the reschedule will fail and you can verify the port allocated
  for the instance still exists after trying to boot the instance.

  
  [1] https://github.com/openstack/nova/blob/master/nova/virt/driver.py#L1273
  [2] https://bugs.launchpad.net/nova/+bug/1410739
  [3] https://bugs.launchpad.net/nova/+bug/1327124

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1510979/+subscriptions