yahoo-eng-team team mailing list archive

[OpenStack Infra <1678326@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/452351
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=032937ce51c64e0f5292ca3149e3b244863fffca
Submitter: Jenkins
Branch:    master

commit 032937ce51c64e0f5292ca3149e3b244863fffca
Author: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date:   Fri Mar 31 19:56:14 2017 -0400

    Add regression test for quota decrement bug 1678326
    
    This was spotted from someone validating the fix for
    bug 1670627. They reported that even though they failed
    to delete an instance in ERROR state that was in cell0,
    the quota usage was decremented.
    
    This is because we committed the quota reservation
    to decrement the usage before actually attempting to destroy
    the instance, rather than upon successful deletion.
    
    The rollback after InstanceNotFound is a noop because of
    how the Quotas.rollback method noops if the reservations
    were already committed. That is in itself arguably a bug,
    but not fixed here, especially since the counting quotas
    work in Pike will remove all of the reservations commit and
    rollback code.
    
    Change-Id: I12d1fa1a10f9014863123ac9cc3c63ddfb48378e
    Partial-Bug: #1678326


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1678326

Title:
  Quota is decremented during instance delete in cell0 even if the
  instance destroy fails

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) ocata series:
  In Progress

Bug description:
  This is a follow on from bug 1670627. As pointed out in comments 23
  and 25, in the local delete case where there is no host and the
  instance is in a cell, we're decrementing quota even if the
  instance.destroy() operation fails.

  We commit the usage decrement here:

  https://github.com/openstack/nova/blob/88bc8dc5ce32748452c9d3acda9f35e77fedb6ce/nova/compute/api.py#L1878

  Attempt to destroy the instance here:

  https://github.com/openstack/nova/blob/88bc8dc5ce32748452c9d3acda9f35e77fedb6ce/nova/compute/api.py#L1887

  And if the instance.destroy() fails, we rollback the usage decrement
  here:

  https://github.com/openstack/nova/blob/88bc8dc5ce32748452c9d3acda9f35e77fedb6ce/nova/compute/api.py#L1891

  That rollback has no effect because once we commit a reservation, it's
  wiped out in the quotas object:

  https://github.com/openstack/nova/blob/88bc8dc5ce32748452c9d3acda9f35e77fedb6ce/nova/objects/quotas.py#L105

  Attempting to rollback reservations on a quotas object that has
  already committed reservations is a noop:

  https://github.com/openstack/nova/blob/88bc8dc5ce32748452c9d3acda9f35e77fedb6ce/nova/objects/quotas.py#L111

  --

  Unlike the 'normal' (pre-cellsv2) local delete case which does the
  commit after the instance is destroyed:

  https://github.com/openstack/nova/blob/88bc8dc5ce32748452c9d3acda9f35e77fedb6ce/nova/compute/api.py#L2023

  And we rollback (but not commit) if instance.destroy() fails because
  the instance is already deleted:

  https://github.com/openstack/nova/blob/88bc8dc5ce32748452c9d3acda9f35e77fedb6ce/nova/compute/api.py#L2028

  --

  The code in _delete() is wrong because it was copied from the code in
  _delete_while_booting() which is also wrong:

  https://github.com/openstack/nova/blob/88bc8dc5ce32748452c9d3acda9f35e77fedb6ce/nova/compute/api.py#L1784

  So we have to fix both places.

  I suggest that we also change the noop behavior on the Quotas object
  such that if we attempt a reservation commit or rollback operation on
  a Quotas object that does not have any reservations, it is considered
  a programming error rather than a noop, because that's how this bug
  was introduced in the first place.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1678326/+subscriptions