yahoo-eng-team team mailing list archive

[OpenStack Infra <1620736@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/366245
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e84396b5232cb52228bc75dee92429913d72468e
Submitter: Jenkins
Branch:    master

commit e84396b5232cb52228bc75dee92429913d72468e
Author: Sean Dague <sean@xxxxxxxxx>
Date:   Tue Sep 6 12:33:46 2016 -0400

    correctly join the usage to inventory for capacity accounting
    
    This updates the SQL code to match the comment about how the tables
    should be joined. It also adds a detailed warning when we go over
    capacity, that will be useful to developers and operators when fail to
    land on an environment.
    
    The bug was deep in the SQL, and can only be tested by testing the
    SQL, which we may want to do later. In leiu of a unit test, an
    additional safety check is put in the code that if we ever end up with
    multiple records with the same key, we explode with a KeyError.
    
    Closes-Bug: #1620736
    
    Change-Id: Ic3f83a881e7d98fc6ed007b62a179b3153190543


** Changed in: nova
       Status: Confirmed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1620736

Title:
  Incorrect SQL in placement API causes spurious
  InvalidAllocationCapacityExceeded error

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Upstream master

  The SQL that joins the allocations to the inventory for
  _check_capacity_exceeded was incorrect. It was doing a left outer join
  with the allocations which meant we got resource accounting in an NxN
  matrix with all inventory.

  
   1
   2
   3
   4
   5
   6
   7
   8
   9
  10
  2016-09-06 12:12:10.978 DEBUG nova.objects.resource_provider [req-299c61e7-fc99-4cdc-b633-dc20d5886367 placement service] Allocation Record: (1, u'62b97cd4-5dc8-45d9-89ad-988273895635', 324, 0, 4, 0, 16.0, Decimal('0')) from (pid=32242) _check_capacity_exceeded /opt/stack/nova/nova/objects/resource_provider.py:706
  2016-09-06 12:12:10.979 DEBUG nova.objects.resource_provider [req-299c61e7-fc99-4cdc-b633-dc20d5886367 placement service] Allocation Record: (1, u'62b97cd4-5dc8-45d9-89ad-988273895635', 324, 0, 4, 0, 16.0, Decimal('11')) from (pid=32242) _check_capacity_exceeded /opt/stack/nova/nova/objects/resource_provider.py:706
  2016-09-06 12:12:10.979 DEBUG nova.objects.resource_provider [req-299c61e7-fc99-4cdc-b633-dc20d5886367 placement service] Allocation Record: (1, u'62b97cd4-5dc8-45d9-89ad-988273895635', 324, 0, 4, 0, 16.0, Decimal('704')) from (pid=32242) _check_capacity_exceeded /opt/stack/nova/nova/objects/resource_provider.py:706
  2016-09-06 12:12:10.980 DEBUG nova.objects.resource_provider [req-299c61e7-fc99-4cdc-b633-dc20d5886367 placement service] Allocation Record: (1, u'62b97cd4-5dc8-45d9-89ad-988273895635', 324, 1, 15947, 512, 1.5, Decimal('0')) from (pid=32242) _check_capacity_exceeded /opt/stack/nova/nova/objects/resource_provider.py:706
  2016-09-06 12:12:10.980 DEBUG nova.objects.resource_provider [req-299c61e7-fc99-4cdc-b633-dc20d5886367 placement service] Allocation Record: (1, u'62b97cd4-5dc8-45d9-89ad-988273895635', 324, 1, 15947, 512, 1.5, Decimal('11')) from (pid=32242) _check_capacity_exceeded /opt/stack/nova/nova/objects/resource_provider.py:706
  2016-09-06 12:12:10.981 DEBUG nova.objects.resource_provider [req-299c61e7-fc99-4cdc-b633-dc20d5886367 placement service] Allocation Record: (1, u'62b97cd4-5dc8-45d9-89ad-988273895635', 324, 1, 15947, 512, 1.5, Decimal('704')) from (pid=32242) _check_capacity_exceeded /opt/stack/nova/nova/objects/resource_provider.py:706
  2016-09-06 12:12:10.981 DEBUG nova.objects.resource_provider [req-299c61e7-fc99-4cdc-b633-dc20d5886367 placement service] Allocation Record: (1, u'62b97cd4-5dc8-45d9-89ad-988273895635', 324, 2, 218, 0, 1.0, Decimal('0')) from (pid=32242) _check_capacity_exceeded /opt/stack/nova/nova/objects/resource_provider.py:706
  2016-09-06 12:12:10.982 DEBUG nova.objects.resource_provider [req-299c61e7-fc99-4cdc-b633-dc20d5886367 placement service] Allocation Record: (1, u'62b97cd4-5dc8-45d9-89ad-988273895635', 324, 2, 218, 0, 1.0, Decimal('11')) from (pid=32242) _check_capacity_exceeded /opt/stack/nova/nova/objects/resource_provider.py:706
  2016-09-06 12:12:10.983 DEBUG nova.objects.resource_provider [req-299c61e7-fc99-4cdc-b633-dc20d5886367 placement service] Allocation Record: (1, u'62b97cd4-5dc8-45d9-89ad-988273895635', 324, 2, 218, 0, 1.0, Decimal('704')) from (pid=32242) _check_capacity_exceeded /opt/stack/nova/nova/objects/resource_provider.py:706
  2016-09-06 12:12:10.983 WARNING nova.objects.resource_provider [req-299c61e7-fc99-4cdc-b633-dc20d5886367 placement service] Attempting to allocate 1 for VCPU. Currently using 704, amount available 64.0

  The Decimal allocation for memory ('704') is reported here against CPU
  and Disk resources in addition to Memory. Depending on the order rows
  are returned, these get squashed in a dict later, and we end up with
  the wrong usage record for CPU.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1620736/+subscriptions