yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #25409
[Bug 1400944] [NEW] Nova per-project quota usage can become inaccurate.
Public bug reported:
I am still working on diagnosing the exact steps to reproduce, but on
test systems it is possible to get nova's quota tracking into an
inaccurate state.
This was triggered via randomized testing with a focus on create server
images as well as rebooting and resizing the test servers.
In the example below, after running some randomized tests, there are no
active servers on the project, yet 'nova absolute-limits' output lists
cores as being used (the quotas-usage table data reflects this).
No cores are listed as used in the admin view for hypervisor usage, but
this inaccurate state will prevent the project from spinning up
machines.
nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+
pcrews@erlking-dev:~/git/rannsaka$ nova absolute-limits
+-------------------------+-------+
| Name | Value |
+-------------------------+-------+
| maxServerMeta | 128 |
| maxPersonality | 5 |
| totalServerGroupsUsed | 0 |
| maxImageMeta | 128 |
| maxPersonalitySize | 10240 |
| maxTotalRAMSize | 51200 |
| maxTotalKeypairs | 100 |
| maxSecurityGroupRules | 20 |
| maxServerGroups | 10 |
| totalCoresUsed | 8 |
| totalRAMUsed | 26368 |
| maxSecurityGroups | 10 |
| totalFloatingIpsUsed | 0 |
| totalInstancesUsed | 0 |
| totalSecurityGroupsUsed | 1 |
| maxTotalFloatingIps | 10 |
| maxTotalInstances | 20 |
| maxTotalCores | 20 |
| maxServerGroupMembers | 10 |
+-------------------------+-------+
mysql> select * from quota_usages;
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
| created_at | updated_at | deleted_at | id | project_id | resource | in_use | reserved | until_refresh | deleted | user_id |
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
| 2014-12-09 23:28:12 | 2014-12-10 00:40:06 | NULL | 1 | 078e0e1371f44e2e9e6d9691342ed02d | instances | 0 | 0 | NULL | 0 | d2197b1accca4a51b2dbb964d9fc7683 |
| 2014-12-09 23:28:12 | 2014-12-10 00:40:06 | NULL | 2 | 078e0e1371f44e2e9e6d9691342ed02d | ram | 26368 | 3968 | NULL | 0 | d2197b1accca4a51b2dbb964d9fc7683 |
| 2014-12-09 23:28:12 | 2014-12-10 00:40:06 | NULL | 3 | 078e0e1371f44e2e9e6d9691342ed02d | cores | 8 | 2 | NULL | 0 | d2197b1accca4a51b2dbb964d9fc7683 |
| 2014-12-09 23:28:12 | 2014-12-09 23:28:12 | NULL | 4 | 078e0e1371f44e2e9e6d9691342ed02d | security_groups | 1 | 0 | NULL | 0 | d2197b1accca4a51b2dbb964d9fc7683 |
| 2014-12-09 23:28:14 | 2014-12-10 00:40:05 | NULL | 5 | 078e0e1371f44e2e9e6d9691342ed02d | fixed_ips | 0 | 0 | NULL | 0 | NULL |
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
** Affects: nova
Importance: Undecided
Status: New
** Description changed:
I am still working on diagnosing the exact steps to reproduce, but on
test systems it is possible to get nova's quota tracking into an
inaccurate state.
This was triggered via randomized testing with a focus on create server
- images, rebooting, and resizing them.
+ images as well as rebooting and resizing the test servers.
In the example below, after running some randomized tests, there are no
active servers on the project, yet 'nova absolute-limits' output lists
cores as being used (the quotas-usage table data reflects this).
No cores are listed as used in the admin view for hypervisor usage, but
this inaccurate state will prevent the project from spinning up
machines.
nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+
pcrews@erlking-dev:~/git/rannsaka$ nova absolute-limits
+-------------------------+-------+
| Name | Value |
+-------------------------+-------+
| maxServerMeta | 128 |
| maxPersonality | 5 |
| totalServerGroupsUsed | 0 |
| maxImageMeta | 128 |
| maxPersonalitySize | 10240 |
| maxTotalRAMSize | 51200 |
| maxTotalKeypairs | 100 |
| maxSecurityGroupRules | 20 |
| maxServerGroups | 10 |
| totalCoresUsed | 8 |
| totalRAMUsed | 26368 |
| maxSecurityGroups | 10 |
| totalFloatingIpsUsed | 0 |
| totalInstancesUsed | 0 |
| totalSecurityGroupsUsed | 1 |
| maxTotalFloatingIps | 10 |
| maxTotalInstances | 20 |
| maxTotalCores | 20 |
| maxServerGroupMembers | 10 |
+-------------------------+-------+
mysql> select * from quota_usages;
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
| created_at | updated_at | deleted_at | id | project_id | resource | in_use | reserved | until_refresh | deleted | user_id |
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
| 2014-12-09 23:28:12 | 2014-12-10 00:40:06 | NULL | 1 | 078e0e1371f44e2e9e6d9691342ed02d | instances | 0 | 0 | NULL | 0 | d2197b1accca4a51b2dbb964d9fc7683 |
| 2014-12-09 23:28:12 | 2014-12-10 00:40:06 | NULL | 2 | 078e0e1371f44e2e9e6d9691342ed02d | ram | 26368 | 3968 | NULL | 0 | d2197b1accca4a51b2dbb964d9fc7683 |
| 2014-12-09 23:28:12 | 2014-12-10 00:40:06 | NULL | 3 | 078e0e1371f44e2e9e6d9691342ed02d | cores | 8 | 2 | NULL | 0 | d2197b1accca4a51b2dbb964d9fc7683 |
| 2014-12-09 23:28:12 | 2014-12-09 23:28:12 | NULL | 4 | 078e0e1371f44e2e9e6d9691342ed02d | security_groups | 1 | 0 | NULL | 0 | d2197b1accca4a51b2dbb964d9fc7683 |
| 2014-12-09 23:28:14 | 2014-12-10 00:40:05 | NULL | 5 | 078e0e1371f44e2e9e6d9691342ed02d | fixed_ips | 0 | 0 | NULL | 0 | NULL |
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1400944
Title:
Nova per-project quota usage can become inaccurate.
Status in OpenStack Compute (Nova):
New
Bug description:
I am still working on diagnosing the exact steps to reproduce, but on
test systems it is possible to get nova's quota tracking into an
inaccurate state.
This was triggered via randomized testing with a focus on create
server images as well as rebooting and resizing the test servers.
In the example below, after running some randomized tests, there are
no active servers on the project, yet 'nova absolute-limits' output
lists cores as being used (the quotas-usage table data reflects this).
No cores are listed as used in the admin view for hypervisor usage,
but this inaccurate state will prevent the project from spinning up
machines.
nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+
pcrews@erlking-dev:~/git/rannsaka$ nova absolute-limits
+-------------------------+-------+
| Name | Value |
+-------------------------+-------+
| maxServerMeta | 128 |
| maxPersonality | 5 |
| totalServerGroupsUsed | 0 |
| maxImageMeta | 128 |
| maxPersonalitySize | 10240 |
| maxTotalRAMSize | 51200 |
| maxTotalKeypairs | 100 |
| maxSecurityGroupRules | 20 |
| maxServerGroups | 10 |
| totalCoresUsed | 8 |
| totalRAMUsed | 26368 |
| maxSecurityGroups | 10 |
| totalFloatingIpsUsed | 0 |
| totalInstancesUsed | 0 |
| totalSecurityGroupsUsed | 1 |
| maxTotalFloatingIps | 10 |
| maxTotalInstances | 20 |
| maxTotalCores | 20 |
| maxServerGroupMembers | 10 |
+-------------------------+-------+
mysql> select * from quota_usages;
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
| created_at | updated_at | deleted_at | id | project_id | resource | in_use | reserved | until_refresh | deleted | user_id |
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
| 2014-12-09 23:28:12 | 2014-12-10 00:40:06 | NULL | 1 | 078e0e1371f44e2e9e6d9691342ed02d | instances | 0 | 0 | NULL | 0 | d2197b1accca4a51b2dbb964d9fc7683 |
| 2014-12-09 23:28:12 | 2014-12-10 00:40:06 | NULL | 2 | 078e0e1371f44e2e9e6d9691342ed02d | ram | 26368 | 3968 | NULL | 0 | d2197b1accca4a51b2dbb964d9fc7683 |
| 2014-12-09 23:28:12 | 2014-12-10 00:40:06 | NULL | 3 | 078e0e1371f44e2e9e6d9691342ed02d | cores | 8 | 2 | NULL | 0 | d2197b1accca4a51b2dbb964d9fc7683 |
| 2014-12-09 23:28:12 | 2014-12-09 23:28:12 | NULL | 4 | 078e0e1371f44e2e9e6d9691342ed02d | security_groups | 1 | 0 | NULL | 0 | d2197b1accca4a51b2dbb964d9fc7683 |
| 2014-12-09 23:28:14 | 2014-12-10 00:40:05 | NULL | 5 | 078e0e1371f44e2e9e6d9691342ed02d | fixed_ips | 0 | 0 | NULL | 0 | NULL |
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1400944/+subscriptions
Follow ups
References
