yahoo-eng-team team mailing list archive

[Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>]

** Changed in: nova
       Status: Fix Committed => Fix Released

** Changed in: nova
    Milestone: None => kilo-1

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1333145

Title:
  quota-usage error in soft-delete

Status in OpenStack Compute (Nova):
  Fix Released

Bug description:
  condition: reclaim_instance_interval > 0 in nova.conf

  I am testing soft-delete , found that quota_usages table will lead to error
  result. when an instance was soft-deleted , before it was deleted completely by period task i soft-delete it again, the quato_usages table will reduce double resource of this instance.

  the reason is that every execution of soft-delete instance,  the
  reservation will commit.

  how to fixed it: we should make reservations=None when
  instance.vm_state='soft-deleted'.

  
  how to reproduct it:

  i am project_id='30528b0d602c4a9c9d8b4cd3d416d710', and I have an
  instance:

  ubuntu@xfolsom:/opt/stack/nova$ nova list
  +--------------------------------------+------+--------+------------+-------------+------------------+
  | ID                                   | Name | Status | Task State | Power State | Networks         |
  +--------------------------------------+------+--------+------------+-------------+------------------+
  | 6f6c1258-6eda-43f1-9531-7a4eb0b44724 | test | ACTIVE | -          | Running     | private=10.0.0.2 |
  +--------------------------------------+------+--------+------------+-------------+------------------+

  1.first select from quota_usage, the result is :

  mysql> select * from quota_usages;
  +---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
  | created_at          | updated_at          | deleted_at | id | project_id                       | resource        | in_use | reserved | until_refresh | deleted | user_id                          |
  +---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
  | 2014-06-20 08:24:35 | 2014-06-23 08:35:03 | NULL       |  1 | 30528b0d602c4a9c9d8b4cd3d416d710 | instances       |      1 |        0 |          NULL |       0 | e522bb6fecaa4a69b6d7df69211dab13 |
  | 2014-06-20 08:24:35 | 2014-06-23 08:35:03 | NULL       |  2 | 30528b0d602c4a9c9d8b4cd3d416d710 | ram             |     64 |        0 |          NULL |       0 | e522bb6fecaa4a69b6d7df69211dab13 |
  | 2014-06-20 08:24:35 | 2014-06-23 08:35:03 | NULL       |  3 | 30528b0d602c4a9c9d8b4cd3d416d710 | cores           |      1 |        0 |          NULL |       0 | e522bb6fecaa4a69b6d7df69211dab13 |
  | 2014-06-20 08:24:35 | 2014-06-20 08:24:35 | NULL       |  4 | 30528b0d602c4a9c9d8b4cd3d416d710 | security_groups |      1 |        0 |          NULL |       0 | e522bb6fecaa4a69b6d7df69211dab13 |
  | 2014-06-20 08:24:36 | 2014-06-23 03:56:03 | NULL       |  5 | 30528b0d602c4a9c9d8b4cd3d416d710 | fixed_ips       |      1 |        0 |          NULL |       0 | NULL                             |
  +---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
  5 rows in set (0.00 sec)

  2.using nova-network, set reclaim_instance_interval=600 in nova.conf.
  3.nova delete 6f6c1258-6eda-43f1-9531-7a4eb0b44724
  4. select from quota_usages, result is :

  mysql> select * from quota_usages;
  +---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
  | created_at          | updated_at          | deleted_at | id | project_id                       | resource        | in_use | reserved | until_refresh | deleted | user_id                          |
  +---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
  | 2014-06-20 08:24:35 | 2014-06-23 08:42:30 | NULL       |  1 | 30528b0d602c4a9c9d8b4cd3d416d710 | instances       |      0 |        0 |          NULL |       0 | e522bb6fecaa4a69b6d7df69211dab13 |
  | 2014-06-20 08:24:35 | 2014-06-23 08:42:30 | NULL       |  2 | 30528b0d602c4a9c9d8b4cd3d416d710 | ram             |      0 |        0 |          NULL |       0 | e522bb6fecaa4a69b6d7df69211dab13 |
  | 2014-06-20 08:24:35 | 2014-06-23 08:42:30 | NULL       |  3 | 30528b0d602c4a9c9d8b4cd3d416d710 | cores           |      0 |        0 |          NULL |       0 | e522bb6fecaa4a69b6d7df69211dab13 |
  | 2014-06-20 08:24:35 | 2014-06-20 08:24:35 | NULL       |  4 | 30528b0d602c4a9c9d8b4cd3d416d710 | security_groups |      1 |        0 |          NULL |       0 | e522bb6fecaa4a69b6d7df69211dab13 |
  | 2014-06-20 08:24:36 | 2014-06-23 03:56:03 | NULL       |  5 | 30528b0d602c4a9c9d8b4cd3d416d710 | fixed_ips       |      1 |        0 |          NULL |       0 | NULL                             |
  +---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
  5 rows in set (0.00 sec)

  5. nova delete 6f6c1258-6eda-43f1-9531-7a4eb0b44724 again

  then select from quota_usages, result is :

  mysql> select * from quota_usages;
  +---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
  | created_at          | updated_at          | deleted_at | id | project_id                       | resource        | in_use | reserved | until_refresh | deleted | user_id                          |
  +---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
  | 2014-06-20 08:24:35 | 2014-06-23 08:43:38 | NULL       |  1 | 30528b0d602c4a9c9d8b4cd3d416d710 | instances       |     -1 |        0 |          NULL |       0 | e522bb6fecaa4a69b6d7df69211dab13 |
  | 2014-06-20 08:24:35 | 2014-06-23 08:43:38 | NULL       |  2 | 30528b0d602c4a9c9d8b4cd3d416d710 | ram             |    -64 |        0 |          NULL |       0 | e522bb6fecaa4a69b6d7df69211dab13 |
  | 2014-06-20 08:24:35 | 2014-06-23 08:43:38 | NULL       |  3 | 30528b0d602c4a9c9d8b4cd3d416d710 | cores           |     -1 |        0 |          NULL |       0 | e522bb6fecaa4a69b6d7df69211dab13 |
  | 2014-06-20 08:24:35 | 2014-06-20 08:24:35 | NULL       |  4 | 30528b0d602c4a9c9d8b4cd3d416d710 | security_groups |      1 |        0 |          NULL |       0 | e522bb6fecaa4a69b6d7df69211dab13 |
  | 2014-06-20 08:24:36 | 2014-06-23 03:56:03 | NULL       |  5 | 30528b0d602c4a9c9d8b4cd3d416d710 | fixed_ips       |      1 |        0 |          NULL |       0 | NULL                             |
  +---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
  5 rows in set (0.00 sec)

  oh my god!!! instance in_use is -1 and ram in_use is -64Mb, do you
  think this is right?

  6.nova restore 6f6c1258-6eda-43f1-9531-7a4eb0b44724
   now the db result is:

  mysql> select * from quota_usages;
  +---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
  | created_at          | updated_at          | deleted_at | id | project_id                       | resource        | in_use | reserved | until_refresh | deleted | user_id                          |
  +---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
  | 2014-06-20 08:24:35 | 2014-06-23 08:44:41 | NULL       |  1 | 30528b0d602c4a9c9d8b4cd3d416d710 | instances       |      2 |        0 |          NULL |       0 | e522bb6fecaa4a69b6d7df69211dab13 |
  | 2014-06-20 08:24:35 | 2014-06-23 08:44:41 | NULL       |  2 | 30528b0d602c4a9c9d8b4cd3d416d710 | ram             |    128 |        0 |          NULL |       0 | e522bb6fecaa4a69b6d7df69211dab13 |
  | 2014-06-20 08:24:35 | 2014-06-23 08:44:41 | NULL       |  3 | 30528b0d602c4a9c9d8b4cd3d416d710 | cores           |      2 |        0 |          NULL |       0 | e522bb6fecaa4a69b6d7df69211dab13 |
  | 2014-06-20 08:24:35 | 2014-06-20 08:24:35 | NULL       |  4 | 30528b0d602c4a9c9d8b4cd3d416d710 | security_groups |      1 |        0 |          NULL |       0 | e522bb6fecaa4a69b6d7df69211dab13 |
  | 2014-06-20 08:24:36 | 2014-06-23 03:56:03 | NULL       |  5 | 30528b0d602c4a9c9d8b4cd3d416d710 | fixed_ips       |      1 |        0 |          NULL |       0 | NULL                             |
  +---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
  5 rows in set (0.00 sec)

  obviously, the result in db is error.now I have only one instance, but
  in db, i have used 2 instances, 2 cores , 128Mb ram and so on.

  The correct way is that we should not allow deleting instance which
  status  in  soft-delete.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1333145/+subscriptions