yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1490855] [NEW] resource_tracker keeps complaining instance not found for deleted instances

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Zhenzan Zhou <zhenzan.zhou@xxxxxxxxx>
Date: Tue, 01 Sep 2015 07:29:10 -0000
Reply-to: Bug 1490855 <1490855@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Some instances met error during resize or live-migration,  then
resource_tracker keeps complaining instance not found even after they
got deleted.

===== n-cpu.log =====
647686:2015-09-01 07:26:27.172 DEBUG nova.compute.resource_tracker [req-36bf129b-5ed3-403d-9d88-143b0cc3810e None None] Migration instance not found: Instance 0a130eb8-01b5-4ad7-bffd-64abe360e459 could not be found.
647712:2015-09-01 07:26:27.224 DEBUG nova.compute.resource_tracker [req-36bf129b-5ed3-403d-9d88-143b0cc3810e None None] Migration instance not found: Instance 57ec3d85-fd77-4b9a-b4f4-8910e892b612 could not be found.
647738:2015-09-01 07:26:27.277 DEBUG nova.compute.resource_tracker [req-36bf129b-5ed3-403d-9d88-143b0cc3810e None None] Migration instance not found: Instance 7a54fd3b-fee4-4b40-9074-adb515f7fc50 could not be found.


====== db ========
ubuntu@node-01:/opt/stack/nova$ nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+

mysql> select created_at,deleted_at,uuid  from instances;
+---------------------+---------------------+--------------------------------------+
| created_at          | deleted_at          | uuid                                 |
+---------------------+---------------------+--------------------------------------+
| 2015-08-28 06:03:36 | 2015-08-31 06:51:18 | 57ec3d85-fd77-4b9a-b4f4-8910e892b612 |
| 2015-08-28 06:04:30 | 2015-08-31 06:51:12 | 0a130eb8-01b5-4ad7-bffd-64abe360e459 |
| 2015-08-28 06:10:56 | 2015-08-28 06:11:14 | 8217c85d-6e74-48b6-940d-7eeb6f7afddd |
| 2015-08-28 06:12:26 | 2015-08-28 06:12:36 | 347218bb-166c-4228-8737-2b7baf310f21 |
| 2015-08-28 06:12:54 | 2015-08-28 06:13:09 | 8815d09a-3b07-4156-a451-d6182ace8b61 |
| 2015-08-28 06:13:27 | 2015-08-28 06:13:39 | 50c40182-ee2f-4553-9a40-828a45ba7026 |
| 2015-08-31 06:54:58 | 2015-09-01 07:10:50 | 7a54fd3b-fee4-4b40-9074-adb515f7fc50 |
| 2015-08-31 07:45:16 | 2015-09-01 07:10:40 | 0b141529-2127-4559-bf88-75de321631cd |
| 2015-09-01 07:12:40 | 2015-09-01 07:14:33 | 99363eb8-682a-4110-ae0b-93e4e7b7c4e9 |
+---------------------+---------------------+--------------------------------------+
9 rows in set (0.00 sec)


mysql> select created_at,deleted_at,status,migration_type,instance_uuid from migrations;
+---------------------+------------+-----------+----------------+--------------------------------------+
| created_at          | deleted_at | status    | migration_type | instance_uuid                        |
+---------------------+------------+-----------+----------------+--------------------------------------+
| 2015-08-28 08:31:10 | NULL       | error     | live-migration | 0a130eb8-01b5-4ad7-bffd-64abe360e459 |
| 2015-08-31 05:58:49 | NULL       | confirmed | resize         | 0a130eb8-01b5-4ad7-bffd-64abe360e459 |
| 2015-08-31 06:36:06 | NULL       | error     | resize         | 0a130eb8-01b5-4ad7-bffd-64abe360e459 |
| 2015-08-31 06:40:56 | NULL       | migrating | resize         | 0a130eb8-01b5-4ad7-bffd-64abe360e459 |
| 2015-08-31 06:49:08 | NULL       | migrating | resize         | 57ec3d85-fd77-4b9a-b4f4-8910e892b612 |
| 2015-08-31 06:56:56 | NULL       | migrating | resize         | 7a54fd3b-fee4-4b40-9074-adb515f7fc50 |
| 2015-08-31 07:07:07 | NULL       | confirmed | resize         | 7a54fd3b-fee4-4b40-9074-adb515f7fc50 |
| 2015-08-31 07:47:30 | NULL       | confirmed | resize         | 0b141529-2127-4559-bf88-75de321631cd |
| 2015-08-31 07:50:14 | NULL       | error     | live-migration | 0b141529-2127-4559-bf88-75de321631cd |
| 2015-08-31 07:52:36 | NULL       | completed | live-migration | 0b141529-2127-4559-bf88-75de321631cd |
| 2015-09-01 07:14:21 | NULL       | error     | resize         | 99363eb8-682a-4110-ae0b-93e4e7b7c4e9 |
+---------------------+------------+-----------+----------------+--------------------------------------+
11 rows in set (0.00 sec)

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1490855

Title:
  resource_tracker keeps complaining instance not found for deleted
  instances

Status in OpenStack Compute (nova):
  New

Bug description:
  Some instances met error during resize or live-migration,  then
  resource_tracker keeps complaining instance not found even after they
  got deleted.

  ===== n-cpu.log =====
  647686:2015-09-01 07:26:27.172 DEBUG nova.compute.resource_tracker [req-36bf129b-5ed3-403d-9d88-143b0cc3810e None None] Migration instance not found: Instance 0a130eb8-01b5-4ad7-bffd-64abe360e459 could not be found.
  647712:2015-09-01 07:26:27.224 DEBUG nova.compute.resource_tracker [req-36bf129b-5ed3-403d-9d88-143b0cc3810e None None] Migration instance not found: Instance 57ec3d85-fd77-4b9a-b4f4-8910e892b612 could not be found.
  647738:2015-09-01 07:26:27.277 DEBUG nova.compute.resource_tracker [req-36bf129b-5ed3-403d-9d88-143b0cc3810e None None] Migration instance not found: Instance 7a54fd3b-fee4-4b40-9074-adb515f7fc50 could not be found.

  
  ====== db ========
  ubuntu@node-01:/opt/stack/nova$ nova list
  +----+------+--------+------------+-------------+----------+
  | ID | Name | Status | Task State | Power State | Networks |
  +----+------+--------+------------+-------------+----------+
  +----+------+--------+------------+-------------+----------+

  mysql> select created_at,deleted_at,uuid  from instances;
  +---------------------+---------------------+--------------------------------------+
  | created_at          | deleted_at          | uuid                                 |
  +---------------------+---------------------+--------------------------------------+
  | 2015-08-28 06:03:36 | 2015-08-31 06:51:18 | 57ec3d85-fd77-4b9a-b4f4-8910e892b612 |
  | 2015-08-28 06:04:30 | 2015-08-31 06:51:12 | 0a130eb8-01b5-4ad7-bffd-64abe360e459 |
  | 2015-08-28 06:10:56 | 2015-08-28 06:11:14 | 8217c85d-6e74-48b6-940d-7eeb6f7afddd |
  | 2015-08-28 06:12:26 | 2015-08-28 06:12:36 | 347218bb-166c-4228-8737-2b7baf310f21 |
  | 2015-08-28 06:12:54 | 2015-08-28 06:13:09 | 8815d09a-3b07-4156-a451-d6182ace8b61 |
  | 2015-08-28 06:13:27 | 2015-08-28 06:13:39 | 50c40182-ee2f-4553-9a40-828a45ba7026 |
  | 2015-08-31 06:54:58 | 2015-09-01 07:10:50 | 7a54fd3b-fee4-4b40-9074-adb515f7fc50 |
  | 2015-08-31 07:45:16 | 2015-09-01 07:10:40 | 0b141529-2127-4559-bf88-75de321631cd |
  | 2015-09-01 07:12:40 | 2015-09-01 07:14:33 | 99363eb8-682a-4110-ae0b-93e4e7b7c4e9 |
  +---------------------+---------------------+--------------------------------------+
  9 rows in set (0.00 sec)

  
  mysql> select created_at,deleted_at,status,migration_type,instance_uuid from migrations;
  +---------------------+------------+-----------+----------------+--------------------------------------+
  | created_at          | deleted_at | status    | migration_type | instance_uuid                        |
  +---------------------+------------+-----------+----------------+--------------------------------------+
  | 2015-08-28 08:31:10 | NULL       | error     | live-migration | 0a130eb8-01b5-4ad7-bffd-64abe360e459 |
  | 2015-08-31 05:58:49 | NULL       | confirmed | resize         | 0a130eb8-01b5-4ad7-bffd-64abe360e459 |
  | 2015-08-31 06:36:06 | NULL       | error     | resize         | 0a130eb8-01b5-4ad7-bffd-64abe360e459 |
  | 2015-08-31 06:40:56 | NULL       | migrating | resize         | 0a130eb8-01b5-4ad7-bffd-64abe360e459 |
  | 2015-08-31 06:49:08 | NULL       | migrating | resize         | 57ec3d85-fd77-4b9a-b4f4-8910e892b612 |
  | 2015-08-31 06:56:56 | NULL       | migrating | resize         | 7a54fd3b-fee4-4b40-9074-adb515f7fc50 |
  | 2015-08-31 07:07:07 | NULL       | confirmed | resize         | 7a54fd3b-fee4-4b40-9074-adb515f7fc50 |
  | 2015-08-31 07:47:30 | NULL       | confirmed | resize         | 0b141529-2127-4559-bf88-75de321631cd |
  | 2015-08-31 07:50:14 | NULL       | error     | live-migration | 0b141529-2127-4559-bf88-75de321631cd |
  | 2015-08-31 07:52:36 | NULL       | completed | live-migration | 0b141529-2127-4559-bf88-75de321631cd |
  | 2015-09-01 07:14:21 | NULL       | error     | resize         | 99363eb8-682a-4110-ae0b-93e4e7b7c4e9 |
  +---------------------+------------+-----------+----------------+--------------------------------------+
  11 rows in set (0.00 sec)

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1490855/+subscriptions