← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #53101

[Bug 1598353] [NEW] Instance's cleaned filed was set 1 after rollback at destination when do live-migration failed

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Description
===========
I did live-migration for an instance, error occured by libvirt process.
Then I found field of 'cleaned' was '1' in instances table, but this instance was not deleted.

Steps to reproduce
==================
* I did live-migrate instance:
  $ nova live-migration --block fa0b2b1e-4e00-43bf-bb5d-13ba86277846

* then I found this instance's vm_state was active and instance's host was still on source compute node.
log is as follow:
2016-07-02 12:06:19.656 4078 ERROR nova.virt.libvirt.driver [req-29a088d7-e52c-43bf-b872-92c34175b8d4 0b78e7b5c15c49769d4f5af63628164c e948821962cd4509a6c687c7b7dcb96f - - -] [instance: fa0b2b1e-4e00-43bf-bb5d-13ba86277846] Live Migration failure: failed to connect to monitor socket: No such process
2016-07-02 12:06:20.044 4078 ERROR nova.virt.libvirt.driver [req-29a088d7-e52c-43bf-b872-92c34175b8d4 0b78e7b5c15c49769d4f5af63628164c e948821962cd4509a6c687c7b7dcb96f - - -] [instance: fa0b2b1e-4e00-43bf-bb5d-13ba86277846] Migration operation has aborted

* then I query database
MariaDB [nova]> select uuid,updated_at,deleted,cleaned from instances where deleted=0 and cleaned=1;
+--------------------------------------+---------------------+---------+---------+
| uuid                                 | updated_at          | deleted | cleaned |
+--------------------------------------+---------------------+---------+---------+
| fa0b2b1e-4e00-43bf-bb5d-13ba86277846 | 2016-07-02 04:06:22 |       0 |       1 |
+--------------------------------------+---------------------+---------+---------+
1 row in set (0.00 sec)

MariaDB [nova]> select * from migrations where instance_uuid='fa0b2b1e-4e00-43bf-bb5d-13ba86277846';
+---------------------+---------------------+------------+------+----------------+---------------------+-------------+---------------+--------------------------------------+----------------------+----------------------+-------------+---------------------+---------+----------------+--------+
| created_at          | updated_at          | deleted_at | id   | source_compute | dest_compute        | dest_host   | status        | instance_uuid                        | old_instance_type_id | new_instance_type_id | source_node | dest_node           | deleted | migration_type | hidden |
+---------------------+---------------------+------------+------+----------------+---------------------+-------------+---------------+--------------------------------------+----------------------+----------------------+-------------+---------------------+---------+----------------+--------+
| 2016-07-02 04:06:02 | 2016-07-02 04:06:20 | NULL       | 1912 | SBCRslot2      | SBCR-chenling-slot4 | 10.43.239.3 | error         | fa0b2b1e-4e00-43bf-bb5d-13ba86277846 |                  207 |                  207 | NULL        | SBCR-chenling-slot4 |       0 | live-migration |      0 |
+---------------------+---------------------+------------+------+----------------+---------------------+-------------+---------------+--------------------------------------+----------------------+----------------------+-------------+---------------------+---------+----------------+--------+
1 row in set (0.00 sec)

Expected result
===============
I hope instance's cleaned is 0.

Actual result
=============
+--------------------------------------+---------------------+---------+---------+
| uuid                                 | updated_at          | deleted | cleaned |
+--------------------------------------+---------------------+---------+---------+
| fa0b2b1e-4e00-43bf-bb5d-13ba86277846 | 2016-07-02 04:06:22 |       0 |       1 |
+--------------------------------------+---------------------+---------+---------+

deleted == 0 and cleaned == 1, which is not consistent.

Environment
===========
1. Exact version of OpenStack you are running. See the following
  Mitaka

2. Which hypervisor did you use?
   Libvirt + KVM

3. Which storage type did you use?
   Local storage

3. Which networking type did you use?
   Neutron with OpenVSwitch

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1598353

Title:
  Instance's cleaned filed was set 1 after rollback at destination when
  do live-migration failed

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  I did live-migration for an instance, error occured by libvirt process.
  Then I found field of 'cleaned' was '1' in instances table, but this instance was not deleted.

  Steps to reproduce
  ==================
  * I did live-migrate instance:
    $ nova live-migration --block fa0b2b1e-4e00-43bf-bb5d-13ba86277846

  * then I found this instance's vm_state was active and instance's host was still on source compute node.
  log is as follow:
  2016-07-02 12:06:19.656 4078 ERROR nova.virt.libvirt.driver [req-29a088d7-e52c-43bf-b872-92c34175b8d4 0b78e7b5c15c49769d4f5af63628164c e948821962cd4509a6c687c7b7dcb96f - - -] [instance: fa0b2b1e-4e00-43bf-bb5d-13ba86277846] Live Migration failure: failed to connect to monitor socket: No such process
  2016-07-02 12:06:20.044 4078 ERROR nova.virt.libvirt.driver [req-29a088d7-e52c-43bf-b872-92c34175b8d4 0b78e7b5c15c49769d4f5af63628164c e948821962cd4509a6c687c7b7dcb96f - - -] [instance: fa0b2b1e-4e00-43bf-bb5d-13ba86277846] Migration operation has aborted

  * then I query database
  MariaDB [nova]> select uuid,updated_at,deleted,cleaned from instances where deleted=0 and cleaned=1;
  +--------------------------------------+---------------------+---------+---------+
  | uuid                                 | updated_at          | deleted | cleaned |
  +--------------------------------------+---------------------+---------+---------+
  | fa0b2b1e-4e00-43bf-bb5d-13ba86277846 | 2016-07-02 04:06:22 |       0 |       1 |
  +--------------------------------------+---------------------+---------+---------+
  1 row in set (0.00 sec)

  MariaDB [nova]> select * from migrations where instance_uuid='fa0b2b1e-4e00-43bf-bb5d-13ba86277846';
  +---------------------+---------------------+------------+------+----------------+---------------------+-------------+---------------+--------------------------------------+----------------------+----------------------+-------------+---------------------+---------+----------------+--------+
  | created_at          | updated_at          | deleted_at | id   | source_compute | dest_compute        | dest_host   | status        | instance_uuid                        | old_instance_type_id | new_instance_type_id | source_node | dest_node           | deleted | migration_type | hidden |
  +---------------------+---------------------+------------+------+----------------+---------------------+-------------+---------------+--------------------------------------+----------------------+----------------------+-------------+---------------------+---------+----------------+--------+
  | 2016-07-02 04:06:02 | 2016-07-02 04:06:20 | NULL       | 1912 | SBCRslot2      | SBCR-chenling-slot4 | 10.43.239.3 | error         | fa0b2b1e-4e00-43bf-bb5d-13ba86277846 |                  207 |                  207 | NULL        | SBCR-chenling-slot4 |       0 | live-migration |      0 |
  +---------------------+---------------------+------------+------+----------------+---------------------+-------------+---------------+--------------------------------------+----------------------+----------------------+-------------+---------------------+---------+----------------+--------+
  1 row in set (0.00 sec)

  Expected result
  ===============
  I hope instance's cleaned is 0.

  Actual result
  =============
  +--------------------------------------+---------------------+---------+---------+
  | uuid                                 | updated_at          | deleted | cleaned |
  +--------------------------------------+---------------------+---------+---------+
  | fa0b2b1e-4e00-43bf-bb5d-13ba86277846 | 2016-07-02 04:06:22 |       0 |       1 |
  +--------------------------------------+---------------------+---------+---------+

  deleted == 0 and cleaned == 1, which is not consistent.

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
    Mitaka

  2. Which hypervisor did you use?
     Libvirt + KVM

  3. Which storage type did you use?
     Local storage

  3. Which networking type did you use?
     Neutron with OpenVSwitch

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1598353/+subscriptions
  Thread Previous • Date Previous • • Thread Next