yahoo-eng-team team mailing list archive

[Pawel Koniszewski <pawel.koniszewski@xxxxxxxxx>]

Public bug reported:

When e.g. nova-compute or rabbit-server dies during live migration and
somehow nova-compute is not able to report new host for migrated VM,
then after successful system recovery nova deletes VM instead of sending
host update.  This is from nova log:

09:00:25.704 INFO nova.compute.manager [-] [instance: b8a3bdd6-809f-44b4-875d-df3feafab41a] Deleting instance as its host (node-16) is not equal to our host (node-15).
09:00:27.972 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on 10.4.8.2:5672
09:00:27.972 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
09:00:28.981 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.4.8.2:5672
09:00:36.464 INFO nova.compute.manager [-] Lifecycle event 1 on VM b8a3bdd6-809f-44b4-875d-df3feafab41a
09:00:36.468 INFO nova.virt.libvirt.driver [-] [instance: b8a3bdd6-809f-44b4-875d-df3feafab41a] Instance destroyed successfully.
09:00:36.471 INFO nova.virt.libvirt.firewall [-] [instance: b8a3bdd6-809f-44b4-875d-df3feafab41a] Attempted to unfilter instance which is not filtered
09:00:36.521 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.4.8.2:5672
09:00:36.565 INFO nova.compute.manager [req-93e15eda-8d65-49f5-a195-52b91da7aa68 None] [instance: b8a3bdd6-809f-44b4-875d-df3feafab41a] During the sync_power process the instance has moved from host node-15 to host node-16
09:00:36.566 INFO nova.virt.libvirt.driver [-] [instance: b8a3bdd6-809f-44b4-875d-df3feafab41a] Deleting instance files /var/lib/nova/instances/b8a3bdd6-809f-44b4-875d-df3feafab41a
09:00:36.566 INFO nova.virt.libvirt.driver [-] [instance: b8a3bdd6-809f-44b4-875d-df3feafab41a] Deletion of /var/lib/nova/instances/b8a3bdd6-809f-44b4-875d-df3feafab41a complete

However VM record in database is still present (with state MIGRATING)
and volume is still attached to VM that does not exist.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1396529

Title:
  Nova deletes instance when compute/rabbit is dead at the end of live
  migration

Status in OpenStack Compute (Nova):
  New

Bug description:
  When e.g. nova-compute or rabbit-server dies during live migration and
  somehow nova-compute is not able to report new host for migrated VM,
  then after successful system recovery nova deletes VM instead of
  sending host update.  This is from nova log:

  09:00:25.704 INFO nova.compute.manager [-] [instance: b8a3bdd6-809f-44b4-875d-df3feafab41a] Deleting instance as its host (node-16) is not equal to our host (node-15).
  09:00:27.972 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on 10.4.8.2:5672
  09:00:27.972 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
  09:00:28.981 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.4.8.2:5672
  09:00:36.464 INFO nova.compute.manager [-] Lifecycle event 1 on VM b8a3bdd6-809f-44b4-875d-df3feafab41a
  09:00:36.468 INFO nova.virt.libvirt.driver [-] [instance: b8a3bdd6-809f-44b4-875d-df3feafab41a] Instance destroyed successfully.
  09:00:36.471 INFO nova.virt.libvirt.firewall [-] [instance: b8a3bdd6-809f-44b4-875d-df3feafab41a] Attempted to unfilter instance which is not filtered
  09:00:36.521 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.4.8.2:5672
  09:00:36.565 INFO nova.compute.manager [req-93e15eda-8d65-49f5-a195-52b91da7aa68 None] [instance: b8a3bdd6-809f-44b4-875d-df3feafab41a] During the sync_power process the instance has moved from host node-15 to host node-16
  09:00:36.566 INFO nova.virt.libvirt.driver [-] [instance: b8a3bdd6-809f-44b4-875d-df3feafab41a] Deleting instance files /var/lib/nova/instances/b8a3bdd6-809f-44b4-875d-df3feafab41a
  09:00:36.566 INFO nova.virt.libvirt.driver [-] [instance: b8a3bdd6-809f-44b4-875d-df3feafab41a] Deletion of /var/lib/nova/instances/b8a3bdd6-809f-44b4-875d-df3feafab41a complete

  However VM record in database is still present (with state MIGRATING)
  and volume is still attached to VM that does not exist.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1396529/+subscriptions