yahoo-eng-team team mailing list archive

[Doug Hellmann <doug@xxxxxxxxxxxxxxxx>]

** Changed in: nova
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1488435

Title:
  Resources on destination node doesn't cleanup if live-migration fails

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  I've deployed multinode devstack environment and tried to live-migrate volume-backed instance,
  live-migration fails during copying of glance image to destination. When it happens nova leaves destination host in inconsistent state, and attempt to run live-migration again fails.
  Here is small investigation of a problem:
  nova creates instance directory on target compute node:
  https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L6152
  in second run it fail here, because dest node wasn't properly cleanedup:
  https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L6148

  Also it's really strange, that when pre_live_migration on dest node fails, flow returns to source which decide is that really required to clean_up destination or not. It will be more clearer to wrap pre_live_migration with try/catch and rollback it before response to source node:
  https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4962-L4984

  Devstack nova version:
  stack@node1:/opt/stack/nova$ git log -1
  commit a71912ef463761187ef785118fbb0e51a5e91ba3
  Merge: ea4e95e 6b2d616
  Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
  Date:   Mon Aug 24 16:07:32 2015 +0000

      Merge "Fixed indentation"

  Controller with compute node nova.conf:
  http://paste.openstack.org/show/426934/

  Compute only nova.conf:
  http://paste.openstack.org/show/426936/

  
  *UPDATE* 
  root cause of the problem:
  rollback_live_migration_at_destination 
  do several things: 
  one of them is deallocation nova-network resources over call of setup_networks_on_host with teardown=True. If this call fails no 
  clean-up on driver side happens.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1488435/+subscriptions