yahoo-eng-team team mailing list archive

[Andrea Rosa <1499405@xxxxxxxxxxxxxxxxxx>]

I had time to did more investigation and I found that the live migration for LVM backed instances is not supported:
https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L6665

This change https://review.openstack.org/73387 was for raising the NotImplementedError exception when we try LVM migration.
In my case I didn't see that exception probably because I hit a different exception before, we still need to check that the control about LVM is done at the right time, but that will be a different bug.

I am going to mark  this bug as Invalid as it is a duplicate of
https://bugs.launchpad.net/nova/+bug/1282643

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1499405

Title:
  Live block migration fails for instances with LVM ephemeral devices

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  On a multinode devstack installation I have created a new lvm device:
  sudo dd if=/dev/zero of=/loopy bs=1M count=5124 # 5gb
  sudo losetup /dev/loop2 /loopy

  sudo sfdisk /dev/loop2 << EOF
  ,,8e,,
  EOF
  sudo pvcreate /dev/loop2
  sudo vgcreate encrypted-ephemeral /dev/loop2
   
  Please note that the volume group name is "ecnrypted-ephemeral" but I am not using any encryption.
  I set up my nova compute to use lvm images for the ephemeral disks:
  [libvirt]
  images_type = lvm
  images_volume_group = encrypted-ephemeral

  Then I changed the live migration uri to use tcp instead of ssh:
  live_migration_uri = qemu+tcp://%s/system

  I did the same configuration on a second compute node.

  At this point I booted a vm which became ACTIVE with no error then I tried to migrate the instance:
  nova live-migration <instance_name> devstackmn-2 --block-migrate

  and the migration fails and in the libivirt log file there is this error reported:
  2015-09-24 08:23:17.674+0000: 26629: error : virNetClientProgramDispatchError:175 : Failed to open file '/dev/encrypted-ephemeral/9c1c82cb-cf11-4b2b-a994-2d94d286bfb9_disk': No such file or directory

  and in the nova-compute log file of the source node I see a similar
  error:

  2015-09-24 14:33:19.023 ERROR nova.virt.libvirt.driver [req-3f358ce5-7746-4532-9bb4-bf1189181ce8 admin admin] [instance: 9c1c82cb-cf11-4b2b-a994-2d94d286bfb9] Live Migration failure: Failed to open file '/dev/encrypted-ephemeral/9c1c82cb-cf11-4b2b-a994-2d94d286bfb9_disk': No such file or directory
  2015-09-24 14:33:19.023 DEBUG nova.virt.libvirt.driver [req-3f358ce5-7746-4532-9bb4-bf1189181ce8 admin admin] [instance: 9c1c82cb-cf11-4b2b-a994-2d94d286bfb9] Migration operation thread notification from (pid=21098) thread_finished /opt/stack/nova/nova/virt/libvirt/driver.py:6041
  Traceback (most recent call last):
    File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 457, in fire_timers
      timer()
    File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
      cb(*args, **kw)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 168, in _do_send
      waiter.switch(result)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
      result = function(*args, **kwargs)
    File "/opt/stack/nova/nova/utils.py", line 1154, in context_wrapper
      return func(*args, **kwargs)
    File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5643, in _live_migration_operation
      instance=instance)
    File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
      six.reraise(self.type_, self.value, self.tb)
    File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5611, in _live_migration_operation
      CONF.libvirt.live_migration_bandwidth)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
      result = proxy_call(self._autowrap, f, *args, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
      rv = execute(f, *args, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
      six.reraise(c, e, tb)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
      rv = meth(*args, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 1586, in migrateToURI2
      if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self)
  libvirtError: Failed to open file '/dev/encrypted-ephemeral/9c1c82cb-cf11-4b2b-a994-2d94d286bfb9_disk': No such file or directory
  2015-09-24 14:33:19.224 DEBUG nova.virt.libvirt.driver [req-3f358ce5-7746-4532-9bb4-bf1189181ce8 admin admin] [instance: 9c1c82cb-cf11-4b2b-a994-2d94d286bfb9] VM running on src, migration failed from (pid=21098) _live_migration_monitor /opt/stack/nova/nova/virt/libvirt/driver.py:5847
  2015-09-24 14:33:19.224 DEBUG nova.virt.libvirt.driver [req-3f358ce5-7746-4532-9bb4-bf1189181ce8 admin admin] [instance: 9c1c82cb-cf11-4b2b-a994-2d94d286bfb9] Fixed incorrect job type to be 4 from (pid=21098) _live_migration_monitor /opt/stack/nova/nova/virt/libvirt/driver.py:5867
  2015-09-24 14:33:19.224 ERROR nova.virt.libvirt.driver [req-3f358ce5-7746-4532-9bb4-bf1189181ce8 admin admin] [instance: 9c1c82cb-cf11-4b2b-a994-2d94d286bfb9] Migration operation has aborted


  Nova version as per git log

  commit 806dc025dbc9a102f7bace78be9c56f8e9af0929
  Merge: 424aaf3 ae3d970
  Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
  Date:   Sat Sep 19 05:03:38 2015 +0000

      Merge "libvirt:Remove duplicated check code for config option
  sysinfo_serial"

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1499405/+subscriptions