yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1499405] [NEW] Live-migration fails for instances with LVM ephemeral devices

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Andrea Rosa <1499405@xxxxxxxxxxxxxxxxxx>
Date: Thu, 24 Sep 2015 14:38:25 -0000
Reply-to: Bug 1499405 <1499405@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

On a multinode devstack installation I have created a new lvm device:
sudo dd if=/dev/zero of=/loopy bs=1M count=5124 # 5gb
sudo losetup /dev/loop2 /loopy

sudo sfdisk /dev/loop2 << EOF
,,8e,,
EOF
sudo pvcreate /dev/loop2
sudo vgcreate encrypted-ephemeral /dev/loop2
 
Please note that the volume group name is "ecnrypted-ephemeral" but I am not using any encryption.
I set up my nova compute to use lvm images for the ephemeral disks:
[libvirt]
images_type = lvm
images_volume_group = encrypted-ephemeral

Then I changed the live migration uri to use tcp instead of ssh:
live_migration_uri = qemu+tcp://%s/system

I did the same configuration on a second compute node.

At this point I booted a vm which became ACTIVE with no error then I tried to migrate the instance:
nova live-migration <instance_name> devstackmn-2 --block-migrate

and the migration fails and in the libivirt log file there is this error reported:
2015-09-24 08:23:17.674+0000: 26629: error : virNetClientProgramDispatchError:175 : Failed to open file '/dev/encrypted-ephemeral/9c1c82cb-cf11-4b2b-a994-2d94d286bfb9_disk': No such file or directory

and in the nova-compute log file of the source node I see a similar
error:

2015-09-24 14:33:19.023 ERROR nova.virt.libvirt.driver [req-3f358ce5-7746-4532-9bb4-bf1189181ce8 admin admin] [instance: 9c1c82cb-cf11-4b2b-a994-2d94d286bfb9] Live Migration failure: Failed to open file '/dev/encrypted-ephemeral/9c1c82cb-cf11-4b2b-a994-2d94d286bfb9_disk': No such file or directory
2015-09-24 14:33:19.023 DEBUG nova.virt.libvirt.driver [req-3f358ce5-7746-4532-9bb4-bf1189181ce8 admin admin] [instance: 9c1c82cb-cf11-4b2b-a994-2d94d286bfb9] Migration operation thread notification from (pid=21098) thread_finished /opt/stack/nova/nova/virt/libvirt/driver.py:6041
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 457, in fire_timers
    timer()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
    cb(*args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 168, in _do_send
    waiter.switch(result)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
    result = function(*args, **kwargs)
  File "/opt/stack/nova/nova/utils.py", line 1154, in context_wrapper
    return func(*args, **kwargs)
  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5643, in _live_migration_operation
    instance=instance)
  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5611, in _live_migration_operation
    CONF.libvirt.live_migration_bandwidth)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
    result = proxy_call(self._autowrap, f, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
    rv = execute(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
    six.reraise(c, e, tb)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
    rv = meth(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 1586, in migrateToURI2
    if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self)
libvirtError: Failed to open file '/dev/encrypted-ephemeral/9c1c82cb-cf11-4b2b-a994-2d94d286bfb9_disk': No such file or directory
2015-09-24 14:33:19.224 DEBUG nova.virt.libvirt.driver [req-3f358ce5-7746-4532-9bb4-bf1189181ce8 admin admin] [instance: 9c1c82cb-cf11-4b2b-a994-2d94d286bfb9] VM running on src, migration failed from (pid=21098) _live_migration_monitor /opt/stack/nova/nova/virt/libvirt/driver.py:5847
2015-09-24 14:33:19.224 DEBUG nova.virt.libvirt.driver [req-3f358ce5-7746-4532-9bb4-bf1189181ce8 admin admin] [instance: 9c1c82cb-cf11-4b2b-a994-2d94d286bfb9] Fixed incorrect job type to be 4 from (pid=21098) _live_migration_monitor /opt/stack/nova/nova/virt/libvirt/driver.py:5867
2015-09-24 14:33:19.224 ERROR nova.virt.libvirt.driver [req-3f358ce5-7746-4532-9bb4-bf1189181ce8 admin admin] [instance: 9c1c82cb-cf11-4b2b-a994-2d94d286bfb9] Migration operation has aborted


Nova version as per git log

commit 806dc025dbc9a102f7bace78be9c56f8e9af0929
Merge: 424aaf3 ae3d970
Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
Date:   Sat Sep 19 05:03:38 2015 +0000

    Merge "libvirt:Remove duplicated check code for config option
sysinfo_serial"

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1499405

Title:
  Live-migration fails for instances with LVM ephemeral devices

Status in OpenStack Compute (nova):
  New

Bug description:
  On a multinode devstack installation I have created a new lvm device:
  sudo dd if=/dev/zero of=/loopy bs=1M count=5124 # 5gb
  sudo losetup /dev/loop2 /loopy

  sudo sfdisk /dev/loop2 << EOF
  ,,8e,,
  EOF
  sudo pvcreate /dev/loop2
  sudo vgcreate encrypted-ephemeral /dev/loop2
   
  Please note that the volume group name is "ecnrypted-ephemeral" but I am not using any encryption.
  I set up my nova compute to use lvm images for the ephemeral disks:
  [libvirt]
  images_type = lvm
  images_volume_group = encrypted-ephemeral

  Then I changed the live migration uri to use tcp instead of ssh:
  live_migration_uri = qemu+tcp://%s/system

  I did the same configuration on a second compute node.

  At this point I booted a vm which became ACTIVE with no error then I tried to migrate the instance:
  nova live-migration <instance_name> devstackmn-2 --block-migrate

  and the migration fails and in the libivirt log file there is this error reported:
  2015-09-24 08:23:17.674+0000: 26629: error : virNetClientProgramDispatchError:175 : Failed to open file '/dev/encrypted-ephemeral/9c1c82cb-cf11-4b2b-a994-2d94d286bfb9_disk': No such file or directory

  and in the nova-compute log file of the source node I see a similar
  error:

  2015-09-24 14:33:19.023 ERROR nova.virt.libvirt.driver [req-3f358ce5-7746-4532-9bb4-bf1189181ce8 admin admin] [instance: 9c1c82cb-cf11-4b2b-a994-2d94d286bfb9] Live Migration failure: Failed to open file '/dev/encrypted-ephemeral/9c1c82cb-cf11-4b2b-a994-2d94d286bfb9_disk': No such file or directory
  2015-09-24 14:33:19.023 DEBUG nova.virt.libvirt.driver [req-3f358ce5-7746-4532-9bb4-bf1189181ce8 admin admin] [instance: 9c1c82cb-cf11-4b2b-a994-2d94d286bfb9] Migration operation thread notification from (pid=21098) thread_finished /opt/stack/nova/nova/virt/libvirt/driver.py:6041
  Traceback (most recent call last):
    File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 457, in fire_timers
      timer()
    File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
      cb(*args, **kw)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 168, in _do_send
      waiter.switch(result)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
      result = function(*args, **kwargs)
    File "/opt/stack/nova/nova/utils.py", line 1154, in context_wrapper
      return func(*args, **kwargs)
    File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5643, in _live_migration_operation
      instance=instance)
    File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
      six.reraise(self.type_, self.value, self.tb)
    File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5611, in _live_migration_operation
      CONF.libvirt.live_migration_bandwidth)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
      result = proxy_call(self._autowrap, f, *args, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
      rv = execute(f, *args, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
      six.reraise(c, e, tb)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
      rv = meth(*args, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 1586, in migrateToURI2
      if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self)
  libvirtError: Failed to open file '/dev/encrypted-ephemeral/9c1c82cb-cf11-4b2b-a994-2d94d286bfb9_disk': No such file or directory
  2015-09-24 14:33:19.224 DEBUG nova.virt.libvirt.driver [req-3f358ce5-7746-4532-9bb4-bf1189181ce8 admin admin] [instance: 9c1c82cb-cf11-4b2b-a994-2d94d286bfb9] VM running on src, migration failed from (pid=21098) _live_migration_monitor /opt/stack/nova/nova/virt/libvirt/driver.py:5847
  2015-09-24 14:33:19.224 DEBUG nova.virt.libvirt.driver [req-3f358ce5-7746-4532-9bb4-bf1189181ce8 admin admin] [instance: 9c1c82cb-cf11-4b2b-a994-2d94d286bfb9] Fixed incorrect job type to be 4 from (pid=21098) _live_migration_monitor /opt/stack/nova/nova/virt/libvirt/driver.py:5867
  2015-09-24 14:33:19.224 ERROR nova.virt.libvirt.driver [req-3f358ce5-7746-4532-9bb4-bf1189181ce8 admin admin] [instance: 9c1c82cb-cf11-4b2b-a994-2d94d286bfb9] Migration operation has aborted


  Nova version as per git log

  commit 806dc025dbc9a102f7bace78be9c56f8e9af0929
  Merge: 424aaf3 ae3d970
  Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
  Date:   Sat Sep 19 05:03:38 2015 +0000

      Merge "libvirt:Remove duplicated check code for config option
  sysinfo_serial"

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1499405/+subscriptions
Follow ups

[Bug 1499405] Re: Live block migration fails for instances with LVM ephemeral devices
From: Andrea Rosa, 2015-10-05