← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #66002

[Bug 1705700] [NEW] live migration does not work after volume migration

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Hi,

$subj

[#]> rpm -qi openstack-nova-common.noarch
Name        : openstack-nova-common
Epoch       : 1
Version     : 15.0.0
Release     : 1.el7

STR:
-configure 2x compute node with cinder storage and lvm back-ends;
-crate bootable volume from image;
-create instance and use bootable volume;
-migrate volume from current node to another
-try to perform live migration

Actual Result:
-live migration fails:
-------------------------
2017-07-21 11:33:00.554 4552 ERROR nova.virt.libvirt.driver [req-792a37bf-3a2e-4976-84ad-cc308eb1ffbf - - - - -] [instance: 5dd0b6ab-8743-42a2-af7a-38f1fb10dedf] Live Migration failure: missing source information for device vda
2017-07-21 11:33:00.599 4552 ERROR nova.virt.libvirt.driver [req-792a37bf-3a2e-4976-84ad-cc308eb1ffbf - - - - -] [instance: 5dd0b6ab-8743-42a2-af7a-38f1fb10dedf] Migration operation has aborted


Jul 21 11:33:00 compute-02 nova-compute: Traceback (most recent call last):
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 457, in fire_timers
Jul 21 11:33:00 compute-02 nova-compute: timer()
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 58, in __call__
Jul 21 11:33:00 compute-02 nova-compute: cb(*args, **kw)
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 168, in _do_send
Jul 21 11:33:00 compute-02 nova-compute: waiter.switch(result)
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
Jul 21 11:33:00 compute-02 nova-compute: result = function(*args, **kwargs)
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/nova/utils.py", line 1087, in context_wrapper
Jul 21 11:33:00 compute-02 nova-compute: return func(*args, **kwargs)
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6150, in _live_migration_operation
Jul 21 11:33:00 compute-02 nova-compute: instance=instance)
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Jul 21 11:33:00 compute-02 nova-compute: self.force_reraise()
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jul 21 11:33:00 compute-02 nova-compute: six.reraise(self.type_, self.value, self.tb)
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6143, in _live_migration_operation
Jul 21 11:33:00 compute-02 nova-compute: bandwidth=CONF.libvirt.live_migration_bandwidth)
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 623, in migrate
Jul 21 11:33:00 compute-02 nova-compute: destination, params=params, flags=flags)
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
Jul 21 11:33:00 compute-02 nova-compute: result = proxy_call(self._autowrap, f, *args, **kwargs)
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
Jul 21 11:33:00 compute-02 nova-compute: rv = execute(f, *args, **kwargs)
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
Jul 21 11:33:00 compute-02 nova-compute: six.reraise(c, e, tb)
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
Jul 21 11:33:00 compute-02 nova-compute: rv = meth(*args, **kwargs)
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1939, in migrateToURI3
Jul 21 11:33:00 compute-02 nova-compute: if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self)
Jul 21 11:33:00 compute-02 nova-compute: libvirtError: missing source information for device vda
-------------------------

Here is disk description when instance was created
--------------------------
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/disk/by-path/ip-10.30.30.101:3260-iscsi-iqn.2010-10.org.openstack:volume-fcc11b41-514e-4a91-a29e-4632c4f42648-lun-0'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <serial>4c0b28d5-fc0b-4907-b563-b47b521bc945</serial>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
--------------------------

Here is disk description during migration:
--------------------------
   <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/disk/by-path/ip-10.30.30.101:3260-iscsi-iqn.2010-10.org.openstack:volume-fcc11b41-514e-4a91-a29e-4632c4f42648-lun-0'/>
      <backingStore/>
      <mirror type='file' file='/dev/disk/by-path/ip-10.30.30.100:3260-iscsi-iqn.2010-10.org.openstack:volume-7604d08d-e987-4c8d-9a9b-56f3258d5b60-lun-0' format='raw' job='copy'>
        <format type='raw'/>
        <source file='/dev/disk/by-path/ip-10.30.30.100:3260-iscsi-iqn.2010-10.org.openstack:volume-7604d08d-e987-4c8d-9a9b-56f3258d5b60-lun-0'/>
      </mirror>
      <target dev='vda' bus='virtio'/>
      <serial>4c0b28d5-fc0b-4907-b563-b47b521bc945</serial>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
--------------------------

Here is disk description after the migration:
--------------------------
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/dev/disk/by-path/ip-10.30.30.100:3260-iscsi-iqn.2010-10.org.openstack:volume-7604d08d-e987-4c8d-9a9b-56f3258d5b60-lun-0'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <serial>4c0b28d5-fc0b-4907-b563-b47b521bc945</serial>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
--------------------------


Here is disk description after instance hard reboot:
--------------------------
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/disk/by-path/ip-10.30.30.100:3260-iscsi-iqn.2010-10.org.openstack:volume-7604d08d-e987-4c8d-9a9b-56f3258d5b60-lun-0'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <serial>4c0b28d5-fc0b-4907-b563-b47b521bc945</serial>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
--------------------------

The issue is caused by the fact that dist type is changed from 'block'
to 'file' after volume migration. Please note that after hard reboot
instance the xml description is fixed and live migration works.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1705700

Title:
  live migration does not work after volume migration

Status in OpenStack Compute (nova):
  New

Bug description:
  Hi,

  $subj

  [#]> rpm -qi openstack-nova-common.noarch
  Name        : openstack-nova-common
  Epoch       : 1
  Version     : 15.0.0
  Release     : 1.el7

  STR:
  -configure 2x compute node with cinder storage and lvm back-ends;
  -crate bootable volume from image;
  -create instance and use bootable volume;
  -migrate volume from current node to another
  -try to perform live migration

  Actual Result:
  -live migration fails:
  -------------------------
  2017-07-21 11:33:00.554 4552 ERROR nova.virt.libvirt.driver [req-792a37bf-3a2e-4976-84ad-cc308eb1ffbf - - - - -] [instance: 5dd0b6ab-8743-42a2-af7a-38f1fb10dedf] Live Migration failure: missing source information for device vda
  2017-07-21 11:33:00.599 4552 ERROR nova.virt.libvirt.driver [req-792a37bf-3a2e-4976-84ad-cc308eb1ffbf - - - - -] [instance: 5dd0b6ab-8743-42a2-af7a-38f1fb10dedf] Migration operation has aborted

  
  Jul 21 11:33:00 compute-02 nova-compute: Traceback (most recent call last):
  Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 457, in fire_timers
  Jul 21 11:33:00 compute-02 nova-compute: timer()
  Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 58, in __call__
  Jul 21 11:33:00 compute-02 nova-compute: cb(*args, **kw)
  Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 168, in _do_send
  Jul 21 11:33:00 compute-02 nova-compute: waiter.switch(result)
  Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
  Jul 21 11:33:00 compute-02 nova-compute: result = function(*args, **kwargs)
  Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/nova/utils.py", line 1087, in context_wrapper
  Jul 21 11:33:00 compute-02 nova-compute: return func(*args, **kwargs)
  Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6150, in _live_migration_operation
  Jul 21 11:33:00 compute-02 nova-compute: instance=instance)
  Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jul 21 11:33:00 compute-02 nova-compute: self.force_reraise()
  Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jul 21 11:33:00 compute-02 nova-compute: six.reraise(self.type_, self.value, self.tb)
  Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6143, in _live_migration_operation
  Jul 21 11:33:00 compute-02 nova-compute: bandwidth=CONF.libvirt.live_migration_bandwidth)
  Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 623, in migrate
  Jul 21 11:33:00 compute-02 nova-compute: destination, params=params, flags=flags)
  Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
  Jul 21 11:33:00 compute-02 nova-compute: result = proxy_call(self._autowrap, f, *args, **kwargs)
  Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
  Jul 21 11:33:00 compute-02 nova-compute: rv = execute(f, *args, **kwargs)
  Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
  Jul 21 11:33:00 compute-02 nova-compute: six.reraise(c, e, tb)
  Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
  Jul 21 11:33:00 compute-02 nova-compute: rv = meth(*args, **kwargs)
  Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1939, in migrateToURI3
  Jul 21 11:33:00 compute-02 nova-compute: if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self)
  Jul 21 11:33:00 compute-02 nova-compute: libvirtError: missing source information for device vda
  -------------------------

  Here is disk description when instance was created
  --------------------------
      <disk type='block' device='disk'>
        <driver name='qemu' type='raw' cache='none' io='native'/>
        <source dev='/dev/disk/by-path/ip-10.30.30.101:3260-iscsi-iqn.2010-10.org.openstack:volume-fcc11b41-514e-4a91-a29e-4632c4f42648-lun-0'/>
        <backingStore/>
        <target dev='vda' bus='virtio'/>
        <serial>4c0b28d5-fc0b-4907-b563-b47b521bc945</serial>
        <alias name='virtio-disk0'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
      </disk>
  --------------------------

  Here is disk description during migration:
  --------------------------
     <disk type='block' device='disk'>
        <driver name='qemu' type='raw' cache='none' io='native'/>
        <source dev='/dev/disk/by-path/ip-10.30.30.101:3260-iscsi-iqn.2010-10.org.openstack:volume-fcc11b41-514e-4a91-a29e-4632c4f42648-lun-0'/>
        <backingStore/>
        <mirror type='file' file='/dev/disk/by-path/ip-10.30.30.100:3260-iscsi-iqn.2010-10.org.openstack:volume-7604d08d-e987-4c8d-9a9b-56f3258d5b60-lun-0' format='raw' job='copy'>
          <format type='raw'/>
          <source file='/dev/disk/by-path/ip-10.30.30.100:3260-iscsi-iqn.2010-10.org.openstack:volume-7604d08d-e987-4c8d-9a9b-56f3258d5b60-lun-0'/>
        </mirror>
        <target dev='vda' bus='virtio'/>
        <serial>4c0b28d5-fc0b-4907-b563-b47b521bc945</serial>
        <alias name='virtio-disk0'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
      </disk>
  --------------------------

  Here is disk description after the migration:
  --------------------------
      <disk type='file' device='disk'>
        <driver name='qemu' type='raw' cache='none' io='native'/>
        <source file='/dev/disk/by-path/ip-10.30.30.100:3260-iscsi-iqn.2010-10.org.openstack:volume-7604d08d-e987-4c8d-9a9b-56f3258d5b60-lun-0'/>
        <backingStore/>
        <target dev='vda' bus='virtio'/>
        <serial>4c0b28d5-fc0b-4907-b563-b47b521bc945</serial>
        <alias name='virtio-disk0'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
      </disk>
  --------------------------

  
  Here is disk description after instance hard reboot:
  --------------------------
      <disk type='block' device='disk'>
        <driver name='qemu' type='raw' cache='none' io='native'/>
        <source dev='/dev/disk/by-path/ip-10.30.30.100:3260-iscsi-iqn.2010-10.org.openstack:volume-7604d08d-e987-4c8d-9a9b-56f3258d5b60-lun-0'/>
        <backingStore/>
        <target dev='vda' bus='virtio'/>
        <serial>4c0b28d5-fc0b-4907-b563-b47b521bc945</serial>
        <alias name='virtio-disk0'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
      </disk>
  --------------------------

  The issue is caused by the fact that dist type is changed from 'block'
  to 'file' after volume migration. Please note that after hard reboot
  instance the xml description is fixed and live migration works.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1705700/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next