yahoo-eng-team team mailing list archive

[Alexandre arents <1829000@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Description
===========
When we run live block migration on instance with a deleted glance image,
it may failed with following logs:

-- nova-compute-log: --
2019-05-10 11:06:27.417 248758 ERROR nova.virt.libvirt.driver [req-b28b9aca-9135-4258-93a6-a802e6192c60 f7929cd1d8994661b88aff12977c8b9e 54f4d231201b4944a5fa4587a09bda28 - - -] [instance: 84601bd4-a6ee-4e00-a5bc-f7c80def7ec5] Migration operation has aborted
2019-05-10 11:06:27.566 248758 ERROR nova.virt.libvirt.driver [req-b28b9aca-9135-4258-93a6-a802e6192c60 f7929cd1d8994661b88aff12977c8b9e 54f4d231201b4944a5fa4587a09bda28 - - -] [instance: 84601bd4-a6ee-4e00-a5bc-f7c80def7ec5] Live Migration failure: internal error: info migration reply was missing return status

-- on target host /var/log/libvirt/qemu/instance-xxx.log: --
/build/qemu-nffE1h/qemu-2.x/nbd.c:nbd_trip():L1310: From: 2416967680, Len: 65536, Size: 2361393152, Offset: 0
/build/qemu-nffE1h/qemu-2.x/nbd.c:nbd_trip():L1311: requested operation past EOF--bad client?
/build/qemu-nffE1h/qemu-2.x/nbd.c:nbd_trip():L1310: From: 3624927232, Len: 589824, Size: 2361393152, Offset: 0

Seems that pre_live_migration task do not setup correctly target instance disk:
-because glance image is not existant, it fallbacks to remote host copy method.
-in this context, image.cache() is called without instance disk size parameter.
-consequence is instance disk is not resized to the correct size and remain with the size of backing file, so the disk is too small, making failed libvirt live migration.


Steps to reproduce
==================
* Spawn qcow2 instance with glance image size << of flavor disk instance size
* generate few user data in instance.
* delete glance image.
* run live block migration.

Environment
===========
Issue observed in Newton, still present in master.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1829000

Title:
  live migration (block-migrate) may failed if instance image is deleted
  in glance

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  When we run live block migration on instance with a deleted glance image,
  it may failed with following logs:

  -- nova-compute-log: --
  2019-05-10 11:06:27.417 248758 ERROR nova.virt.libvirt.driver [req-b28b9aca-9135-4258-93a6-a802e6192c60 f7929cd1d8994661b88aff12977c8b9e 54f4d231201b4944a5fa4587a09bda28 - - -] [instance: 84601bd4-a6ee-4e00-a5bc-f7c80def7ec5] Migration operation has aborted
  2019-05-10 11:06:27.566 248758 ERROR nova.virt.libvirt.driver [req-b28b9aca-9135-4258-93a6-a802e6192c60 f7929cd1d8994661b88aff12977c8b9e 54f4d231201b4944a5fa4587a09bda28 - - -] [instance: 84601bd4-a6ee-4e00-a5bc-f7c80def7ec5] Live Migration failure: internal error: info migration reply was missing return status

  -- on target host /var/log/libvirt/qemu/instance-xxx.log: --
  /build/qemu-nffE1h/qemu-2.x/nbd.c:nbd_trip():L1310: From: 2416967680, Len: 65536, Size: 2361393152, Offset: 0
  /build/qemu-nffE1h/qemu-2.x/nbd.c:nbd_trip():L1311: requested operation past EOF--bad client?
  /build/qemu-nffE1h/qemu-2.x/nbd.c:nbd_trip():L1310: From: 3624927232, Len: 589824, Size: 2361393152, Offset: 0

  Seems that pre_live_migration task do not setup correctly target instance disk:
  -because glance image is not existant, it fallbacks to remote host copy method.
  -in this context, image.cache() is called without instance disk size parameter.
  -consequence is instance disk is not resized to the correct size and remain with the size of backing file, so the disk is too small, making failed libvirt live migration.

  
  Steps to reproduce
  ==================
  * Spawn qcow2 instance with glance image size << of flavor disk instance size
  * generate few user data in instance.
  * delete glance image.
  * run live block migration.

  Environment
  ===========
  Issue observed in Newton, still present in master.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1829000/+subscriptions