yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2052473] Re: Live migration post-copy not working as Expected

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: keerthivasan <2052473@xxxxxxxxxxxxxxxxxx>
Date: Tue, 06 Feb 2024 19:46:08 -0000
Reply-to: Bug 2052473 <2052473@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
** Changed in: nova
       Status: Invalid => New

** Changed in: nova
     Assignee: (unassigned) => keerthivasan (keerthivassan86)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2052473

Title:
  Live migration post-copy not working as Expected

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  I am trying to enable live migration feature post copy using below config, seeing post-copy is not supported error

  
  block_migration_flag = VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_NON_SHARED_INC
  cpu_mode = custom
  cpu_model_extra_flags = -ds,-acpi,+ss,-ht,-tm,-pbe,-dtes64,-monitor,-ds_cpl,+vmx,-smx,-est,-tm2,-xtpr,+pdcm,-dca,+tsc_adjust,-intel-pt,+md-clear,+stibp,+ssbd,+pdpe1gb,-invtsc,-hle,-rtm,-mpx,-xsavec,-xgetbv1
  cpu_models = Skylake-Client-IBRS
  live_migration_bandwidth = 900
  live_migration_downtime = 100
  live_migration_flag = VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE
  live_migration_permit_post_copy = True
  live_migration_timeout_action=force_complete


  Steps to reproduce
  ==================

  KVM hypervisor

  Using Openstack Antelope base version

  qemu-system-x86_64 --version
  QEMU emulator version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.16)
  Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers

  libvirtd (libvirt) 8.0.0

  #Create general vm once config is set
  #perform live migration either with block-migration or without

  After Pre-migration phase, able  to see migration got trigger
  successfully & while copying memory , seeing postcopy is not supported


  Expected result
  ===============
  Migration should be sucessfull

  Actual result
  =============

  compute.log

  Feb 05 22:16:02 cdc-appblx095-37 nova-compute[1156821]: 2024-02-05 22:16:02.988 1156821 INFO nova.virt.libvirt.migration [None req-f6650a9f-9465-40ca-9981-fff470751fc7 4807f132b7bb47bbabbe50de9bd974c8 b61fc56101024f498d4d95e863c7333f - - default default] [instance: 31fcf3ba-c0b1-4c74-afdd-685ba45a11f0] Increasing downtime to 10 ms after 0 sec elapsed time
  Feb 05 22:16:03 cdc-appblx095-37 nova-compute[1156821]: 2024-02-05 22:16:03.069 1156821 INFO nova.virt.libvirt.driver [None req-f6650a9f-9465-40ca-9981-fff470751fc7 4807f132b7bb47bbabbe50de9bd974c8 b61fc56101024f498d4d95e863c7333f - - default default] [instance: 31fcf3ba-c0b1-4c74-afdd-685ba45a11f0] Migration running for 0 secs, memory 100% remaining (bytes processed=0, remaining=0, total=0); disk 100% remaining (bytes processed=0, remaining=0, total=0).
  Feb 05 22:16:03 cdc-appblx095-37 nova-compute[1156821]: 2024-02-05 22:16:03.571 1156821 DEBUG nova.virt.libvirt.migration [None req-f6650a9f-9465-40ca-9981-fff470751fc7 4807f132b7bb47bbabbe50de9bd974c8 b61fc56101024f498d4d95e863c7333f - - default default] [instance: 31fcf3ba-c0b1-4c74-afdd-685ba45a11f0] Current 10 elapsed 1 steps [(0, 10), (960, 19), (1920, 28), (2880, 37), (3840, 46), (4800, 55), (5760, 64), (6720, 73), (7680, 82), (8640, 91), (9600, 100)] update_downtime /openstack/venvs/nova-27.4.0/lib/python3.10/site-packages/nova/virt/libvirt/migration.py:512
  Feb 05 22:16:03 cdc-appblx095-37 nova-compute[1156821]: 2024-02-05 22:16:03.572 1156821 DEBUG nova.virt.libvirt.migration [None req-f6650a9f-9465-40ca-9981-fff470751fc7 4807f132b7bb47bbabbe50de9bd974c8 b61fc56101024f498d4d95e863c7333f - - default default] [instance: 31fcf3ba-c0b1-4c74-afdd-685ba45a11f0] Downtime does not need to change update_downtime /openstack/venvs/nova-27.4.0/lib/python3.10/site-packages/nova/virt/libvirt/migration.py:525
  Feb 05 22:16:04 cdc-appblx095-37 nova-compute[1156821]: 2024-02-05 22:16:04.074 1156821 DEBUG nova.virt.libvirt.migration [None req-f6650a9f-9465-40ca-9981-fff470751fc7 4807f132b7bb47bbabbe50de9bd974c8 b61fc56101024f498d4d95e863c7333f - - default default] [instance: 31fcf3ba-c0b1-4c74-afdd-685ba45a11f0] Current 10 elapsed 1 steps [(0, 10), (960, 19), (1920, 28), (2880, 37), (3840, 46), (4800, 55), (5760, 64), (6720, 73), (7680, 82), (8640, 91), (9600, 100)] update_downtime /openstack/venvs/nova-27.4.0/lib/python3.10/site-packages/nova/virt/libvirt/migration.py:512
  Feb 05 22:16:04 cdc-appblx095-37 nova-compute[1156821]: 2024-02-05 22:16:04.075 1156821 DEBUG nova.virt.libvirt.migration [None req-f6650a9f-9465-40ca-9981-fff470751fc7 4807f132b7bb47bbabbe50de9bd974c8 b61fc56101024f498d4d95e863c7333f - - default default] [instance: 31fcf3ba-c0b1-4c74-afdd-685ba45a11f0] Downtime does not need to change update_downtime /openstack/venvs/nova-27.4.0/lib/python3.10/site-packages/nova/virt/libvirt/migration.py:525
  Feb 05 22:16:04 cdc-appblx095-37 nova-compute[1156821]: 2024-02-05 22:16:04.577 1156821 DEBUG nova.virt.libvirt.migration [None req-f6650a9f-9465-40ca-9981-fff470751fc7 4807f132b7bb47bbabbe50de9bd974c8 b61fc56101024f498d4d95e863c7333f - - default default] [instance: 31fcf3ba-c0b1-4c74-afdd-685ba45a11f0] Current 10 elapsed 2 steps [(0, 10), (960, 19), (1920, 28), (2880, 37), (3840, 46), (4800, 55), (5760, 64), (6720, 73), (7680, 82), (8640, 91), (9600, 100)] update_downtime /openstack/venvs/nova-27.4.0/lib/python3.10/site-packages/nova/virt/libvirt/migration.py:512
  Feb 05 22:16:04 cdc-appblx095-37 nova-compute[1156821]: 2024-02-05 22:16:04.577 1156821 DEBUG nova.virt.libvirt.migration [None req-f6650a9f-9465-40ca-9981-fff470751fc7 4807f132b7bb47bbabbe50de9bd974c8 b61fc56101024f498d4d95e863c7333f - - default default] [instance: 31fcf3ba-c0b1-4c74-afdd-685ba45a11f0] Downtime does not need to change update_downtime /openstack/venvs/nova-27.4.0/lib/python3.10/site-packages/nova/virt/libvirt/migration.py:525
  Feb 05 22:16:04 cdc-appblx095-37 nova-compute[1156821]: 2024-02-05 22:16:04.638 1156821 ERROR nova.virt.libvirt.driver [None req-f6650a9f-9465-40ca-9981-fff470751fc7 4807f132b7bb47bbabbe50de9bd974c8 b61fc56101024f498d4d95e863c7333f - - default default] [instance: 31fcf3ba-c0b1-4c74-afdd-685ba45a11f0] Live Migration failure: internal error: unable to execute QEMU command 'migrate-set-capabilities': Postcopy is not supported: libvirt.libvirtError: internal error: unable to execute QEMU command 'migrate-set-capabilities': Postcopy is not supported

  
  libvirtd.log
  -------

  2024-02-05 21:27:56.254+0000: 2177204: debug : qemuMonitorSetMigrationCapabilities:3689 : mon:0x7f8d68080460 vm:0x7f8d6804b930 fd:82
  2024-02-05 21:27:56.254+0000: 2177204: info : qemuMonitorSend:914 : QEMU_MONITOR_SEND_MSG: mon=0x7f8d68080460 msg={"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"xbzrle","state":false},{"capability":"auto-converge","state":false},{"capability":"rdma-pin-all","state":false},{"capability":"postcopy-ram","state":true},{"capability":"compress","state":false},{"capability":"pause-before-switchover","state":false},{"capability":"late-block-activate","state":true},{"capability":"multifd","state":false},{"capability":"dirty-bitmaps","state":false},{"capability":"return-path","state":true}]},"id":"libvirt-402"}^M
   fd=-1
  2024-02-05 21:27:56.254+0000: 16270: info : qemuMonitorIOWrite:402 : QEMU_MONITOR_IO_WRITE: mon=0x7f8d68080460 buf={"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"xbzrle","state":false},{"capability":"auto-converge","state":false},{"capability":"rdma-pin-all","state":false},{"capability":"postcopy-ram","state":true},{"capability":"compress","state":false},{"capability":"pause-before-switchover","state":false},{"capability":"late-block-activate","state":true},{"capability":"multifd","state":false},{"capability":"dirty-bitmaps","state":false},{"capability":"return-path","state":true}]},"id":"libvirt-402"}^M
   len=531 ret=531 errno=0
  2024-02-05 21:27:56.258+0000: 16270: debug : qemuMonitorJSONIOProcessLine:220 : Line [{"id": "libvirt-402", "error": {"class": "GenericError", "desc": "Postcopy is not supported"}}]
  2024-02-05 21:27:56.258+0000: 16270: info : qemuMonitorJSONIOProcessLine:239 : QEMU_MONITOR_RECV_REPLY: mon=0x7f8d68080460 reply={"id": "libvirt-402", "error": {"class": "GenericError", "desc": "Postcopy is not supported"}}
  2024-02-05 21:27:56.258+0000: 2177204: debug : qemuMonitorJSONCheckErrorFull:385 : unable to execute QEMU command {"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"xbzrle","state":false},{"capability":"auto-converge","state":false},{"capability":"rdma-pin-all","state":false},{"capability":"postcopy-ram","state":true},{"capability":"compress","state":false},{"capability":"pause-before-switchover","state":false},{"capability":"late-block-activate","state":true},{"capability":"multifd","state":false},{"capability":"dirty-bitmaps","state":false},{"capability":"return-path","state":true}]},"id":"libvirt-402"}: {"id":"libvirt-402","error":{"class":"GenericError","desc":"Postcopy is not supported"}}
  2024-02-05 21:27:56.258+0000: 2177204: error : qemuMonitorJSONCheckErrorFull:397 : internal error: unable to execute QEMU command 'migrate-set-capabilities': Postcopy is not supported

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2052473/+subscriptions
References

[Bug 2052473] [NEW] Live migration post-copy not working as Expected
From: keerthivasan, 2024-02-05