yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2114710] [NEW] During the hot migration process of the virtual machine, if the destination host experiences an OS panic and the migration rollback fails, the virtual machine status will be displayed as error

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Mao Xinghao <2114710@xxxxxxxxxxxxxxxxxx>
Date: Mon, 16 Jun 2025 08:20:57 -0000
Reply-to: Bug 2114710 <2114710@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

1、Description：
    During the hot migration process of the virtual machine, if the destination host experiences an OS panic and the migration rollback fails, the virtual machine status will be displayed as error.
（The expected virtual machine status is running）

2、Steps to reproduce：
（1）.OpenStack triggers the hot migration of the virtual machine
（2）Check the nova compute log of the source node. After executing the live migration operation method and printing the log of 'About to invoke the migrate API', Execute the command 'echo c > /proc/sysrq-trigger' on the destination host to trigger its OS panic 
（3）Observe the results of thermal migration of the virtual machine

3、Environment：
Openstack version：Wallaby

4、Logs（Source node(Virtual machine migrates out of the node)'s nova-compute.log）：
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191] Traceback (most recent call last):
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/nova/virt/libvirt/driver.py", line 9969, in _live_migration
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     self._live_migration_monitor(context, instance, guest, dest,
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/nova/virt/libvirt/driver.py", line 9890, in _live_migration_monitor
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     recover_method(context, instance, dest, migrate_data)
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/nova/exception_wrapper.py", line 65, in wrapped
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     with excutils.save_and_reraise_exception():
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     self.force_reraise()
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     raise self.value
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/nova/exception_wrapper.py", line 63, in wrapped
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     return f(self, context, *args, **kw)
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/nova/compute/manager.py", line 213, in decorated_function
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     with excutils.save_and_reraise_exception():
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     self.force_reraise()
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     raise self.value
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/nova/compute/manager.py", line 203, in decorated_function
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     return function(self, context, *args, **kwargs)
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/nova/compute/manager.py", line 9205, in _rollback_live_migration
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     self.compute_rpcapi.drop_move_claim_at_destination(context,
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/nova/compute/rpcapi.py", line 1246, in drop_move_claim_at_destination
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     cctxt.call(ctxt, 'drop_move_claim_at_destination', instance=instance)
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/oslo_messaging/rpc/client.py", line 175, in call
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     self.transport._send(self.target, msg_ctxt, msg,
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/oslo_messaging/transport.py", line 123, in _send
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     return self._driver.send(target, ctxt, message,
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in send
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     return self._send(target, ctxt, message, wait_for_reply, timeout,
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 670, in _send
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     result = self._waiter.wait(msg_id, timeout,
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in wait
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     message = self.waiters.get(msg_id, timeout=timeout)
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 435, in get
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     raise oslo_messaging.MessagingTimeout(
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191] oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID aa8370ce52654ae9b516f4456293eb9b
2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]
2025-06-04 10:29:17.552 5174 ERROR nova.virt.libvirt.driver [-] [instance: 50377e38-2ffc-41db-a618-bfb044cdd191] Failed to cancel migration Requested operation is not valid: no job is active on the domain: libvirt.libvirtError: Requested operation is not valid: no job is active on the domain

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2114710

Title:
  During the hot migration process of the virtual machine, if the
  destination host experiences an OS panic and the migration rollback
  fails, the virtual machine status will be displayed as error

Status in OpenStack Compute (nova):
  New

Bug description:
  1、Description：
      During the hot migration process of the virtual machine, if the destination host experiences an OS panic and the migration rollback fails, the virtual machine status will be displayed as error.
  （The expected virtual machine status is running）

  2、Steps to reproduce：
  （1）.OpenStack triggers the hot migration of the virtual machine
  （2）Check the nova compute log of the source node. After executing the live migration operation method and printing the log of 'About to invoke the migrate API', Execute the command 'echo c > /proc/sysrq-trigger' on the destination host to trigger its OS panic 
  （3）Observe the results of thermal migration of the virtual machine

  3、Environment：
  Openstack version：Wallaby

  4、Logs（Source node(Virtual machine migrates out of the node)'s nova-compute.log）：
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191] Traceback (most recent call last):
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/nova/virt/libvirt/driver.py", line 9969, in _live_migration
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     self._live_migration_monitor(context, instance, guest, dest,
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/nova/virt/libvirt/driver.py", line 9890, in _live_migration_monitor
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     recover_method(context, instance, dest, migrate_data)
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/nova/exception_wrapper.py", line 65, in wrapped
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     with excutils.save_and_reraise_exception():
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     self.force_reraise()
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     raise self.value
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/nova/exception_wrapper.py", line 63, in wrapped
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     return f(self, context, *args, **kw)
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/nova/compute/manager.py", line 213, in decorated_function
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     with excutils.save_and_reraise_exception():
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     self.force_reraise()
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     raise self.value
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/nova/compute/manager.py", line 203, in decorated_function
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     return function(self, context, *args, **kwargs)
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/nova/compute/manager.py", line 9205, in _rollback_live_migration
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     self.compute_rpcapi.drop_move_claim_at_destination(context,
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/nova/compute/rpcapi.py", line 1246, in drop_move_claim_at_destination
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     cctxt.call(ctxt, 'drop_move_claim_at_destination', instance=instance)
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/oslo_messaging/rpc/client.py", line 175, in call
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     self.transport._send(self.target, msg_ctxt, msg,
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/oslo_messaging/transport.py", line 123, in _send
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     return self._driver.send(target, ctxt, message,
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in send
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     return self._send(target, ctxt, message, wait_for_reply, timeout,
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 670, in _send
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     result = self._waiter.wait(msg_id, timeout,
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in wait
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     message = self.waiters.get(msg_id, timeout=timeout)
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]   File "/usr/lib/python3.11/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 435, in get
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]     raise oslo_messaging.MessagingTimeout(
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191] oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID aa8370ce52654ae9b516f4456293eb9b
  2025-06-04 10:29:17.233 5174 ERROR nova.virt.libvirt.driver [instance: 50377e38-2ffc-41db-a618-bfb044cdd191]
  2025-06-04 10:29:17.552 5174 ERROR nova.virt.libvirt.driver [-] [instance: 50377e38-2ffc-41db-a618-bfb044cdd191] Failed to cancel migration Requested operation is not valid: no job is active on the domain: libvirt.libvirtError: Requested operation is not valid: no job is active on the domain

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2114710/+subscriptions