yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1921306] [NEW] When Instance is paused state, live migration failure.

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Xinxin Shen <1921306@xxxxxxxxxxxxxxxxxx>
Date: Thu, 25 Mar 2021 02:26:47 -0000
Reply-to: Bug 1921306 <1921306@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

When the virtual machine is in the paused state and continuously live
migrated twice, the Instance state changes to ERROR.

2021-03-22 19:34:38.388 6 DEBUG nova.virt.libvirt.guest [- req-None - - - - -] Failed to get job stats: Unable to read from monitor: Connection reset by peer get_job_info /var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/guest.py:767
2021-03-22 19:34:38.389 6 WARNING nova.virt.libvirt.driver [- req-None - - - - -] [instance: d1d2af1f-e973-438c-a6b6-b628091d3596] Error monitoring migration: Unable to read from monitor: Connection reset by peer: libvirtError: Unable to read from monitor: Connection reset by peer
2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596] Traceback (most recent call last):
2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7793, in _live_migration
2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     finish_event, disk_paths)
2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7593, in _live_migration_monitor
2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     info = guest.get_job_info()
2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 751, in get_job_info
2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     stats = self._domain.jobStats()
2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     rv = execute(f, *args, **kwargs)
2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     six.reraise(c, e, tb)
2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     rv = meth(*args, **kwargs)
2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1433, in jobStats
2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     if ret is None: raise libvirtError ('virDomainGetJobStats() failed', dom=self)
2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596] libvirtError: Unable to read from monitor: Connection reset by peer
2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]
2021-03-22 19:34:38.394 6 DEBUG nova.virt.libvirt.driver [- req-None - - - - -] [instance: d1d2af1f-e973-438c-a6b6-b628091d3596] Live migration monitoring is all done _live_migration /var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:7800
2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [- req-None - - - - -] [instance: d1d2af1f-e973-438c-a6b6-b628091d3596] Live migration failed.: libvirtError: Unable to read from monitor: Connection reset by peer
2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596] Traceback (most recent call last):
2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 6510, in _do_live_migration
2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     block_migration, migrate_data)
2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7279, in live_migration
2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     migrate_data)
2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7793, in _live_migration
2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     finish_event, disk_paths)
2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7593, in _live_migration_monitor
2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     info = guest.get_job_info()
2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 751, in get_job_info
2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     stats = self._domain.jobStats()
2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     rv = execute(f, *args, **kwargs)
2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     six.reraise(c, e, tb)
2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     rv = meth(*args, **kwargs)
2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1433, in jobStats
2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     if ret is None: raise libvirtError ('virDomainGetJobStats() failed', dom=self)
2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596] libvirtError: Unable to read from monitor: Connection reset by peer
2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]
2021-03-22 19:34:38.637 6 ERROR nova.inspur.virt.inspur.driver [- req-None - - - - -] [instance: d1d2af1f-e973-438c-a6b6-b628091d3596] Live Migration failure: operation failed: domain is not running: libvirtError: operation failed: domain is not running

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1921306

Title:
  When Instance is paused state,live migration failure.

Status in OpenStack Compute (nova):
  New

Bug description:
  When the virtual machine is in the paused state and continuously live
  migrated twice, the Instance state changes to ERROR.

  2021-03-22 19:34:38.388 6 DEBUG nova.virt.libvirt.guest [- req-None - - - - -] Failed to get job stats: Unable to read from monitor: Connection reset by peer get_job_info /var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/guest.py:767
  2021-03-22 19:34:38.389 6 WARNING nova.virt.libvirt.driver [- req-None - - - - -] [instance: d1d2af1f-e973-438c-a6b6-b628091d3596] Error monitoring migration: Unable to read from monitor: Connection reset by peer: libvirtError: Unable to read from monitor: Connection reset by peer
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596] Traceback (most recent call last):
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7793, in _live_migration
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     finish_event, disk_paths)
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7593, in _live_migration_monitor
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     info = guest.get_job_info()
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 751, in get_job_info
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     stats = self._domain.jobStats()
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     rv = execute(f, *args, **kwargs)
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     six.reraise(c, e, tb)
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     rv = meth(*args, **kwargs)
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1433, in jobStats
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     if ret is None: raise libvirtError ('virDomainGetJobStats() failed', dom=self)
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596] libvirtError: Unable to read from monitor: Connection reset by peer
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]
  2021-03-22 19:34:38.394 6 DEBUG nova.virt.libvirt.driver [- req-None - - - - -] [instance: d1d2af1f-e973-438c-a6b6-b628091d3596] Live migration monitoring is all done _live_migration /var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:7800
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [- req-None - - - - -] [instance: d1d2af1f-e973-438c-a6b6-b628091d3596] Live migration failed.: libvirtError: Unable to read from monitor: Connection reset by peer
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596] Traceback (most recent call last):
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 6510, in _do_live_migration
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     block_migration, migrate_data)
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7279, in live_migration
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     migrate_data)
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7793, in _live_migration
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     finish_event, disk_paths)
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7593, in _live_migration_monitor
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     info = guest.get_job_info()
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 751, in get_job_info
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     stats = self._domain.jobStats()
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     rv = execute(f, *args, **kwargs)
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     six.reraise(c, e, tb)
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     rv = meth(*args, **kwargs)
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1433, in jobStats
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]     if ret is None: raise libvirtError ('virDomainGetJobStats() failed', dom=self)
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596] libvirtError: Unable to read from monitor: Connection reset by peer
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: d1d2af1f-e973-438c-a6b6-b628091d3596]
  2021-03-22 19:34:38.637 6 ERROR nova.inspur.virt.inspur.driver [- req-None - - - - -] [instance: d1d2af1f-e973-438c-a6b6-b628091d3596] Live Migration failure: operation failed: domain is not running: libvirtError: operation failed: domain is not running

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1921306/+subscriptions
Follow ups

[Bug 1921306] Re: When Instance is paused state, live migration failure.
From: Lee Yarwood, 2021-07-09
[Bug 1921306] Re: When Instance is paused state, live migration failure.
From: Pierre Thévenet, 2021-06-21
[Bug 1921306] Re: When Instance is paused state, live migration failure.
From: Launchpad Bug Tracker, 2021-05-25