yahoo-eng-team team mailing list archive

[Launchpad Bug Tracker <1840159@xxxxxxxxxxxxxxxxxx>]

[Expired for OpenStack Compute (nova) because there has been no activity
for 60 days.]

** Changed in: nova
       Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1840159

Title:
  nova-grenade-(live-migration|multinode) intermittently fails with
  "Error monitoring migration: Timed out during operation: cannot
  acquire state change lock (held by
  remoteDispatchDomainMigratePerform3Params)"

Status in OpenStack Compute (nova):
  Expired

Bug description:
  Seen here:

  https://logs.opendev.org/21/655721/14/check/nova-grenade-live-
  migration/2ee634d/logs/subnode-2/screen-n-
  cpu.txt.gz?level=TRACE#_Aug_13_10_03_49_974378

  Aug 13 10:03:49.974378 ubuntu-bionic-limestone-regionone-0010083920 nova-compute[25863]: WARNING nova.virt.libvirt.driver [-] [instance: a1637e8b-6f2d-4127-9799-31cefb3f43a6] Error monitoring migration: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainMigratePerform3Params): libvirtError: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainMigratePerform3Params)
  Aug 13 10:03:49.974378 ubuntu-bionic-limestone-regionone-0010083920 nova-compute[25863]: ERROR nova.virt.libvirt.driver [instance: a1637e8b-6f2d-4127-9799-31cefb3f43a6] Traceback (most recent call last):
  Aug 13 10:03:49.974378 ubuntu-bionic-limestone-regionone-0010083920 nova-compute[25863]: ERROR nova.virt.libvirt.driver [instance: a1637e8b-6f2d-4127-9799-31cefb3f43a6]   File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 8052, in _live_migration
  Aug 13 10:03:49.974378 ubuntu-bionic-limestone-regionone-0010083920 nova-compute[25863]: ERROR nova.virt.libvirt.driver [instance: a1637e8b-6f2d-4127-9799-31cefb3f43a6]     finish_event, disk_paths)
  Aug 13 10:03:49.974378 ubuntu-bionic-limestone-regionone-0010083920 nova-compute[25863]: ERROR nova.virt.libvirt.driver [instance: a1637e8b-6f2d-4127-9799-31cefb3f43a6]   File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 7857, in _live_migration_monitor
  Aug 13 10:03:49.974378 ubuntu-bionic-limestone-regionone-0010083920 nova-compute[25863]: ERROR nova.virt.libvirt.driver [instance: a1637e8b-6f2d-4127-9799-31cefb3f43a6]     info = guest.get_job_info()
  Aug 13 10:03:49.974378 ubuntu-bionic-limestone-regionone-0010083920 nova-compute[25863]: ERROR nova.virt.libvirt.driver [instance: a1637e8b-6f2d-4127-9799-31cefb3f43a6]   File "/opt/stack/old/nova/nova/virt/libvirt/guest.py", line 709, in get_job_info
  Aug 13 10:03:49.974378 ubuntu-bionic-limestone-regionone-0010083920 nova-compute[25863]: ERROR nova.virt.libvirt.driver [instance: a1637e8b-6f2d-4127-9799-31cefb3f43a6]     stats = self._domain.jobStats()
  Aug 13 10:03:49.974378 ubuntu-bionic-limestone-regionone-0010083920 nova-compute[25863]: ERROR nova.virt.libvirt.driver [instance: a1637e8b-6f2d-4127-9799-31cefb3f43a6]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 190, in doit
  Aug 13 10:03:49.974378 ubuntu-bionic-limestone-regionone-0010083920 nova-compute[25863]: ERROR nova.virt.libvirt.driver [instance: a1637e8b-6f2d-4127-9799-31cefb3f43a6]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  Aug 13 10:03:49.974378 ubuntu-bionic-limestone-regionone-0010083920 nova-compute[25863]: ERROR nova.virt.libvirt.driver [instance: a1637e8b-6f2d-4127-9799-31cefb3f43a6]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 148, in proxy_call
  Aug 13 10:03:49.974378 ubuntu-bionic-limestone-regionone-0010083920 nova-compute[25863]: ERROR nova.virt.libvirt.driver [instance: a1637e8b-6f2d-4127-9799-31cefb3f43a6]     rv = execute(f, *args, **kwargs)
  Aug 13 10:03:49.974378 ubuntu-bionic-limestone-regionone-0010083920 nova-compute[25863]: ERROR nova.virt.libvirt.driver [instance: a1637e8b-6f2d-4127-9799-31cefb3f43a6]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 129, in execute
  Aug 13 10:03:49.974378 ubuntu-bionic-limestone-regionone-0010083920 nova-compute[25863]: ERROR nova.virt.libvirt.driver [instance: a1637e8b-6f2d-4127-9799-31cefb3f43a6]     six.reraise(c, e, tb)
  Aug 13 10:03:49.974378 ubuntu-bionic-limestone-regionone-0010083920 nova-compute[25863]: ERROR nova.virt.libvirt.driver [instance: a1637e8b-6f2d-4127-9799-31cefb3f43a6]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
  Aug 13 10:03:49.974378 ubuntu-bionic-limestone-regionone-0010083920 nova-compute[25863]: ERROR nova.virt.libvirt.driver [instance: a1637e8b-6f2d-4127-9799-31cefb3f43a6]     rv = meth(*args, **kwargs)
  Aug 13 10:03:49.974378 ubuntu-bionic-limestone-regionone-0010083920 nova-compute[25863]: ERROR nova.virt.libvirt.driver [instance: a1637e8b-6f2d-4127-9799-31cefb3f43a6]   File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 1403, in jobStats
  Aug 13 10:03:49.974378 ubuntu-bionic-limestone-regionone-0010083920 nova-compute[25863]: ERROR nova.virt.libvirt.driver [instance: a1637e8b-6f2d-4127-9799-31cefb3f43a6]     if ret is None: raise libvirtError ('virDomainGetJobStats() failed', dom=self)
  Aug 13 10:03:49.974378 ubuntu-bionic-limestone-regionone-0010083920 nova-compute[25863]: ERROR nova.virt.libvirt.driver [instance: a1637e8b-6f2d-4127-9799-31cefb3f43a6] libvirtError: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainMigratePerform3Params)
  Aug 13 10:03:49.974378 ubuntu-bionic-limestone-regionone-0010083920 nova-compute[25863]: ERROR nova.virt.libvirt.driver [instance: a1637e8b-6f2d-4127-9799-31cefb3f43a6] 
  Aug 13 10:03:49.977652 ubuntu-bionic-limestone-regionone-0010083920 nova-compute[25863]: ERROR nova.compute.manager [-] [instance: a1637e8b-6f2d-4127-9799-31cefb3f43a6] Live migration failed.: libvirtError: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainMigratePerform3Params)

  I thought we had another bug for this but I couldn't find one. This
  doesn't show up often, only 1 hit in 10 days:

  http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Error%20monitoring%20migration:%20Timed%20out%20during%20operation:%20cannot%20acquire%20state%20change%20lock%20(held%20by%20remoteDispatchDomainMigratePerform3Params)%5C%22%20AND%20tags:%5C%22screen-
  n-cpu.txt%5C%22&from=10d

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1840159/+subscriptions