yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1763410] [NEW] swap volume intermittently fails with "libvirtError: block copy still active: domain has active block job"

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Thu, 12 Apr 2018 15:08:41 -0000
Reply-to: Bug 1763410 <1763410@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Seeing this in some CI runs:

http://logs.openstack.org/67/560467/1/check/nova-
next/9ceb996/logs/screen-n-cpu.txt.gz?level=TRACE#_Apr_11_16_19_23_843152

Apr 11 16:19:23.843152 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver [None req-4ca5f968-bbce-4826-b16c-cc02d63c07c2 service nova] Failure rebasing volume /dev/sda on vdb.: libvirtError: internal error: unable to execute QEMU command 'block-job-complete': The active block job 'drive-virtio-disk1' cannot be completed
Apr 11 16:19:23.843399 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver Traceback (most recent call last):
Apr 11 16:19:23.843646 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1529, in _swap_volume
Apr 11 16:19:23.843882 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver     dev.abort_job(pivot=True)
Apr 11 16:19:23.844165 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 757, in abort_job
Apr 11 16:19:23.844416 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver     self._guest._domain.blockJobAbort(self._disk, flags=flags)
Apr 11 16:19:23.844666 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
Apr 11 16:19:23.844905 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver     result = proxy_call(self._autowrap, f, *args, **kwargs)
Apr 11 16:19:23.845168 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
Apr 11 16:19:23.845412 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver     rv = execute(f, *args, **kwargs)
Apr 11 16:19:23.845659 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
Apr 11 16:19:23.845909 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver     six.reraise(c, e, tb)
Apr 11 16:19:23.846168 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
Apr 11 16:19:23.846445 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver     rv = meth(*args, **kwargs)
Apr 11 16:19:23.846707 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 766, in blockJobAbort
Apr 11 16:19:23.846960 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver     if ret == -1: raise libvirtError ('virDomainBlockJobAbort() failed', dom=self)
Apr 11 16:19:23.878201 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver libvirtError: internal error: unable to execute QEMU command 'block-job-complete': The active block job 'drive-virtio-disk1' cannot be completed
Apr 11 16:19:23.878506 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver 

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Failed%20to%20swap%20volume%5C%22%20AND%20message%3A%5C%22libvirtError%3A%20block%20copy%20still%20active%3A%20domain%20has%20active%20block%20job%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d

7 hits in 7 days, check and gate, all failures.

** Affects: nova
     Importance: Medium
         Status: Confirmed


** Tags: libvirt swap-volume

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1763410

Title:
  swap volume intermittently fails with "libvirtError: block copy still
  active: domain has active block job"

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  Seeing this in some CI runs:

  http://logs.openstack.org/67/560467/1/check/nova-
  next/9ceb996/logs/screen-n-cpu.txt.gz?level=TRACE#_Apr_11_16_19_23_843152

  Apr 11 16:19:23.843152 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver [None req-4ca5f968-bbce-4826-b16c-cc02d63c07c2 service nova] Failure rebasing volume /dev/sda on vdb.: libvirtError: internal error: unable to execute QEMU command 'block-job-complete': The active block job 'drive-virtio-disk1' cannot be completed
  Apr 11 16:19:23.843399 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver Traceback (most recent call last):
  Apr 11 16:19:23.843646 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1529, in _swap_volume
  Apr 11 16:19:23.843882 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver     dev.abort_job(pivot=True)
  Apr 11 16:19:23.844165 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 757, in abort_job
  Apr 11 16:19:23.844416 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver     self._guest._domain.blockJobAbort(self._disk, flags=flags)
  Apr 11 16:19:23.844666 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
  Apr 11 16:19:23.844905 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver     result = proxy_call(self._autowrap, f, *args, **kwargs)
  Apr 11 16:19:23.845168 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
  Apr 11 16:19:23.845412 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver     rv = execute(f, *args, **kwargs)
  Apr 11 16:19:23.845659 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
  Apr 11 16:19:23.845909 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver     six.reraise(c, e, tb)
  Apr 11 16:19:23.846168 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
  Apr 11 16:19:23.846445 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver     rv = meth(*args, **kwargs)
  Apr 11 16:19:23.846707 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 766, in blockJobAbort
  Apr 11 16:19:23.846960 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver     if ret == -1: raise libvirtError ('virDomainBlockJobAbort() failed', dom=self)
  Apr 11 16:19:23.878201 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver libvirtError: internal error: unable to execute QEMU command 'block-job-complete': The active block job 'drive-virtio-disk1' cannot be completed
  Apr 11 16:19:23.878506 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver 

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Failed%20to%20swap%20volume%5C%22%20AND%20message%3A%5C%22libvirtError%3A%20block%20copy%20still%20active%3A%20domain%20has%20active%20block%20job%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d

  7 hits in 7 days, check and gate, all failures.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1763410/+subscriptions
Follow ups

[Bug 1763410] Re: swap volume intermittently fails with "libvirtError: block copy still active: domain has active block job"
From: Launchpad Bug Tracker, 2020-05-25