group.of.nepali.translators team mailing list archive

[Launchpad Bug Tracker <1681839@xxxxxxxxxxxxxxxxxx>]

This bug was fixed in the package libvirt - 1.3.1-1ubuntu10.29

---------------
libvirt (1.3.1-1ubuntu10.29) xenial; urgency=medium

  * debian/patches/lp1681839-*.patch: Fix block commit timeout
    races, and ensure that once commit has reached 100%, timeouts
    no longer apply. (LP: #1681839)

 -- Matthew Ruffell <matthew.ruffell@xxxxxxxxxxxxx>  Thu, 31 Oct 2019
10:52:41 +1300

** Changed in: libvirt (Ubuntu Xenial)
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of नेपाली
भाषा समायोजकहरुको समूह, which is subscribed to Xenial.
Matching subscriptions: Ubuntu 16.04 Bugs
https://bugs.launchpad.net/bugs/1681839

Title:
  libvirt: blockcommit fails - disk not ready for pivot yet

Status in libvirt package in Ubuntu:
  Fix Released
Status in libvirt source package in Xenial:
  Fix Released
Status in libvirt source package in Artful:
  Fix Released
Status in libvirt source package in Bionic:
  Fix Released

Bug description:
  [Impact]

  On xenial, if you manually invoke blockcommit through virsh in
  libvirt, the command immediately fails with blockcommit supposedly
  being 100%, and that the disk is not ready for pivot yet:

  root@xenial-apparmor:~# virsh blockcommit snapvm vda --active --verbose --pivot --wait
  Block commit: [100 %]
  error: failed to pivot job for disk vda
  error: block copy still active: disk 'vda' not ready for pivot yet

  However, if you look at the status of the active blockjob, we see that
  the blockcommit is still active in the background:

  root@xenial-apparmor:~# virsh blockjob snapvm vda --info
  Active Block Commit: [0 %]

  root@xenial-apparmor:~# virsh blockjob snapvm vda --info
  Active Block Commit: [2 %]

  root@xenial-apparmor:~# virsh blockjob snapvm vda --info
  Active Block Commit: [6 %]

  This happens until it reaches 100%, where it gets stuck. To un-stick
  things, you must then manually --abort the blockjob.

  root@xenial-apparmor:~# virsh blockjob snapvm vda --info
  Active Block Commit: [100 %]

  This happens in VMs which are experiencing load, and is caused by a
  race condition in libvirt. Users are not able to commit their
  snapshots to disk.

  [Test Case]

  Credit goes to Fabio Martins, who determined how to reproduce this
  issue.

  On a Ubuntu 16.04 host with libvirt 1.3.1-1ubuntu10.27:

  1) Create a VG and define a LVM pool:

  root@xenial-apparmor:~# cat lvmpool.xml
  <pool type="logical">
  <name>LVMpool_vg</name>
  <source>
  <device path="/dev/sdb"/>
  </source>
  <target>
  <path>/dev/LVMpool_vg</path>
  </target>
  </pool>

  # virsh pool-define lvmpool.xml
  # virsh pool-start LVMpool_vg
  # virsh pool-autostart LVMpool_vg

  2) Create a config file to use as a cdrom device with the new VM (will
  be created in next steps), just to inject a password with cloud-init:

  # cat > config <<EOF
  > #cloud-config
  > password: passw0rd
  > chpasswd: { expire: False }
  > ssh_pwauth: True
  > EOF

  # apt install cloud-image-utils

  # cloud-localds config.img config

  # mv config.img /var/lib/libvirt/images/
  # chown libvirt-qemu:kvm /var/lib/libvirt/images/config.img
  # chmod 664 /var/lib/libvirt/images/config.img 

  3) Create one VM using this pool:

  # virt-install --connect=qemu:///system --name snapvm --ram 2048
  --vcpus=1 --os-type=linux --disk pool=LVMpool_vg,size=15,bus=virtio
  --disk /var/lib/libvirt/images/config.img,device=cdrom --network
  network=kvm-br0 --graphics none --import --noautoconsole

  4) Stop the VM

  # virsh destroy snapvm

  5) Download a Ubuntu cloud image, convert to raw and restore it into
  the LV used as a disk by our VM:

  # wget https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img
  # qemu-img convert ./bionic-server-cloudimg-amd64.img ./bionic-server-cloudimg-amd64.raw
  # dd if=./bionic-server-cloudimg-amd64.raw of=/dev/LVMpool_vg/snapvm bs=8M conv=sparse

  6) Start the VM and connect to it in another window

  # virsh start snapvm

  7) Check that the VM is using the LV as the disk:

  root@xenial-apparmor:~# virsh domblklist snapvm
  Target Source
  ------------------------------------------------
  vda /dev/LVMpool_vg/snapvm
  hda /var/lib/libvirt/images/config.img

  8) Create a snapshot and check that the new domblklist points to the
  snapshot file:

  # virsh snapshot-create-as --domain snapvm --diskspec
  vda,file=/var/lib/libvirt/images/xenial-snapvm.qcow2,snapshot=external
  --disk-only --atomic

  root@xenial-apparmor:~# virsh domblklist snapvm
  Target Source
  ------------------------------------------------
  vda /var/lib/libvirt/images/xenial-snapvm.qcow2
  hda /var/lib/libvirt/images/config.img

  9) Connect to your VM and start an I/O intensive job. In this case I'm
  starting a 'dd' writing zeroes to a file until it gets to 10GBs:

  ubuntu@ubuntu:~$ dd if=/dev/zero of=file.txt count=1024 bs=10240000

  10) Back to the host, monitor the snapshot file and let it grow until
  at list a bit more than 1GB, as in the example below (where we can see
  the file with 3.9G):

  root@xenial-apparmor:~# ls -lh /var/lib/libvirt/images/
  total 5.2G
  -rw-rw-r-- 1 libvirt-qemu kvm 329M Sep 3 03:18 bionic-server-cloudimg-amd64.img
  -rw-r--r-- 1 root root 10G Sep 3 03:28 bionic-server-cloudimg-amd64.raw
  -rw-rw-r-- 1 libvirt-qemu kvm 366K Sep 3 03:19 config.img
  -rw------- 1 libvirt-qemu kvm 3.9G Sep 3 04:41 xenial-snapvm.qcow2

  11) Start a blockcommit job with --active --verbose --pivot --wait and
  we'll hit the error when the job gets to 100%:

  root@xenial-apparmor:~# virsh blockcommit snapvm vda --active --verbose --pivot --wait
  Block commit: [100 %]
  error: failed to pivot job for disk vda
  error: block copy still active: disk 'vda' not ready for pivot yet

  12) The blkjob will continue in the background, and status increments:

  root@xenial-apparmor:~# virsh blockjob snapvm vda --info
  Active Block Commit: [0 %]

  root@xenial-apparmor:~# virsh blockjob snapvm vda --info
  Active Block Commit: [2 %]

  root@xenial-apparmor:~# virsh blockjob snapvm vda --info
  Active Block Commit: [6 %]

  13) The blkjob will show it is stuck at 100% until you --abort the
  blkjob:

  root@xenial-apparmor:~# virsh blockjob snapvm vda --info
  Active Block Commit: [100 %]

  I have created a test package with the commits needed to solve the
  problem, and it is available here:

  https://launchpad.net/~mruffell/+archive/ubuntu/sf242822-test

  What should happen:

  If you install the test libvirt-bin and libvirt0 packages from the
  above ppa, and run through the test case, when blockcommit is invoked,
  it will not fail immediately, and instead, will continue on until it
  reaches 100%. Once 100% is reached, the blockjob will complete
  successfully.

  [Regression Potential]

  While there are four commits which are required to fix this issue, all
  of them are fairly minor and only modify the way the current status
  percentage is counted, and how states are being changed, upon reaching
  100% blockcommit. All changes are localised to one file.

  Most of the commits are limited to blockcommit, and in event of
  regression, only blockcommit and by extension, some blockjobs would be
  impacted.

  The commits have been present in upstream for a long time, have been
  well tested by the community, and are from a release of libvirt with
  very small delta to the one in xenial (1.3.2 versus 1.3.1 in xenial),
  I believe there is little risk of regression.

  [Other Info]

  The following commits were identified in the upstream bug:
  https://bugzilla.redhat.com/show_bug.cgi?id=1197592

  which are also listed in comment #6.

  commit 86c4df83b913dd73b79caeed2038291374384dc5
  Author: Michael Chapman <mike@xxxxxxxxxxxxxxxxx>
  Date:   Wed Jan 27 13:24:54 2016 +1100
  Subject: virsh: improve waiting for block job readiness

  commit 8fa216bbb40df33e7fce5d727aa3dc334480878a
  Author: Michael Chapman <mike@xxxxxxxxxxxxxxxxx>
  Date:   Wed Jan 27 13:24:53 2016 +1100
  Subject: virsh: ensure SIGINT action is reset on all errors

  commit 15dee2ef24f2f19f6dcd30d997b81c8a14582361
  Author: Michael Chapman <mike@xxxxxxxxxxxxxxxxx>
  Date:   Wed Jan 27 13:24:52 2016 +1100
  Subject: virsh: be consistent with style of loop exit

  commit 704dfd6b0fafe7eafca93a03793389239f8ab869
  Author: Michael Chapman <mike@xxxxxxxxxxxxxxxxx>
  Date:   Wed Jan 27 13:24:51 2016 +1100
  Subject: virsh: avoid unnecessary progress updates

  These fix the problem, and were introduced in libvirt 1.3.2 upstream.
  All commits are clean cherry picks, and the code is still present in
  B, D, E and F.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1681839/+subscriptions