group.of.nepali.translators team mailing list archive
-
group.of.nepali.translators team
-
Mailing list archive
-
Message #16085
[Bug 1571209] Re: Sockfile check retries too short for a busy system boot
To check on the systemd case I picked a rather slow system and spawned 30 guests.
I set all those to autostart and then shut them down.
Starting them sequentially as well as concurrently takes about ~20
seconds.
/var/run/libvirt/libvirt-sock is gone after the service is stopped.
With the following script this can be tested:
$ cat test-restart.sh
#!/bin/bash
set -uxe
date
sudo systemctl start libvirtd
date
# right after start (1-2 seconds usually) check status of sockets and guests
while /bin/true; do
ls -laF /var/run/libvirt/libvirt-sock
virsh list
sudo systemctl status libvirtd --no-pager
date
sleep 2s
done
You will see that the "virsh list" is kind of blocking until other internal tasks are done.
That pretty much ends up after the ~20 seconds that I counted until it goes on.
All guests are started then and things are good.
But the socket is available right after start, yet if you do a request you are enqueued and have to wait - which is fine.
Output:
$ ./test-restart.sh
+ date
Mi 6. Sep 04:40:34 EDT 2017
+ sudo systemctl start libvirtd
+ date
Mi 6. Sep 04:40:34 EDT 2017
+ /bin/true
+ ls -laF /var/run/libvirt/libvirt-sock
srwxrwx--- 1 root libvirtd 0 Sep 6 04:40 /var/run/libvirt/libvirt-sock=
+ virsh list
Id Name State
----------------------------------------------------
1 artful-test30 running
2 artful-test9 running
3 artful-test8 running
4 artful-test23 running
5 artful-test13 running
6 artful-test19 running
7 artful-test11 running
8 artful-test15 running
9 artful-test10 running
10 artful-test17 running
11 artful-test5 running
12 artful-test24 running
13 artful-test26 running
14 artful-test20 running
15 artful-test2 running
16 artful-test29 running
17 artful-test22 running
18 artful-test27 running
19 artful-test14 running
20 artful-test6 running
21 artful-test25 running
22 artful-test12 running
23 artful-test3 running
24 artful-test21 running
25 artful-test7 running
26 artful-test16 running
27 artful-test28 running
28 artful-test18 running
29 artful-test1 running
30 artful-test4 running
+ sudo systemctl status libvirtd --no-pager
● libvirtd.service - Virtualization daemon
Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2017-09-06 04:40:34 EDT; 23s ago
Docs: man:libvirtd(8)
http://libvirt.org
Main PID: 171137 (libvirtd)
Tasks: 303 (limit: 32768)
Memory: 93.7M
CPU: 15.383s
CGroup: /system.slice/libvirtd.service
├─ 3848 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_…er
├─ 3849 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_…er
├─171137 /usr/sbin/libvirtd
├─171530 qemu-system-s390x -enable-kvm -name guest=artful-test30,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/…on
├─171640 qemu-system-s390x -enable-kvm -name guest=artful-test9,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/v…on
├─171789 qemu-system-s390x -enable-kvm -name guest=artful-test8,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/v…on
├─171897 qemu-system-s390x -enable-kvm -name guest=artful-test23,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/…on
├─171970 qemu-system-s390x -enable-kvm -name guest=artful-test13,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/…on
├─172041 qemu-system-s390x -enable-kvm -name guest=artful-test19,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/…on
├─172116 qemu-system-s390x -enable-kvm -name guest=artful-test11,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/…on
├─172196 qemu-system-s390x -enable-kvm -name guest=artful-test15,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/…on
├─172273 qemu-system-s390x -enable-kvm -name guest=artful-test10,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/…on
├─172348 qemu-system-s390x -enable-kvm -name guest=artful-test17,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/…on
├─172429 qemu-system-s390x -enable-kvm -name guest=artful-test5,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/v…on
├─172506 qemu-system-s390x -enable-kvm -name guest=artful-test24,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/…on
├─172588 qemu-system-s390x -enable-kvm -name guest=artful-test26,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/…on
├─172671 qemu-system-s390x -enable-kvm -name guest=artful-test20,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/…on
├─172748 qemu-system-s390x -enable-kvm -name guest=artful-test2,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/v…on
├─172827 qemu-system-s390x -enable-kvm -name guest=artful-test29,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/…on
├─172899 qemu-system-s390x -enable-kvm -name guest=artful-test22,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/…on
├─172977 qemu-system-s390x -enable-kvm -name guest=artful-test27,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/…on
├─173175 qemu-system-s390x -enable-kvm -name guest=artful-test14,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/…on
├─173318 qemu-system-s390x -enable-kvm -name guest=artful-test6,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/v…on
├─173404 qemu-system-s390x -enable-kvm -name guest=artful-test25,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/…on
├─173477 qemu-system-s390x -enable-kvm -name guest=artful-test12,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/…on
├─173554 qemu-system-s390x -enable-kvm -name guest=artful-test3,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/v…on
├─173629 qemu-system-s390x -enable-kvm -name guest=artful-test21,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/…on
├─173706 qemu-system-s390x -enable-kvm -name guest=artful-test7,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/v…on
├─173784 qemu-system-s390x -enable-kvm -name guest=artful-test16,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/…on
├─173878 qemu-system-s390x -enable-kvm -name guest=artful-test28,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/…on
├─173954 qemu-system-s390x -enable-kvm -name guest=artful-test18,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/…on
├─174190 qemu-system-s390x -enable-kvm -name guest=artful-test1,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/v…on
└─174303 qemu-system-s390x -enable-kvm -name guest=artful-test4,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/v…on
Sep 06 04:40:55 s1lp5 dnsmasq-dhcp[3848]: DHCPDISCOVER(virbr0) 192.168.122.48 52:54:00:52:27:0d
Sep 06 04:40:55 s1lp5 dnsmasq-dhcp[3848]: DHCPOFFER(virbr0) 192.168.122.48 52:54:00:52:27:0d
Sep 06 04:40:55 s1lp5 dnsmasq-dhcp[3848]: DHCPREQUEST(virbr0) 192.168.122.225 52:54:00:bf:24:14
Sep 06 04:40:55 s1lp5 dnsmasq-dhcp[3848]: DHCPACK(virbr0) 192.168.122.225 52:54:00:bf:24:14 artful-test17
Sep 06 04:40:55 s1lp5 dnsmasq-dhcp[3848]: DHCPREQUEST(virbr0) 192.168.122.115 52:54:00:d7:97:d4
Sep 06 04:40:55 s1lp5 dnsmasq-dhcp[3848]: DHCPACK(virbr0) 192.168.122.115 52:54:00:d7:97:d4 artful-test24
Sep 06 04:40:55 s1lp5 dnsmasq-dhcp[3848]: DHCPREQUEST(virbr0) 192.168.122.18 52:54:00:75:b1:b2
Sep 06 04:40:55 s1lp5 dnsmasq-dhcp[3848]: DHCPACK(virbr0) 192.168.122.18 52:54:00:75:b1:b2 artful-test26
Sep 06 04:40:55 s1lp5 dnsmasq-dhcp[3848]: DHCPREQUEST(virbr0) 192.168.122.172 52:54:00:99:82:92
Sep 06 04:40:55 s1lp5 dnsmasq-dhcp[3848]: DHCPACK(virbr0) 192.168.122.172 52:54:00:99:82:92 artful-test15
+ date
Mi 6. Sep 04:40:57 EDT 2017
+ sleep 2s
That said artful is good, I have to check former versions if that was an upstream change in the daemons behavior to only return once the socket is avail.
** Changed in: libvirt (Ubuntu Artful)
Status: Incomplete => Fix Released
--
You received this bug notification because you are a member of नेपाली
भाषा समायोजकहरुको समूह, which is subscribed to Xenial.
Matching subscriptions: Ubuntu 16.04 Bugs
https://bugs.launchpad.net/bugs/1571209
Title:
Sockfile check retries too short for a busy system boot
Status in libvirt package in Ubuntu:
Fix Released
Status in libvirt source package in Precise:
Won't Fix
Status in libvirt source package in Trusty:
Triaged
Status in libvirt source package in Wily:
Won't Fix
Status in libvirt source package in Xenial:
Incomplete
Status in libvirt source package in Zesty:
Incomplete
Status in libvirt source package in Artful:
Fix Released
Bug description:
[ problem description ]
sockfile_check_retries is first introduced by #1455608, for preventing
the failure case of sockfile not ready, but it was default to a hard-
coded value "5", it might be too short for a busy system boot.
#1455608 -
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1455608
[ step to reproduce ]
setup a clean install system (Ubuntu Server 14.04.4 LTS), and assemble
os disk as RAID-1, boot up some guest instances (count > 10, start-at-
boot), force shutdown host by pressing power-button for 3s ~ 5s, or
via IPMI command, then power-on afterward. it may sometimes failed to
get sockfile ready after in "post-start" script, with an line of error
in /var/log/syslog,
==> kernel: [ 313.059830] init: libvirt-bin post-start process (2430)
terminated with status 1 <==
since there's multiple VMs Read/Write before a non-graceful shutdown,
RAID devices need to re-sync after boot, and lead to a slow response,
but start-up script for libvirt-bin can only wait 5 cycles, 2 seconds
wait for each cycle, so it will timed-out after 10s, and exit with
"1".
[ possible solution ]
extend the retry times for sockfile waiting, and make it possible to
change via editing `/etc/default/libvirt-bin` file.
<please see the patch file as attachment>
[ sysinfo ]
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.4 LTS
Release: 14.04
Codename: trusty
$ uname -a
Linux host2 4.2.0-35-generic #40~14.04.1-Ubuntu SMP Fri Mar 18 16:37:35 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[ related issue ]
#1386465 -
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1386465
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1571209/+subscriptions
