yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #02424
[Bug 1046313] Re: At termination, LXC rootfs is not always unmounted before rmtree() is called
This bug was fixed in the package nova -
2012.1.3+stable-20130423-e52e6912-0ubuntu1
---------------
nova (2012.1.3+stable-20130423-e52e6912-0ubuntu1) precise-proposed; urgency=low
* Resynchronize with stable/essex (e52e6912) (LP: #1089488):
- [48e81f1] VNC proxy can be made to connect to wrong VM LP: 1125378
- [3bf5a58] snat rule too broad for some network configurations LP: 1048765
- [efaacda] DOS by allocating all fixed ips LP: 1125468
- [b683ced] Add nosehtmloutput as a test dependency.
- [45274c8] Nova unit tests not running, but still passing for stable/essex
LP: 1132835
- [e02b459] vnc unit-test fixes
- [87361d3] Jenkins jobs fail because of incompatibility between sqlalchemy-
migrate and the newest sqlalchemy-0.8.0b1 (LP: #1073569)
- [e98928c] VNC proxy can be made to connect to wrong VM LP: 1125378
- [c0a10db] DoS through XML entity expansion (CVE-2013-1664) LP: 1100282
- [243d516] No authentication on block device used for os-volume_boot
LP: 1069904
- [80fefe5] use_single_default_gateway does not function correctly
(LP: #1075859)
- [bd10241] Essex 2012.1.3 : Error deleting instance with 2 Nova Volumes
attached (LP: #1079745)
- [86a5937] do_refresh_security_group_rules in nova.virt.firewall is very
slow (LP: #1062314)
- [ae9c5f4] deallocate_fixed_ip attempts to update an already deleted
fixed_ip (LP: #1017633)
- [20f98c5] failed to allocate fixed ip because old deleted one exists
(LP: #996482)
- [75f6922] snapshot stays in saving state if the vm base image is deleted
(LP: #921774)
- [1076699] lock files may be removed in error dues to permissions issues
(LP: #1051924)
- [40c5e94] ensure_default_security_group() does not call sgh (LP: #1050982)
- [4eebe76] At termination, LXC rootfs is not always unmounted before
rmtree() is called (LP: #1046313)
- [47dabb3] Heavily loaded nova-compute instances don't sent reports
frequently enough (LP: #1045152)
- [b375b4f] When attach volume lost attach when node restart (LP: #1004791)
- [4ac2dcc] nova usage-list returns wrong usage (LP: #1043999)
- [014fcbc] Bridge port's hairpin mode not set after resuming a machine
(LP: #1040537)
- [2f35f8e] Nova flavor ephemeral space size reported incorrectly
(LP: #1026210)
* Dropped, superseeded by new snapshot:
- debian/patches/CVE-2013-0335.patch: [48e81f1]
- debian/patches/CVE-2013-1838.patch: [efaacda]
- debian/patches/CVE-2013-1664.patch: [c0a10db]
- debian/patches/CVE-2013-0208.patch: [243d516]
-- Yolanda <yolanda.robla@xxxxxxxxxxxxx> Mon, 22 Apr 2013 12:37:08 +0200
** Changed in: nova (Ubuntu Precise)
Status: Fix Committed => Fix Released
** CVE added: http://www.cve.mitre.org/cgi-
bin/cvename.cgi?name=2013-0208
** CVE added: http://www.cve.mitre.org/cgi-
bin/cvename.cgi?name=2013-0335
** CVE added: http://www.cve.mitre.org/cgi-
bin/cvename.cgi?name=2013-1664
** CVE added: http://www.cve.mitre.org/cgi-
bin/cvename.cgi?name=2013-1838
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1046313
Title:
At termination, LXC rootfs is not always unmounted before rmtree() is
called
Status in OpenStack Compute (Nova):
Fix Released
Status in OpenStack Compute (nova) essex series:
Fix Released
Status in “nova” package in Ubuntu:
Fix Released
Status in “nova” source package in Precise:
Fix Released
Bug description:
nova version used:
commit 20c6bb6c9000fa0d193f688b668f5f3eeda8fb05
Merge: aedaf10 0876cf5
Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
Date: Wed Aug 29 14:33:01 2012 +0000
Merge "Do not run pylint by default"
Symptom:
The rootfs of LXC instance is not unmounted before rmtree() is called in the nova/virt/libvirt/driver.py file.
I've seen this problem in Essex and in Folsom.
It does not happen always, though.
I suspect there is timing issues between unmount() and rmtree().
This bug eventually leads to "no free nbd device".
Example:
After terminating instance i-00000005, I still see that its rootfs is mounted to /dev/nbd15:
$ mount
/dev/nbd15 on /usr/local/upstream-Aug-29/instances/instance-00000005/rootfs type ext2 (rw)
Since it is not unmounted before rmtree() is called, nova-compute
complains.
Here is the log of nova-compute:
2012-09-04 09:11:46 INFO nova.virt.libvirt.driver [-] [instance: 8e0b9d15-2c4b-40e7-a932-90c8d39d9657] Instance destroyed successfully.
2012-09-04 09:11:46 DEBUG nova.utils [req-52c4813e-2ae8-4307-af31-158d896fe374 admin admin] Attempting to grab semaphore "iptables" for method "_apply"... from (pid=10672) inner /usr/local/nova/nova/utils.py:708
2012-09-04 09:11:46 DEBUG nova.utils [req-52c4813e-2ae8-4307-af31-158d896fe374 admin admin] Got semaphore "iptables" for method "_apply"... from (pid=10672) inner /usr/local/nova/nova/utils.py:712
2012-09-04 09:11:46 DEBUG nova.utils [req-52c4813e-2ae8-4307-af31-158d896fe374 admin admin] Attempting to grab file lock "iptables" for method "_apply"... from (pid=10672) inner /usr/local/nova/nova/utils.py:716
2012-09-04 09:11:46 DEBUG nova.utils [req-52c4813e-2ae8-4307-af31-158d896fe374 admin admin] Got file lock "iptables" for method "_apply"... from (pid=10672) inner /usr/local/nova/nova/utils.py:724
2012-09-04 09:11:46 DEBUG nova.utils [req-52c4813e-2ae8-4307-af31-158d896fe374 admin admin] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c -t filter from (pid=10672) execute /usr/local/nova/nova/utils.py:176
2012-09-04 09:11:46 DEBUG nova.utils [req-52c4813e-2ae8-4307-af31-158d896fe374 admin admin] Result was 0 from (pid=10672) execute /usr/local/nova/nova/utils.py:191
2012-09-04 09:11:46 DEBUG nova.utils [req-52c4813e-2ae8-4307-af31-158d896fe374 admin admin] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c from (pid=10672) execute /usr/local/nova/nova/utils.py:176
2012-09-04 09:11:46 DEBUG nova.utils [req-52c4813e-2ae8-4307-af31-158d896fe374 admin admin] Result was 0 from (pid=10672) execute /usr/local/nova/nova/utils.py:191
2012-09-04 09:11:46 DEBUG nova.utils [req-52c4813e-2ae8-4307-af31-158d896fe374 admin admin] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c -t nat from (pid=10672) execute /usr/local/nova/nova/utils.py:176
2012-09-04 09:11:46 DEBUG nova.utils [req-52c4813e-2ae8-4307-af31-158d896fe374 admin admin] Result was 0 from (pid=10672) execute /usr/local/nova/nova/utils.py:191
2012-09-04 09:11:46 DEBUG nova.utils [req-52c4813e-2ae8-4307-af31-158d896fe374 admin admin] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c from (pid=10672) execute /usr/local/nova/nova/utils.py:176
2012-09-04 09:11:47 DEBUG nova.utils [req-52c4813e-2ae8-4307-af31-158d896fe374 admin admin] Result was 0 from (pid=10672) execute /usr/local/nova/nova/utils.py:191
2012-09-04 09:11:47 DEBUG nova.network.linux_net [req-52c4813e-2ae8-4307-af31-158d896fe374 admin admin] IPTablesManager.apply completed with success from (pid=10672) _apply /usr/local/nova/nova/network/linux_net.py:369
2012-09-04 09:11:47 INFO nova.virt.libvirt.driver [req-52c4813e-2ae8-4307-af31-158d896fe374 admin admin] [instance: 8e0b9d15-2c4b-40e7-a932-90c8d39d9657] Deleting instance files /usr/local/nova/instances/instance-00000005
2012-09-04 09:11:47 ERROR nova.virt.libvirt.driver [req-52c4813e-2ae8-4307-af31-158d896fe374 admin admin] Failed to cleanup directory /usr/local/nova/instances/instance-00000005: [Errno 13] Permission denied: '/usr/local/nova/instances/instance-00000005/rootfs/lost+found'
I can manually unmount it and release /dev/nbd15 to finish the
termination process.
Without doing that, nbd15 is permanently occupied by the terminated instance.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1046313/+subscriptions
