yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1491563] Re: Failed to spawn lxc instance due to nbd mount in use or rootfs busy

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Launchpad Bug Tracker <1491563@xxxxxxxxxxxxxxxxxx>
Date: Mon, 17 Jul 2017 04:17:30 -0000
Reply-to: Bug 1491563 <1491563@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
[Expired for OpenStack Compute (nova) because there has been no activity
for 60 days.]

** Changed in: nova
       Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1491563

Title:
  Failed to spawn lxc instance due to nbd mount in use or rootfs busy

Status in OpenStack Compute (nova):
  Expired

Bug description:
  I have a devstack change up which runs tempest with nova + the lxc
  backend and it fails some instances on boot because the nbd device is
  in use or the rootfs mount is busy:

  https://review.openstack.org/#/c/219448/

  http://logs.openstack.org/48/219448/1/check/gate-tempest-dsvm-
  full/bc0fc2a/logs/screen-n-cpu.txt.gz?level=TRACE

  It looks like we're leaking nbd devices and failing to tear down
  instances properly, e.g.:

  2015-09-01 22:10:50.141 ERROR nova.virt.disk.api [req-7e5384a8-34f3-4e3a-98cd-b4bb544804a7 tempest-ServersAdminTestJSON-276960936 tempest-ServersAdminTestJSON-718419646] Failed to mount container filesystem '<nova.virt.disk.api._DiskImage object at 0x7f94ec58cd90>' on '/opt/stack/data/nova/instances/a9317161-ad31-4948-9c2d-3d4899718a78/rootfs': 
  --
  Failed to mount filesystem: Unexpected error while running command.
  Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/nbd4 /opt/stack/data/nova/instances/a9317161-ad31-4948-9c2d-3d4899718a78/rootfs
  Exit code: 32
  Stdout: u''
  Stderr: u'mount: /dev/nbd4 already mounted or /opt/stack/data/nova/instances/a9317161-ad31-4948-9c2d-3d4899718a78/rootfs busy\n'
  2015-09-01 22:10:50.142 ERROR nova.compute.manager [req-7e5384a8-34f3-4e3a-98cd-b4bb544804a7 tempest-ServersAdminTestJSON-276960936 tempest-ServersAdminTestJSON-718419646] [instance: a9317161-ad31-4948-9c2d-3d4899718a78] Instance failed to spawn
  2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] Traceback (most recent call last):
  2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2138, in _build_resources
  2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78]     yield resources
  2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2008, in _build_and_run_instance
  2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78]     block_device_info=block_device_info)
  2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2449, in spawn
  2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78]     block_device_info=block_device_info)
  2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4521, in _create_domain_and_network
  2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78]     block_device_info, disk_info):
  2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78]   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
  2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78]     return self.gen.next()
  2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4430, in _lxc_disk_handler
  2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78]     block_device_info, disk_info)
  2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4382, in _create_domain_setup_lxc
  2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78]     container_dir=container_dir)
  2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78]   File "/opt/stack/new/nova/nova/virt/disk/api.py", line 452, in setup_container
  2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78]     raise exception.NovaException(img.errors)
  2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] NovaException: 
  2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] --
  2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] Failed to mount filesystem: Unexpected error while running command.
  2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/nbd4 /opt/stack/data/nova/instances/a9317161-ad31-4948-9c2d-3d4899718a78/rootfs
  2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] Exit code: 32
  2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] Stdout: u''
  2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] Stderr: u'mount: /dev/nbd4 already mounted or /opt/stack/data/nova/instances/a9317161-ad31-4948-9c2d-3d4899718a78/rootfs busy\n'
  2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] 


  
  2015-09-01 22:10:50.025 DEBUG nova.virt.disk.mount.api [req-7e5384a8-34f3-4e3a-98cd-b4bb544804a7 tempest-ServersAdminTestJSON-276960936 tempest-ServersAdminTestJSON-718419646] Failed to mount filesystem: Unexpected error while running command.
  Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/nbd4 /opt/stack/data/nova/instances/a9317161-ad31-4948-9c2d-3d4899718a78/rootfs
  Exit code: 32
  Stdout: u''
  Stderr: u'mount: /dev/nbd4 already mounted or /opt/stack/data/nova/instances/a9317161-ad31-4948-9c2d-3d4899718a78/rootfs busy\n' mnt_dev /opt/stack/new/nova/nova/virt/disk/mount/api.py:237

  etc

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1491563/+subscriptions
References

[Bug 1491563] [NEW] Failed to spawn lxc instance due to nbd mount in use or rootfs busy
From: Matt Riedemann, 2015-09-02