yahoo-eng-team team mailing list archive

[Sean Dague <sean@xxxxxxxxx>]

** No longer affects: nova/folsom

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1152519

Title:
  nbd device can not be released after injecting files into image if
  mount operation failed

Status in OpenStack Compute (Nova):
  Fix Released

Bug description:
  I got this issue with stable folsom when I wanted to run a windows8 instance, but I believe this issue still exists in grizzly.
  My host distro is debian wheezy, and the version of 'mount' is mount_2.20.1-5.3_amd64.deb.
  error logs:
  2013-03-07 17:52:50 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf kpartx -a /dev/nbd14 from (pid=19455) execute /usr/local/lib/python2.7/dist-packages/nova/utils.py:192
  2013-03-07 17:52:50 DEBUG nova.utils [-] Result was 0 from (pid=19455) execute /usr/local/lib/python2.7/dist-packages/nova/utils.py:207
  2013-03-07 17:52:50 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/mapper/nbd14p1 /tmp/openstack-disk-mount-tmp9Plrpc from (pid=19455) execute /usr/local/lib/python2.7/dist-packages/nova/utils.py:192
  2013-03-07 17:52:51 DEBUG nova.utils [-] Result was 0 from (pid=19455) execute /usr/local/lib/python2.7/dist-packages/nova/utils.py:207
  =============here we can see that the 'umount' operation isn't executed=============
  2013-03-07 17:52:51 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf kpartx -d /dev/nbd14 from (pid=19455) execute /usr/local/lib/python2.7/dist-packages/nova/utils.py:192
  2013-03-07 17:52:51 DEBUG nova.utils [-] Result was 1 from (pid=19455) execute /usr/local/lib/python2.7/dist-packages/nova/utils.py:207
  2013-03-07 17:52:51 WARNING nova.virt.libvirt.driver [-] [instance: 172195e0-7074-450e-9840-937e9173c20b] Ignoring error injecting data into image 4bd31030-7410-4dc4-8c01-71bdc06b7f86 ([Errno 16] Device or resource busy: '/tmp/openstack-disk-mount-tmp9Plrpc')

  I have debug this issue inline, and find that error occurs in the 'mount' operation, (===but the nbd device has been mounted!===):
          _out, err = utils.trycmd('mount', self.mapped_device, self.mount_dir,
                                   run_as_root=True)
          if err:  ######### err is set by the utils.trycmd() method.
              self.error = _('Failed to mount filesystem: %s') % err
              return False   #### return here
  so the do_umount method will not call the unmnt_dev method because self.mounted is False:
  def do_umount(self):
          """Call the unmnt, unmap and unget operations."""
          if self.mounted:
              self.unmnt_dev()
  and then the 'kpartx -d' operation will fail(raise ProcessExecutionError), the 'qemu-nbd -d' operation will not be executed.
  Finally, the nbd device used by qemu-nbd isn't released!

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1152519/+subscriptions