yahoo-eng-team team mailing list archive

[Adam Gandelman <1207422@xxxxxxxxxxxxxxxxxx>]

** Changed in: nova/grizzly
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1207422

Title:
  Spawning multiple instances can cause race conditions with nbd

Status in OpenStack Compute (Nova):
  Fix Released
Status in OpenStack Compute (nova) folsom series:
  New
Status in OpenStack Compute (nova) grizzly series:
  Fix Released

Bug description:
  Spawning a number of instances using a single request (or number of
  requests close enough to each other) can result in the same nbd device
  chosen for more than one instance. For example, see the log of a
  collision below. The python threads switch after the device is
  selected and qemu-nbd is spawned, but before the qemu-nbd gets a
  chance to connect and create a /sys/block/nbd*/pid file.

  Solution is most likely to put a lock around the _inner_get_dev() part
  of NbdMount.

  2013-08-01 12:18:02.893 23130 DEBUG nova.virt.disk.mount.nbd [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Get nbd device /dev/nbd14 for /var/lib/nova/instances/0fde8031-0152-49ca-bd59-fd7a9353afc0/disk _inner_get_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/nbd.py:87
  2013-08-01 12:18:02.893 23130 DEBUG nova.openstack.common.processutils [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf qemu-nbd -c /dev/nbd14 /var/lib/nova/instances/0fde8031-0152-49ca-bd59-fd7a9353afc0/disk execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:142
  2013-08-01 12:18:02.924 23130 DEBUG nova.virt.disk.mount.nbd [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Get nbd device /dev/nbd14 for /var/lib/nova/instances/10474a10-4bfa-41f2-8bd6-9add1e6aff1f/disk _inner_get_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/nbd.py:87
  2013-08-01 12:18:02.924 23130 DEBUG nova.openstack.common.processutils [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf qemu-nbd -c /dev/nbd14 /var/lib/nova/instances/10474a10-4bfa-41f2-8bd6-9add1e6aff1f/disk execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:142
  2013-08-01 12:18:02.968 23130 DEBUG nova.virt.disk.mount.api [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Map dev /dev/nbd14 map_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/api.py:135
  2013-08-01 12:18:02.968 23130 DEBUG nova.virt.disk.mount.api [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Mount /dev/nbd14p1 on /tmp/openstack-vfs-localfsnW0rfa mnt_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/api.py:188
  2013-08-01 12:18:02.968 23130 DEBUG nova.openstack.common.processutils [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/nbd14p1 /tmp/openstack-vfs-localfsnW0rfa execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:142
  2013-08-01 12:18:03.000 23130 DEBUG nova.virt.disk.mount.api [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Map dev /dev/nbd14 map_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/api.py:135
  2013-08-01 12:18:03.000 23130 DEBUG nova.openstack.common.processutils [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf kpartx -a /dev/nbd14 execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:142
  Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/nbd14p1 /tmp/openstack-vfs-localfsnW0rfa
  Stderr: 'mount: special device /dev/nbd14p1 does not exist\n' mnt_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/api.py:193
  2013-08-01 12:18:03.044 23130 DEBUG nova.virt.disk.mount.api [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Unmap dev /dev/nbd14 unmap_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/api.py:179
  2013-08-01 12:18:03.045 23130 DEBUG nova.virt.disk.mount.nbd [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Release nbd device /dev/nbd14 unget_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/nbd.py:126
  2013-08-01 12:18:03.045 23130 DEBUG nova.openstack.common.processutils [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf qemu-nbd -d /dev/nbd14 execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:142
  Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/nbd14p1 /tmp/openstack-vfs-localfsnW0rfa
  Stderr: 'mount: special device /dev/nbd14p1 does not exist\n') setup /usr/lib/python2.7/dist-packages/nova/virt/disk/vfs/localfs.py:81
  Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/nbd14p1 /tmp/openstack-vfs-localfsnW0rfa
  Stderr: 'mount: special device /dev/nbd14p1 does not exist\n')
  2013-08-01 12:18:04.117 23130 DEBUG nova.virt.disk.mount.nbd [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Release nbd device /dev/nbd14 unget_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/nbd.py:126
  2013-08-01 12:18:04.118 23130 DEBUG nova.openstack.common.processutils [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf qemu-nbd -d /dev/nbd14 execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:142

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1207422/+subscriptions