openstack team mailing list archive

[Scott Moser <smoser@xxxxxxxxxx>]

On Mon, 11 Jun 2012, Fredric Morenius wrote:

> Hello again,
>
> I did some further tests on this issue with the qcow2 image, the qcow2 image converted to raw and the _blank (raw) image from the UEC tar archive.
>
> I loaded nbd with
>
> $ sudo modprobe nbd max_part=8
>
> *** qcow2 image ***
>
> $ k$ md5sum cirros-0.3.0-x86_64-disk.img
> 50bdc35edb03a38d91b1b071afb20a3c  cirros-0.3.0-x86_64-disk.img

this image is a qcow compressed disk image.
The disk contains a partition table, grub (0.97) installed on the MBR, and
a root filesystem on the first partition.

> That creates the device /dev/nbd15p1
>
> kpartx on top of that:
>
> $ sudo kpartx -a /dev/nbd15
> device-mapper: resume ioctl failed: Invalid argument
> create/reload failed on nbd15p1
>
> $ sudo qemu-nbd -d /dev/nbd15
> /dev/nbd15 disconnected

That seems like "working as designed".  at least on ubuntu 12.04, the kpartx is
no longer necessary.  I'd have to dig more to figure out exactly why, but
some combination of kernel partition scanner and udev "do the right
thing" when this new network block device is set up.

> *** UEC image ***
>
> $ qemu-img info cirros-0.3.0-x86_64-blank.img
> image: cirros-0.3.0-x86_64-blank.img
> file format: raw
> virtual size: 24M (25165824 bytes)
> disk size: 24M
>
> $ sudo qemu-nbd -c /dev/nbd15 $PWD/cirros-0.3.0-x86_64-blank.img

This image is essentially the same as you'd get if you did:
 truncate --size 24M disk.img
 mkfs.ext3 -F disk.img -L cirros-rootfs

Ie, it is a unpartitioned disk with an (empty) filesystem on it.
There are no partitions, or data in the filesystem.

On boot, cirros initramfs copies itself to that disk, and then pivot_roots
to it.  Cirros will run perfectly fine as an initramfs with no root
filesystem too, fwiw.

> That DOES NOT create the device /dev/nbd15p1
>
> kpartx on top of that:
>
> $ sudo kpartx -a /dev/nbd15
>
> kpartx executes without error
>
> $ sudo qemu-nbd -d /dev/nbd15
> /dev/nbd15 disconnected

This also seems like "working as designed".

> *** raw image ***
>
> $ qemu-img convert -p -O raw cirros-0.3.0-x86_64-disk.img cirros-0.3.0-x86_64-disk_raw.img
>
> $ qemu-img info cirros-0.3.0-x86_64-disk_raw.img
> image: cirros-0.3.0-x86_64-disk_raw.img
> file format: raw
> virtual size: 39M (41126400 bytes)
> disk size: 12M
>
> $ sudo qemu-nbd -c /dev/nbd15 $PWD/cirros-0.3.0-x86_64-disk_raw.img
>
> That creates the device /dev/nbd15p1
>
> kpartx on top of that:
>
> $ sudo kpartx -a /dev/nbd15
> device-mapper: resume ioctl failed: Invalid argument
> create/reload failed on nbd15p1
>
> $ sudo qemu-nbd -d /dev/nbd15
> /dev/nbd15 disconnected
>
> So, the issue is not with the qcow2 vs raw image formats. Instead kpartx only executes without error provided that the device /dev/nbd15p1 is not created.
> When p1 is created by qemu-nbd, kpartx is not needed to mount the image
> device, but the problem is qemu-nbd does not create p1 in all cases. I
> have not checked the details of the OpenStack code, but an (ugly) work
> around could be to only execute kpartx when it is needed.

Without digging around on older versions of OS's and their included
kernel/udev/kpartx.... i'd have to say that from your message above, and
my testing that running 'kpartx' is no longer necessary.  That can
reasonably be expected to be created by the kernel or some other plumbing
bits automatically.

If it just happens magically,though, I'm not sure how you would know if it
hasn't happened yet.  That would imply that this might be racey.  I ran
into similar bugs http://pad.lv/937352 and http://pad.lv/942788 that
affected the growing of the root partition from an initramfs.

The thing we ended up settling on was that after re-writing the partition
table, we had to run 'udevadm --settle'.  its quite possible that
'qemu-nbd --connect' sets off a similar flurry of events that would need
to be waited on before a /dev/nbdXpY would exist.