openstack team mailing list archive

[Jagane Sundar <jagane@xxxxxxxxxx>]

Hello Mike, Tomo:

I invite you to take a look at Livebackup for kvm:

http://wiki.qemu.org/Features/Livebackup

This is a feature that I am currently developing. In brief, livebackup
enables full and incremental backups of running VMs.
It has enhancements to qemu that enable it to keep an in-memory
dirty blocks bitmap. The advantage of this approach is that you never
have to calculate hashes in order to determine which blocks are dirty.

Livebackup includes a rudimentary network protocol for transferring
the dirty blocks over to a backup server. The client livebackup_client
can be used to connect to the running VM, create a livebackup
snapshot - this is a special purpose snapshot that tracks the state
of all disks of a virtual machine at the time of snapshot, and then
creates a COW file of blocks that the VM modifies during the
snapshot.

My intent is to develop this in qemu/qemu-kvm, then add support
in libvirt if necessary, and then build support for this in Openstack.
At that point Openstack will be in a position to offer three types of
VMs:
1. ephemeral VMs with virtual disks that can be lost if the
    VM server's hard disk dies,
2. Fully highly available external storage (NAS or SAN) based
    VMs
3. Hourly backuped up VMs - livebackup_client running on another
   VM server can be utilized to backup the VM every hour, and
   the VM can be re-started from the backup if the primary VM
   server dies.

Let me know what you think.

Thanks,
Jagane

On 5/3/2011 3:05 AM, FUJITA Tomonori wrote:
> Thanks for the explanation!
>
> On Mon, 2 May 2011 21:12:22 -0700
> Michael Barton<mike-launchpad@xxxxxxxxxxxxxxxx>  wrote:
>
> > What I've been playing with is having a manifest that contains hashes
> > of (4mb) chunks for the volume's backups.  When a user initiates a new
> > backup, dm-snapshot does its thing and gives me a block device.  I
> > read and hash chunks from that block device and compare them to the
> > manifest, uploading any that differ to Swift, then update the manifest
> > with the new backup.
> Ah, you read and hash chunks from the "whole" block device. I
> misunderstood the point.
>
> > The restore uses fuse with some basic bitmap logic to lazy load chunks
> > from Swift on demand, plus a background thread that fills them in
> > autonomously.  I've been pretty happy with fuse's performance and
> > stability (python-fuse that is; fusepy is really slow).
> You create a loopback device of the fuse file and set up a dm-snap
> device by using it, right?
>
> Tomo
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp