openstack team mailing list archive

["Alex Lyakas" <alex@xxxxxxxxxxxxxxxxx>]

Hello Kei,
thank you for your attention.

According to the code, this should reproduce every time, because of two 
reasons:
1) There is no code that ensures that the same backing file is present at 
the destination within the "/var/lib/nova/instances/_base" folder. At least 
I don't see such code, and it's very easy to verify this (just make sure 
that the same image is not already in "_base", and then do block migraiton).
2) Even if the backing file is manually placed within the "_base" folder at 
the destination, after performing block migration with the 
VIR_MIGRATE_NON_SHARED_INC flag, the destination image does not report to 
have a backing file (when asked by "qemu-img info").

These two issues reproduce every single time. Please let me know if you need 
any additional info reproducing them.

Thanks,
 Alex.


-----Original Message----- 
From: masumotok@xxxxxxxxxxxxx
Sent: 11 September, 2011 8:18 PM
To: vishvananda@xxxxxxxxx ; alex@xxxxxxxxxxxxxxxxx
Cc: openstack@xxxxxxxxxxxxxxxxxxx
Subject: RE: [Openstack] Questions on nova code: networking & live/offline 
migration

Hello Alex,

I currently focusing on your statement, "data corrupted". that is priority. 
I and other 2 testers( my colleague) never seen before. Recently, I  haven't 
touched block migration thing. there may be  something that I forgot, so let 
me check again, then I can say what kind of information necessary to you.

P.S. we also use ubuntu natty(11.04), nothing special.
       one host is nova-api, nova-scheduler.... anything. other host is 
nova-compute only.

Regards,
Kei

________________________________
差出人: Vishvananda Ishaya [vishvananda@xxxxxxxxx]
送信日時: 2011年9月11日 1:09
宛先: Alex Lyakas
CC: RDH 桝本 圭(ＩＴアーキテクチャ); 
openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>
件名: Re: [Openstack] Questions on nova code: networking & live/offline 
migration

I have also been testing block migration a bit.  I see a similar issue. 
When you migrate a to another host, it copies without a backing file.  It 
seems to work ok, it just loses its backing file so you end up with very 
large disks.  It would be great to add a feature that could move the backing 
files for both disk and disk.local and not need to create very large files 
on the destination.

Vish

On Sep 9, 2011, at 4:23 PM, Alex Lyakas wrote:

Hello Kei,
yes, I was doing "block_migration" and not "live_migration".
I saw two issues.
First is that I see no code that ensures that there is a backing file on the 
destination (within the _base folder). Second is when I manually ensured the 
base image is in the _base folder. In this case, on destination the VM came 
up, but its data was corrupted. And indeed, when I asked "qemu-img info" 
about the VM's image file, it did not mention that the image has a backing 
file (you can see the output below). That's when I tried to change the flag 
to VIR_MIGRATE_NON_SHARED_DISK, and then things were fine, but of course the 
whole image got copied.

Our environment is stock ubuntu natty 11.04 and all standard packages, 
nothing special. We use the latest version of nova, we periodically merge 
latest nova code. I can give you any additional details required. Can you 
pls repeat the test in your environment and do 'qemu-img info' on the 
destination image?

Thanks,
 alex.


2011/9/9 <masumotok@xxxxxxxxxxxxx<mailto:masumotok@xxxxxxxxxxxxx>>
Hello,

Sorry for late reply.

Hmm.. regarding to Q1 and Q2,
if you go "nova-manage vm live_migration i-xxxxxxx dest", you have to 
prepare shared storage, no need to move backing file. 
VIR_MIGRATE_NON_SHARED_INC is not used then.
On the other hand, n case of " nova-manage vm block_migration i-xxxxx dest", 
you dont have to prepare shared storage. backing file is automatically 
copied to destination, AFAIK.
The reason why default flag value is not VIR_MIGRATE_NON_SHARED_DISK  is 
VIR_MIGRATE_NON_SHARED_INC is stable and faster in our test environment.

Your situation is backing file was not copied, wasnt it? I appreciate if you 
tell me your environment in detail.

Regards,
Kei

________________________________
差出人: 
openstack-bounces+masumotok=nttdata.co.jp@xxxxxxxxxxxxxxxxxxx<mailto:nttdata.co.jp@xxxxxxxxxxxxxxxxxxx> 
[openstack-bounces+masumotok=nttdata.co.jp@xxxxxxxxxxxxxxxxxxx<mailto:nttdata.co.jp@xxxxxxxxxxxxxxxxxxx>] 
は Vishvananda Ishaya [vishvananda@xxxxxxxxx<mailto:vishvananda@xxxxxxxxx>] 
の代理
送信日時: 2011年9月9日 6:12
宛先: Alex Lyakas
CC: 
openstack@xxxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxxx><mailto:openstack@xxxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxxx>>
件名: Re: [Openstack] Questions on nova code: networking & live/offline 
migration


On Sep 6, 2011, at 9:57 AM, Alex Lyakas wrote:

Greetings everybody,
I have been reading the nova code (pretty close to the latest version) and I 
have a couple of questions regarding it.

Live migration:
I see that VIR_MIGRATE_NON_SHARED_INC flag is used, meaning that at 
destination, the same backing file should be available. (Indeed, the images 
that are created for instances use backing files in '_base' folder).
Question 1: how it is ensured that the backing file on the destination 
really exists? I.e., '_cache_image' has to be called there for the same 
image. It looks like this is not ensured at all. If the same image was never 
spawned earlier on destination, the backing file will not exist there. I 
verified this and indeed there was no backing file on the destination.

live migration should have _base in shared storage so it doesn't need to 
move.  If you are talking about block_migration, then this could definitely 
be a bug.

Question 2: even when the backing file does exist, the image created at the 
destination looks incorrect, it is not using the backing file! When querying 
the file on the destination, after live migration, I don't see that it's 
backed by the cached file:
dst# qemu-img info /var/lib/nova/instances/instance-00000441/disk
image: /var/lib/nova/instances/instance-00000441/disk
file format: qcow2
virtual size: 3.0G (3221225472 bytes)
disk size: 50M
cluster_size: 65536

While on the source it is backed:
src# qemu-img info disk
image: disk
file format: qcow2
virtual size: 3.0G (3221225472 bytes)
disk size: 50M
cluster_size: 65536
backing file: 
/var/lib/nova/instances/_base/af3e133428b9e25c55bc59fe534248e6a0c0f17b 
(actual path: 
/var/lib/nova/instances/_base/af3e133428b9e25c55bc59fe534248e6a0c0f17b)

again, if you are referring to block_migration, this could be a bug.


And indeed, on destination, when I log in into the VM, its image disk is 
corrupted. Various tools like 'less','vi',sudo' fail to run.
When I changed the flag to VIR_MIGRATE_NON_SHARED_DISK, then the whole image 
was copied, and this time the instance on the destination was fine. So it 
looks there is some bug on that path when using VIR_MIGRATE_NON_SHARED_INC 
flag.

add_fixed_ip_to_instance path:
> From the code it looks like per each network, an instance has exactly one 
virtual interface, meaning exactly one MAC address. However, it looks like 
this is possible to have multiple fixed_IPs per same virtual_interface. This 
is reflected in NetworkManager.get_instance_nw_info(), when adding the
info['ips'] = [ip_dict(ip) for ip in network_IPs]
entry.
(On the other hand, in LibvirtBridgeDriver._get_configurations() only the 
first IP is taken from this list. )
Although I see that NetworkManager._setup_network() is called on that path, 
and even generates a new MAC address (in case of FlatDHCP), but a new 
virtual_interface is not created.
What is the purpose of this ability? On the instance we still see a single 
network interface per each network, correct?

tr3buchet.  Comments here?  Not sure there is a need to have multiple ips on 
the same nic.

Offline migration ("resize"):
I see that when using libvirt, this path is not implemented; it is 
implemented only for Xen. It looks like the following methods need to be 
implemented: migrate_disk_and_power_off() & finish_migration(). Is there any 
special reason for not implementing those methods with libvirt?

Just that no one has done it.  We would love to have this supported in 
libvirt.


Thanks everybody,
Alex.


_______________________________________________
Mailing list: 
https://launchpad.net/~openstack<https://launchpad.net/%7Eopenstack>
Post to     : 
openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx><mailto:openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
Unsubscribe : 
https://launchpad.net/~openstack<https://launchpad.net/%7Eopenstack>
More help   : https://help.launchpad.net/ListHelp