yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #83315
[Bug 1888022] [NEW] Detach is broken for multi-attached fs-based volumes
Public bug reported:
Description: Detach is broken for multi-attached
LibvirtMountedFileSystemVolumeDriver-based volumes
Steps to reproduce:
1. Deploy Devstack Master
2. Configure Nova to use KVM/Libvirt
3. Configure Cinder to use any LibvirtMountedFileSystemVolumeDriver-
based volume driver - for example - NFS:
cinder.conf:
[nfs]
volume_driver = cinder.volume.drivers.nfs.NfsDriver
volume_backend_name = nfs
nas_secure_file_operations = False
nfs_snapshot_support = True
nas_host = 10.3.35.41
nas_share_path = /volumes/pool1/nas
4. Create a volume type with enabled multi-attach feature:
$ cinder type-create multiattach
$ cinder type-key multiattach set multiattach="<is> True"
$ cinder type-key multiattach set volume_backend_name=nfs
5. Create a volume:
$ cinder create --volume-type nfs 1
6. Boot two Nova virtual machines:
$ nova boot --flavor m1.nano --image linux --nic none a
$ nova boot --flavor m1.nano --image linux --nic none b
7. Attach the volume to both VM's:
$ nova volume-attach ac100d66-e92d-40da-a765-fea72ae0af3c 31b702b9-423b-4402-8a6e-1c3dcf84f956
$ nova volume-attach 0843c96e-2cfe-49ca-a8eb-0d25f806ffeb 31b702b9-423b-4402-8a6e-1c3dcf84f956
8. Check Nova CPU service log file:
Jul 16 22:40:36 openstack-master-lustre7 nova-compute[74494]: INFO
nova.compute.manager [None req-bc029573-7eea-4f56-ba89-060c158f2f75
admin admin] [instance: ac100d66-e92d-40da-a765-fea72ae0af3c] Attaching
volume 31b702b9-423b-4402-8a6e-1c3dcf84f956 to /dev/sdb
Jul 16 22:40:38 openstack-master-lustre7 nova-compute[74494]: DEBUG nova.virt.libvirt.guest [None req-bc029573-7eea-4f56-ba89-060c158f2f75 admin admin] attach device
xml: <disk type="file" device="disk">
<driver name="qemu" type="raw" cache="none" io="native"/>
<source file="/opt/stack/data/nova/mnt/0abe5ba79045d7dd179ddc8a4ff1991c/volume-31b702b9-423b-4402-8a6e-1c3dcf84f956"/>
<target dev="sdb" bus="scsi"/>
<serial>31b702b9-423b-4402-8a6e-1c3dcf84f956</serial>
<shareable/>
<address type="drive" controller="0" unit="1"/>
</disk>
Jul 16 22:40:47 openstack-master-lustre7 nova-compute[74494]: INFO
nova.compute.manager [None req-d0d5238d-a617-48d4-a439-4c998eea21b5
admin admin] [instance: 0843c96e-2cfe-49ca-a8eb-0d25f806ffeb] Attaching
volume 31b702b9-423b-4402-8a6e-1c3dcf84f956 to /dev/sdb
Jul 16 22:40:48 openstack-master-lustre7 nova-compute[74494]: DEBUG nova.virt.libvirt.guest [None req-d0d5238d-a617-48d4-a439-4c998eea21b5 admin admin] attach device xml: <disk type="file" device="disk">
<driver name="qemu" type="raw" cache="none" io="native"/>
<source file="/opt/stack/data/nova/mnt/0abe5ba79045d7dd179ddc8a4ff1991c/volume-31b702b9-423b-4402-8a6e-1c3dcf84f956"/>
<target dev="sdb" bus="scsi"/>
<serial>31b702b9-423b-4402-8a6e-1c3dcf84f956</serial>
<shareable/>
<address type="drive" controller="0" unit="1"/>
</disk>
9. Check the mountpoint:
$ mount -t nfs4
10.3.35.41:/volumes/pool1/nas on /opt/stack/data/nova/mnt/0abe5ba79045d7dd179ddc8a4ff1991c type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.3.196.7,local_lock=none,addr=10.3.35.41)
Looks good and works fine.
10. Detach the volume at the VM a
11. Detach the volume at the VM b
12. Check the mountpoint:
$ mount -t nfs4
10.3.35.41:/volumes/pool1/nas on /opt/stack/data/nova/mnt/0abe5ba79045d7dd179ddc8a4ff1991c type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.3.196.7,local_lock=none,addr=10.3.35.41)
it still mounted - but should be unmounted, when we detach volume at all VM's!
13. Check Nova CPU service log file:
Jul 16 22:46:43 openstack-master-lustre7 nova-compute[74494]: INFO
nova.virt.libvirt.driver [None req-ec46e195-645e-4ad7-a495-3b5fd9578935
admin admin] [instance: ac100d66-e92d-40da-a765-fea72ae0af3c] Detected
multiple connections on this host for volume: 31b702b9-423b-4402-8a6e-
1c3dcf84f956, skipping target disconnect.
Jul 16 22:46:43 openstack-master-lustre7 nova-compute[74494]: INFO
nova.virt.libvirt.driver [None req-048bb510-7012-405e-af7b-2b39772f5ad0
admin admin] [instance: 0843c96e-2cfe-49ca-a8eb-0d25f806ffeb] Detected
multiple connections on this host for volume: 31b702b9-423b-4402-8a6e-
1c3dcf84f956, skipping target disconnect.
Root cause:
nova/compute/manager.py calls:
self.driver.destroy(context, instance, network_info, block_device_info)
from nova/virt/libvirt/driver.py
and destroy(self, context, instance, network_info, block_device_info=None) calls:
self.cleanup(context, instance, network_info, block_device_info) and
self._cleanup(context, instance, network_info, block_device_info=block_device_info)
_disconnect_volume((self, context, connection_info, instance, ncryption=None)
and _disconnect_volume checks:
if self._should_disconnect_target(context, connection_info, instance)
and _should_disconnect_target(context, connection_info, instance) function
checks the volume multi-attach property and number of attachments.
And if len(attachments) > 1 and VM is running on the current host => it skip volume disconnect.
And this logic is correct for true block device drivers (iSCSI/FC),
but it don't works as expected for LibvirtMountedFileSystemVolumeDriver-based drivers.
All LibvirtMountedFileSystemVolumeDriver-based drivers uses
_HostMountStateManager from nova/virt/libvirt/volume/mount.py and this
class manages mounts/unmounts and itself keeps track of the number of
mounts for a particular volume.
So if we exclude the _should_disconnect_target() logic for
LibvirtMountedFileSystemVolumeDriver -based volumes, then this class
will work correctly for multi-attached volumes.
Please take a look my patch.
Thank you!
** Affects: nova
Importance: Undecided
Assignee: Alex Deiter (deiter)
Status: New
** Changed in: nova
Assignee: (unassigned) => Alex Deiter (deiter)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1888022
Title:
Detach is broken for multi-attached fs-based volumes
Status in OpenStack Compute (nova):
New
Bug description:
Description: Detach is broken for multi-attached
LibvirtMountedFileSystemVolumeDriver-based volumes
Steps to reproduce:
1. Deploy Devstack Master
2. Configure Nova to use KVM/Libvirt
3. Configure Cinder to use any LibvirtMountedFileSystemVolumeDriver-
based volume driver - for example - NFS:
cinder.conf:
[nfs]
volume_driver = cinder.volume.drivers.nfs.NfsDriver
volume_backend_name = nfs
nas_secure_file_operations = False
nfs_snapshot_support = True
nas_host = 10.3.35.41
nas_share_path = /volumes/pool1/nas
4. Create a volume type with enabled multi-attach feature:
$ cinder type-create multiattach
$ cinder type-key multiattach set multiattach="<is> True"
$ cinder type-key multiattach set volume_backend_name=nfs
5. Create a volume:
$ cinder create --volume-type nfs 1
6. Boot two Nova virtual machines:
$ nova boot --flavor m1.nano --image linux --nic none a
$ nova boot --flavor m1.nano --image linux --nic none b
7. Attach the volume to both VM's:
$ nova volume-attach ac100d66-e92d-40da-a765-fea72ae0af3c 31b702b9-423b-4402-8a6e-1c3dcf84f956
$ nova volume-attach 0843c96e-2cfe-49ca-a8eb-0d25f806ffeb 31b702b9-423b-4402-8a6e-1c3dcf84f956
8. Check Nova CPU service log file:
Jul 16 22:40:36 openstack-master-lustre7 nova-compute[74494]: INFO
nova.compute.manager [None req-bc029573-7eea-4f56-ba89-060c158f2f75
admin admin] [instance: ac100d66-e92d-40da-a765-fea72ae0af3c]
Attaching volume 31b702b9-423b-4402-8a6e-1c3dcf84f956 to /dev/sdb
Jul 16 22:40:38 openstack-master-lustre7 nova-compute[74494]: DEBUG nova.virt.libvirt.guest [None req-bc029573-7eea-4f56-ba89-060c158f2f75 admin admin] attach device
xml: <disk type="file" device="disk">
<driver name="qemu" type="raw" cache="none" io="native"/>
<source file="/opt/stack/data/nova/mnt/0abe5ba79045d7dd179ddc8a4ff1991c/volume-31b702b9-423b-4402-8a6e-1c3dcf84f956"/>
<target dev="sdb" bus="scsi"/>
<serial>31b702b9-423b-4402-8a6e-1c3dcf84f956</serial>
<shareable/>
<address type="drive" controller="0" unit="1"/>
</disk>
Jul 16 22:40:47 openstack-master-lustre7 nova-compute[74494]: INFO
nova.compute.manager [None req-d0d5238d-a617-48d4-a439-4c998eea21b5
admin admin] [instance: 0843c96e-2cfe-49ca-a8eb-0d25f806ffeb]
Attaching volume 31b702b9-423b-4402-8a6e-1c3dcf84f956 to /dev/sdb
Jul 16 22:40:48 openstack-master-lustre7 nova-compute[74494]: DEBUG nova.virt.libvirt.guest [None req-d0d5238d-a617-48d4-a439-4c998eea21b5 admin admin] attach device xml: <disk type="file" device="disk">
<driver name="qemu" type="raw" cache="none" io="native"/>
<source file="/opt/stack/data/nova/mnt/0abe5ba79045d7dd179ddc8a4ff1991c/volume-31b702b9-423b-4402-8a6e-1c3dcf84f956"/>
<target dev="sdb" bus="scsi"/>
<serial>31b702b9-423b-4402-8a6e-1c3dcf84f956</serial>
<shareable/>
<address type="drive" controller="0" unit="1"/>
</disk>
9. Check the mountpoint:
$ mount -t nfs4
10.3.35.41:/volumes/pool1/nas on /opt/stack/data/nova/mnt/0abe5ba79045d7dd179ddc8a4ff1991c type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.3.196.7,local_lock=none,addr=10.3.35.41)
Looks good and works fine.
10. Detach the volume at the VM a
11. Detach the volume at the VM b
12. Check the mountpoint:
$ mount -t nfs4
10.3.35.41:/volumes/pool1/nas on /opt/stack/data/nova/mnt/0abe5ba79045d7dd179ddc8a4ff1991c type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.3.196.7,local_lock=none,addr=10.3.35.41)
it still mounted - but should be unmounted, when we detach volume at all VM's!
13. Check Nova CPU service log file:
Jul 16 22:46:43 openstack-master-lustre7 nova-compute[74494]: INFO
nova.virt.libvirt.driver [None req-ec46e195-645e-
4ad7-a495-3b5fd9578935 admin admin] [instance: ac100d66-e92d-
40da-a765-fea72ae0af3c] Detected multiple connections on this host for
volume: 31b702b9-423b-4402-8a6e-1c3dcf84f956, skipping target
disconnect.
Jul 16 22:46:43 openstack-master-lustre7 nova-compute[74494]: INFO
nova.virt.libvirt.driver [None req-048bb510-7012-405e-af7b-
2b39772f5ad0 admin admin] [instance: 0843c96e-2cfe-49ca-a8eb-
0d25f806ffeb] Detected multiple connections on this host for volume:
31b702b9-423b-4402-8a6e-1c3dcf84f956, skipping target disconnect.
Root cause:
nova/compute/manager.py calls:
self.driver.destroy(context, instance, network_info, block_device_info)
from nova/virt/libvirt/driver.py
and destroy(self, context, instance, network_info, block_device_info=None) calls:
self.cleanup(context, instance, network_info, block_device_info) and
self._cleanup(context, instance, network_info, block_device_info=block_device_info)
_disconnect_volume((self, context, connection_info, instance, ncryption=None)
and _disconnect_volume checks:
if self._should_disconnect_target(context, connection_info, instance)
and _should_disconnect_target(context, connection_info, instance) function
checks the volume multi-attach property and number of attachments.
And if len(attachments) > 1 and VM is running on the current host => it skip volume disconnect.
And this logic is correct for true block device drivers (iSCSI/FC),
but it don't works as expected for LibvirtMountedFileSystemVolumeDriver-based drivers.
All LibvirtMountedFileSystemVolumeDriver-based drivers uses
_HostMountStateManager from nova/virt/libvirt/volume/mount.py and this
class manages mounts/unmounts and itself keeps track of the number of
mounts for a particular volume.
So if we exclude the _should_disconnect_target() logic for
LibvirtMountedFileSystemVolumeDriver -based volumes, then this class
will work correctly for multi-attached volumes.
Please take a look my patch.
Thank you!
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1888022/+subscriptions
Follow ups
