← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #81941

[Bug 1685875] Re: Swapping encrypted volumes leaves an decrypted volume attached to the instance

  Thread Previous • Date Previous • • Thread Next 
** Summary changed:

- Swapping encrypted volumes leaves an encrypted volume attached to the instance
+ Swapping encrypted volumes leaves an decrypted volume attached to the instance

** Changed in: nova
       Status: Confirmed => Fix Released

** Changed in: nova
     Assignee: (unassigned) => Lee Yarwood (lyarwood)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1685875

Title:
  Swapping encrypted volumes leaves an decrypted volume attached to the
  instance

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Description
  ===========
  Swapping encrypted volumes via `nova volume-update` currently results in the new volume being attached to the instance still encrypted and the original volume still connected to the host.

  Steps to reproduce
  ==================

  # cinder type-create LUKS
  # cinder encryption-type-create --cipher aes-xts-plain64 \
        --key_size 512  --control_location front-end LUKS  \
        nova.volume.encryptors.luks.LuksEncryptor
  # cinder create --volume-type LUKS 1
  # cinder create --volume-type LUKS 1
  # nova boot --image cirros-0.3.5-x86_64-disk --flavor 2 test

  # nova volume-attach 445a811f-ca80-49d3-b3bd-156cb492c96c \
                       d7c43bc8-dbc4-4668-950a-a57397af7cbf
  # nova volume-update 445a811f-ca80-49d3-b3bd-156cb492c96c \
                       d7c43bc8-dbc4-4668-950a-a57397af7cbf \
                       f998a062-8227-41b4-8400-d82a0f223ec3

  Expected result
  ===============

  The original volume is disconnected from the host, the new volume is
  decrypted and attached to the instance.

  Actual result
  =============

  f998a062, the new volume is now attached to the instance but without
  being decrypted :

  $ sudo virsh domblklist 445a811f-ca80-49d3-b3bd-156cb492c96c                                                                                                                                                             
  Target     Source
  ------------------------------------------------
  vda        /opt/stack/data/nova/instances/445a811f-ca80-49d3-b3bd-156cb492c96c/disk
  vdb        /dev/disk/by-path/ip-192.168.122.224:3260-iscsi-iqn.2010-10.org.openstack:volume-f998a062-8227-41b4-8400-d82a0f223ec3-lun-1

  
  d7c43bc8, the original volumes is still connected and decrypted on the host :

  $ iscsiadm -m session
  tcp: [5] 192.168.122.224:3260,1 iqn.2010-10.org.openstack:volume-d7c43bc8-dbc4-4668-950a-a57397af7cbf (non-flash)
  tcp: [6] 192.168.122.224:3260,1 iqn.2010-10.org.openstack:volume-f998a062-8227-41b4-8400-d82a0f223ec3 (non-flash)
  $ ll /dev/disk/by-path/ip-192.168.122.224\:3260-iscsi-iqn.2010-10.org.openstack\:volume-d7c43bc8-dbc4-4668-950a-a57397af7cbf-lun-1 
  lrwxrwxrwx. 1 root root 123 Apr 24 12:39 /dev/disk/by-path/ip-192.168.122.224:3260-iscsi-iqn.2010-10.org.openstack:volume-d7c43bc8-dbc4-4668-950a-a57397af7cbf-lun-1 -> /dev/mapper/crypt-ip-192.168.122.224:3260-iscsi-iqn.2010-10.org.openstack:volume-d7c43bc8-dbc4-4668-950a-a57397af7cbf-lun-1
  $ ll /dev/mapper/crypt*
  lrwxrwxrwx. 1 root root 7 Apr 24 12:40 /dev/mapper/crypt-ip-192.168.122.224:3260-iscsi-iqn.2010-10.org.openstack:volume-d7c43bc8-dbc4-4668-950a-a57397af7cbf-lun-1 -> ../dm-2

  
  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
     list for all releases: http://docs.openstack.org/releases/

  $ git rev-parse HEAD
  0039231719d2a02c59e7cd76631e2ff03cc86b0d

  
  2. Which hypervisor did you use?
     (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
     What's the version of that?

  Libvirt + KVM

  2. Which storage type did you use?
     (For example: Ceph, LVM, GPFS, ...)
     What's the version of that?

  LVM/iSCSI

  3. Which networking type did you use?
     (For example: nova-network, Neutron with OpenVSwitch, ...)

  N/A

  Logs & Configs
  ==============

  n-cpu.log

  2017-04-24 12:41:00.740 ERROR nova.compute.manager [req-95448809-bf98-4468-82cf-f4681678a8a4 admin admin] [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c] Failed to swap volume d7c43bc                                                      8-dbc4-4668-950a-a57397af7cbf for f998a062-8227-41b4-8400-d82a0f223ec3
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c] Traceback (most recent call last):
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]   File "/opt/stack/nova/nova/compute/manager.py", line 4936, in _swap_volume
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]     resize_to)
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1328, in swap_volume
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]     self._disconnect_volume(old_connection_info, disk_dev) 
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1176, in _disconnect_volume
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]     vol_driver.disconnect_volume(connection_info, disk_dev)
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]   File "/opt/stack/nova/nova/virt/libvirt/volume/iscsi.py", line 74, in disconnect_volume
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]     self.connector.disconnect_volume(connection_info['data'], None)
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]   File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 150, in trace_logging_wrapper
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]     result = f(*args, **kwargs)
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]     return f(*args, **kwargs)
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 502, in disconnect_volume
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]     self._disconnect_volume_iscsi(props)
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 523, in _disconnect_volume_                        iscsi
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]     self._linuxscsi.wait_for_volume_removal(host_device)
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]   File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 61, in _wrapper
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]     return r.call(f, *args, **kwargs)
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]   File "/usr/lib/python2.7/site-packages/retrying.py", line 212, in call
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]     raise attempt.get()
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]   File "/usr/lib/python2.7/site-packages/retrying.py", line 247, in get
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]     six.reraise(self.value[0], self.value[1], self.value[2])
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]   File "/usr/lib/python2.7/site-packages/retrying.py", line 200, in call 
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]   File "/usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py", line 81, in wait_for_volume_removal
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]     volume_path=volume_path)
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c] VolumePathNotRemoved: Volume path /dev/disk/by-path/ip-192.168.122.224:3260-iscsi-iqn.2010-10.org.openstack:volume-d7c43bc8-dbc4-4668-950a-a57397af7cbf-lun-1 was not removed in time.
  2017-04-24 12:41:00.740 TRACE nova.compute.manager [instance: 445a811f-ca80-49d3-b3bd-156cb492c96c]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1685875/+subscriptions

References

  Thread Previous • Date Previous • • Thread Next