yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1925818] [NEW] Retype of non-encrypted root volume to encrypted fails

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Jan Wasilewski <1925818@xxxxxxxxxxxxxxxxxx>
Date: Fri, 23 Apr 2021 14:52:49 -0000
Reply-to: Bug 1925818 <1925818@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========
I tried to retype volume from unencrypted to encrypted one. That works fine for non-root volumes, however it doesn't work when volume was/is root volume.

Steps to reproduce
==================
1. Create an instance with unencrypted root volume(select do not delete volume on termination) 
2. Delete instance
3. Retype left volume from deleted instance to encrypted one and wait till retype will finish successfully
4. Launch an instance from this encrypted volume(increase original size by +2GB)

Expected result
===============
New instance is started from encrypted volume

Actual result
=============
Instance fails to start. At horizon we can see an error message:
Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 610, in build_instances filter_properties, instances[0].uuid) File "/usr/lib/python2.7/dist-packages/nova/scheduler/utils.py", line 680, in populate_retry raise exception.MaxRetriesExceeded(reason=msg) MaxRetriesExceeded: Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance d19504b4-8c05-423b-abaa-b1b8c0658fa8. Last exception: Unexpected error while running command. Command: cryptsetup --batch-mode luksFormat --key-file=- --cipher aes-xts-plain64 --key-size 256 /dev/dm-19 Exit code: 5 Stdout: u'' Stderr: u'Cannot use device /dev/dm-19 which is in use (already mapped or mounted).\n' 

Environment
===========
1. Version of cinder:
ii  cinder-api                             2:13.0.9-0ubuntu1~cloud1.1                      all          Cinder storage service - API server
ii  cinder-backup                          2:13.0.9-0ubuntu1~cloud1.1                      all          Cinder storage service - Scheduler server
ii  cinder-common                          2:13.0.9-0ubuntu1~cloud1.1                      all          Cinder storage service - common files
ii  cinder-scheduler                       2:13.0.9-0ubuntu1~cloud1.1                      all          Cinder storage service - Scheduler server
ii  cinder-volume                          2:13.0.9-0ubuntu1~cloud1.1                      all          Cinder storage service - Volume server
ii  python-cinder                          2:13.0.9-0ubuntu1~cloud1.1                      all          Cinder Python 2 libraries
ii  python-cinderclient                    1:3.5.0-0ubuntu1                                all          Python bindings to the OpenStack Volume API - Python 2.x

Version of barbican:

ii  barbican-api                           1:10.0.0-0ubuntu0.20.04.1~cloud0                all          OpenStack Key Management Service - API Server
ii  barbican-common                        1:10.0.0-0ubuntu0.20.04.1~cloud0                all          OpenStack Key Management Service - common files
ii  barbican-keystone-listener             1:10.0.0-0ubuntu0.20.04.1~cloud0                all          OpenStack Key Management Service - Keystone Listener
ii  barbican-worker                        1:10.0.0-0ubuntu0.20.04.1~cloud0                all          OpenStack Key Management Service - Worker Node
ii  python-barbican                        1:6.0.1-0ubuntu1                                all          OpenStack Key Management Service - Python files
ii  python3-barbican                       1:10.0.0-0ubuntu0.20.04.1~cloud0                all          OpenStack Key Management Service - Python 3 files
ii  python3-barbicanclient                 4.10.0-0ubuntu1~cloud0                          all          OpenStack Key Management API client - Python 3.x


Version of nova:
ii  nova-api                               2:18.3.0-0ubuntu1~cloud1                        all          OpenStack Compute - API frontend
ii  nova-common                            2:18.3.0-0ubuntu1~cloud1                        all          OpenStack Compute - common files
ii  nova-conductor                         2:18.3.0-0ubuntu1~cloud1                        all          OpenStack Compute - conductor service
ii  nova-consoleauth                       2:18.3.0-0ubuntu1~cloud1                        all          OpenStack Compute - Console Authenticator
ii  nova-novncproxy                        2:18.3.0-0ubuntu1~cloud1                        all          OpenStack Compute - NoVNC proxy
ii  nova-placement-api                     2:18.3.0-0ubuntu1~cloud1                        all          OpenStack Compute - placement API frontend
ii  nova-scheduler                         2:18.3.0-0ubuntu1~cloud1                        all          OpenStack Compute - virtual machine scheduler
ii  python-nova                            2:18.3.0-0ubuntu1~cloud1                        all          OpenStack Compute Python 2 libraries
ii  python-novaclient                      2:11.0.0-0ubuntu1~cloud0                        all          client library for OpenStack Compute API - Python 2.7


2. We use iSCSI storage(Huawei dorado)


Logs & Configs
==============
Cinder volume logs:
2021-04-23 16:33:43.288 17298 INFO cinder.volume.drivers.huawei.huawei_driver [req-19f3af30-022a-4c96-ab28-2c3aaa91764c 7d52a2a770e4402982390eb658425083 ab5b748ab5e04fa58c8b4aad9f744e2b - default default] Initialize iscsi connection for volume de2c38b7-bf67-42b7-ae51-76fca6c7c05b, connector info {u'initiator': u'iqn.1993-08.org.debian:01:3a7188e5b55f', u'ip': u'10.10.0.248', u'platform': u'x86_64', u'host': u'comp-c08', u'do_local_attach': False, u'mountpoint': u'/dev/vda', u'os_type': u'linux2', u'multipath': True}.
2021-04-23 16:34:05.883 17298 INFO cinder.volume.drivers.huawei.huawei_driver [req-969c4e48-91a7-4c46-824d-63b924503c34 7d52a2a770e4402982390eb658425083 ab5b748ab5e04fa58c8b4aad9f744e2b - default default] Terminate iscsi connection for volume de2c38b7-bf67-42b7-ae51-76fca6c7c05b, connector info {u'initiator': u'iqn.1993-08.org.debian:01:3a7188e5b55f', u'ip': u'10.10.0.248', u'platform': u'x86_64', u'host': u'comp-c08', u'do_local_attach': False, u'mountpoint': u'/dev/vda', u'os_type': u'linux2', u'multipath': True}.

2021-04-23 16:34:12.452 17298 INFO
cinder.volume.drivers.huawei.huawei_driver [req-314b1423-2650-4a06-9f2e-
945bd648e70f 7d52a2a770e4402982390eb658425083
ab5b748ab5e04fa58c8b4aad9f744e2b - default default] Initialize iscsi
connection for volume de2c38b7-bf67-42b7-ae51-76fca6c7c05b, connector
info {u'initiator': u'iqn.1993-08.org.debian:01:a1edf5552e8', u'ip':
u'10.10.0.253', u'platform': u'x86_64', u'host': u'comp-a09',
u'do_local_attach': False, u'mountpoint': u'/dev/vda', u'os_type':
u'linux2', u'multipath': True}.

2021-04-23 16:34:58.111 17298 INFO
cinder.volume.drivers.huawei.huawei_driver [req-77231901-21bf-
42c3-99a3-5a2e0b13a066 7d52a2a770e4402982390eb658425083
ab5b748ab5e04fa58c8b4aad9f744e2b - default default] Terminate iscsi
connection for volume de2c38b7-bf67-42b7-ae51-76fca6c7c05b, connector
info {u'initiator': u'iqn.1993-08.org.debian:01:36b34c8cbdd8', u'ip':
u'10.10.0.148', u'platform': u'x86_64', u'host': u'comp-a03',
u'do_local_attach': False, u'mountpoint': u'/dev/vda', u'os_type':
u'linux2', u'multipath': True}.


In case some additional logs are needed, just let me know.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1925818

Title:
  Retype of non-encrypted root volume to encrypted fails

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  I tried to retype volume from unencrypted to encrypted one. That works fine for non-root volumes, however it doesn't work when volume was/is root volume.

  Steps to reproduce
  ==================
  1. Create an instance with unencrypted root volume(select do not delete volume on termination) 
  2. Delete instance
  3. Retype left volume from deleted instance to encrypted one and wait till retype will finish successfully
  4. Launch an instance from this encrypted volume(increase original size by +2GB)

  Expected result
  ===============
  New instance is started from encrypted volume

  Actual result
  =============
  Instance fails to start. At horizon we can see an error message:
  Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 610, in build_instances filter_properties, instances[0].uuid) File "/usr/lib/python2.7/dist-packages/nova/scheduler/utils.py", line 680, in populate_retry raise exception.MaxRetriesExceeded(reason=msg) MaxRetriesExceeded: Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance d19504b4-8c05-423b-abaa-b1b8c0658fa8. Last exception: Unexpected error while running command. Command: cryptsetup --batch-mode luksFormat --key-file=- --cipher aes-xts-plain64 --key-size 256 /dev/dm-19 Exit code: 5 Stdout: u'' Stderr: u'Cannot use device /dev/dm-19 which is in use (already mapped or mounted).\n' 

  Environment
  ===========
  1. Version of cinder:
  ii  cinder-api                             2:13.0.9-0ubuntu1~cloud1.1                      all          Cinder storage service - API server
  ii  cinder-backup                          2:13.0.9-0ubuntu1~cloud1.1                      all          Cinder storage service - Scheduler server
  ii  cinder-common                          2:13.0.9-0ubuntu1~cloud1.1                      all          Cinder storage service - common files
  ii  cinder-scheduler                       2:13.0.9-0ubuntu1~cloud1.1                      all          Cinder storage service - Scheduler server
  ii  cinder-volume                          2:13.0.9-0ubuntu1~cloud1.1                      all          Cinder storage service - Volume server
  ii  python-cinder                          2:13.0.9-0ubuntu1~cloud1.1                      all          Cinder Python 2 libraries
  ii  python-cinderclient                    1:3.5.0-0ubuntu1                                all          Python bindings to the OpenStack Volume API - Python 2.x

  Version of barbican:

  ii  barbican-api                           1:10.0.0-0ubuntu0.20.04.1~cloud0                all          OpenStack Key Management Service - API Server
  ii  barbican-common                        1:10.0.0-0ubuntu0.20.04.1~cloud0                all          OpenStack Key Management Service - common files
  ii  barbican-keystone-listener             1:10.0.0-0ubuntu0.20.04.1~cloud0                all          OpenStack Key Management Service - Keystone Listener
  ii  barbican-worker                        1:10.0.0-0ubuntu0.20.04.1~cloud0                all          OpenStack Key Management Service - Worker Node
  ii  python-barbican                        1:6.0.1-0ubuntu1                                all          OpenStack Key Management Service - Python files
  ii  python3-barbican                       1:10.0.0-0ubuntu0.20.04.1~cloud0                all          OpenStack Key Management Service - Python 3 files
  ii  python3-barbicanclient                 4.10.0-0ubuntu1~cloud0                          all          OpenStack Key Management API client - Python 3.x

  
  Version of nova:
  ii  nova-api                               2:18.3.0-0ubuntu1~cloud1                        all          OpenStack Compute - API frontend
  ii  nova-common                            2:18.3.0-0ubuntu1~cloud1                        all          OpenStack Compute - common files
  ii  nova-conductor                         2:18.3.0-0ubuntu1~cloud1                        all          OpenStack Compute - conductor service
  ii  nova-consoleauth                       2:18.3.0-0ubuntu1~cloud1                        all          OpenStack Compute - Console Authenticator
  ii  nova-novncproxy                        2:18.3.0-0ubuntu1~cloud1                        all          OpenStack Compute - NoVNC proxy
  ii  nova-placement-api                     2:18.3.0-0ubuntu1~cloud1                        all          OpenStack Compute - placement API frontend
  ii  nova-scheduler                         2:18.3.0-0ubuntu1~cloud1                        all          OpenStack Compute - virtual machine scheduler
  ii  python-nova                            2:18.3.0-0ubuntu1~cloud1                        all          OpenStack Compute Python 2 libraries
  ii  python-novaclient                      2:11.0.0-0ubuntu1~cloud0                        all          client library for OpenStack Compute API - Python 2.7

  
  2. We use iSCSI storage(Huawei dorado)

  
  Logs & Configs
  ==============
  Cinder volume logs:
  2021-04-23 16:33:43.288 17298 INFO cinder.volume.drivers.huawei.huawei_driver [req-19f3af30-022a-4c96-ab28-2c3aaa91764c 7d52a2a770e4402982390eb658425083 ab5b748ab5e04fa58c8b4aad9f744e2b - default default] Initialize iscsi connection for volume de2c38b7-bf67-42b7-ae51-76fca6c7c05b, connector info {u'initiator': u'iqn.1993-08.org.debian:01:3a7188e5b55f', u'ip': u'10.10.0.248', u'platform': u'x86_64', u'host': u'comp-c08', u'do_local_attach': False, u'mountpoint': u'/dev/vda', u'os_type': u'linux2', u'multipath': True}.
  2021-04-23 16:34:05.883 17298 INFO cinder.volume.drivers.huawei.huawei_driver [req-969c4e48-91a7-4c46-824d-63b924503c34 7d52a2a770e4402982390eb658425083 ab5b748ab5e04fa58c8b4aad9f744e2b - default default] Terminate iscsi connection for volume de2c38b7-bf67-42b7-ae51-76fca6c7c05b, connector info {u'initiator': u'iqn.1993-08.org.debian:01:3a7188e5b55f', u'ip': u'10.10.0.248', u'platform': u'x86_64', u'host': u'comp-c08', u'do_local_attach': False, u'mountpoint': u'/dev/vda', u'os_type': u'linux2', u'multipath': True}.

  2021-04-23 16:34:12.452 17298 INFO
  cinder.volume.drivers.huawei.huawei_driver [req-314b1423-2650-4a06
  -9f2e-945bd648e70f 7d52a2a770e4402982390eb658425083
  ab5b748ab5e04fa58c8b4aad9f744e2b - default default] Initialize iscsi
  connection for volume de2c38b7-bf67-42b7-ae51-76fca6c7c05b, connector
  info {u'initiator': u'iqn.1993-08.org.debian:01:a1edf5552e8', u'ip':
  u'10.10.0.253', u'platform': u'x86_64', u'host': u'comp-a09',
  u'do_local_attach': False, u'mountpoint': u'/dev/vda', u'os_type':
  u'linux2', u'multipath': True}.

  2021-04-23 16:34:58.111 17298 INFO
  cinder.volume.drivers.huawei.huawei_driver [req-77231901-21bf-
  42c3-99a3-5a2e0b13a066 7d52a2a770e4402982390eb658425083
  ab5b748ab5e04fa58c8b4aad9f744e2b - default default] Terminate iscsi
  connection for volume de2c38b7-bf67-42b7-ae51-76fca6c7c05b, connector
  info {u'initiator': u'iqn.1993-08.org.debian:01:36b34c8cbdd8', u'ip':
  u'10.10.0.148', u'platform': u'x86_64', u'host': u'comp-a03',
  u'do_local_attach': False, u'mountpoint': u'/dev/vda', u'os_type':
  u'linux2', u'multipath': True}.

  
  In case some additional logs are needed, just let me know.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1925818/+subscriptions
Follow ups

[Bug 1925818] Re: Retype of non-encrypted root volume to encrypted fails
From: Lee Yarwood, 2021-04-29