← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1996622] [NEW] Cannot mount old encrypted volume to an instance with Invalid password, cannot unlock any keyslot

 

Public bug reported:

Description
===========
After an upgrade of barbican from ussuri to yoga version there is no possibility to attach encrypted volumes created before an upgrade to any instance, because of an error: "libvirt.libvirtError: internal error: unable to execute QEMU command 'blockdev-add': Invalid password, cannot unlock any keyslot". Encrypted volumes created after an upgrade are able to attach to instances, without such error.

Steps to reproduce
==================
1. Have already created encrypted volume
2. Execute command:
openstack server add volume my-new-instance my-old-encrypted-volume
3. Check attachments details by:
openstack server show my-new-instance

Expected result
===============
my-old-encrypted-volume visible in volumes_attached list. Inside VM OS newly attached drive should be visible

Actual result
=============
my-old-encrypted-volume is not visible in volumes_attached list. During attachment I'm able to see such errors in nova-compute logs: https://paste.openstack.org/show/bNbPOHiQJOq8OsKZ5Gn2/
Barbican logs or cinder logs are not saying anything wrong. What is more, I can correctly retrieve a payload of a key from barbican and secret, which is used for keeping passphrase for a my-old-encrypted-volume, by command:
barbican secret get --payload_content_type application/octet-stream secret-id-and-href --file my_symmetric_key.key

The same procedure, executed for a freshly created volume is working
fine - new encrypted disk is visible inside instance OS.

Environment
===========
1. Exact version of OpenStack you are running. See the following
# dpkg -l | grep nova
ii  nova-api                               2:21.2.4-0ubuntu1                                    all          OpenStack Compute - API frontend
ii  nova-common                            2:21.2.4-0ubuntu1                                    all          OpenStack Compute - common files
ii  nova-conductor                         2:21.2.4-0ubuntu1                                    all          OpenStack Compute - conductor service
ii  nova-novncproxy                        2:21.2.4-0ubuntu1                                    all          OpenStack Compute - NoVNC proxy
ii  nova-scheduler                         2:21.2.4-0ubuntu1                                    all          OpenStack Compute - virtual machine scheduler
ii  python3-nova                           2:21.2.4-0ubuntu1                                    all          OpenStack Compute Python 3 libraries
ii  python3-novaclient                     2:17.0.0-0ubuntu1                                    all          client library for OpenStack Compute API - 3.x

# dpkg -l | grep barbican
ii  barbican-api                          2:14.0.0-0ubuntu1~cloud0              all          OpenStack Key Management Service - API Server
ii  barbican-common                       2:14.0.0-0ubuntu1~cloud0              all          OpenStack Key Management Service - common files
ii  barbican-keystone-listener            2:14.0.0-0ubuntu1~cloud0              all          OpenStack Key Management Service - Keystone Listener
ii  barbican-worker                       2:14.0.0-0ubuntu1~cloud0              all          OpenStack Key Management Service - Worker Node
ii  python3-barbican                      2:14.0.0-0ubuntu1~cloud0              all          OpenStack Key Management Service - Python 3 files
ii  python3-barbicanclient                5.2.0-0ubuntu1~cloud0                 all          OpenStack Key Management API client - Python 3.x

2. Which hypervisor did you use?
Libvirt:
# dpkg -l | grep libvirt
ii  libvirt-daemon                         6.0.0-0ubuntu8.16                                    amd64        Virtualization daemon
ii  libvirt-daemon-driver-qemu             6.0.0-0ubuntu8.16                                    amd64        Virtualization daemon QEMU connection driver
ii  libvirt-daemon-driver-storage-rbd      6.0.0-0ubuntu8.16                                    amd64        Virtualization daemon RBD storage driver
ii  libvirt0:amd64                         6.0.0-0ubuntu8.16                                    amd64        library for interfacing with different virtualization systems
ii  python3-libvirt                        6.1.0-1                                              amd64        libvirt Python 3 bindings

2. Which storage type did you use?
iSCSI Huawei dorado

3. Which networking type did you use?
Neutron linuxbridge

Logs & Configs
==============
An error message from nova-compute log: https://paste.openstack.org/show/bNbPOHiQJOq8OsKZ5Gn2/

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: cinder volumes

** Description changed:

  Description
  ===========
- After an upgrade of barbican from ussuri to yoga version there is no possibility to attach encrypted volumes created before an upgrade to any instance, because of an error: "libvirt.libvirtError: internal error: unable to execute QEMU command 'blockdev-add': Invalid password, cannot unlock any keyslot". Encrypted volumes created after an upgrade are able to attach to instances, without such error. 
+ After an upgrade of barbican from ussuri to yoga version there is no possibility to attach encrypted volumes created before an upgrade to any instance, because of an error: "libvirt.libvirtError: internal error: unable to execute QEMU command 'blockdev-add': Invalid password, cannot unlock any keyslot". Encrypted volumes created after an upgrade are able to attach to instances, without such error.
  
  Steps to reproduce
  ==================
  1. Have already created encrypted volume
  2. Execute command:
  openstack server add volume my-new-instance my-old-encrypted-volume
  3. Check attachments details by:
  openstack server show my-new-instance
  
  Expected result
  ===============
- my-old-encrypted-volume visible in volumes_attached list
+ my-old-encrypted-volume visible in volumes_attached list. Inside VM OS newly attached drive should be visible
  
  Actual result
  =============
  my-old-encrypted-volume is not visible in volumes_attached list. During attachment I'm able to see such errors in nova-compute logs: https://paste.openstack.org/show/bNbPOHiQJOq8OsKZ5Gn2/
  Barbican logs or cinder logs are not saying anything wrong. What is more, I can correctly retrieve a payload of a key from barbican and secret, which is used for keeping passphrase for a my-old-encrypted-volume, by command:
- barbican secret get --payload_content_type application/octet-stream secret-id-and-href --file my_symmetric_key.key 
+ barbican secret get --payload_content_type application/octet-stream secret-id-and-href --file my_symmetric_key.key
  
  The same procedure, executed for a freshly created volume is working
  fine - new encrypted disk is visible inside instance OS.
  
  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
  # dpkg -l | grep nova
  ii  nova-api                               2:21.2.4-0ubuntu1                                    all          OpenStack Compute - API frontend
  ii  nova-common                            2:21.2.4-0ubuntu1                                    all          OpenStack Compute - common files
  ii  nova-conductor                         2:21.2.4-0ubuntu1                                    all          OpenStack Compute - conductor service
  ii  nova-novncproxy                        2:21.2.4-0ubuntu1                                    all          OpenStack Compute - NoVNC proxy
  ii  nova-scheduler                         2:21.2.4-0ubuntu1                                    all          OpenStack Compute - virtual machine scheduler
  ii  python3-nova                           2:21.2.4-0ubuntu1                                    all          OpenStack Compute Python 3 libraries
  ii  python3-novaclient                     2:17.0.0-0ubuntu1                                    all          client library for OpenStack Compute API - 3.x
  
  # dpkg -l | grep barbican
  ii  barbican-api                          2:14.0.0-0ubuntu1~cloud0              all          OpenStack Key Management Service - API Server
  ii  barbican-common                       2:14.0.0-0ubuntu1~cloud0              all          OpenStack Key Management Service - common files
  ii  barbican-keystone-listener            2:14.0.0-0ubuntu1~cloud0              all          OpenStack Key Management Service - Keystone Listener
  ii  barbican-worker                       2:14.0.0-0ubuntu1~cloud0              all          OpenStack Key Management Service - Worker Node
  ii  python3-barbican                      2:14.0.0-0ubuntu1~cloud0              all          OpenStack Key Management Service - Python 3 files
  ii  python3-barbicanclient                5.2.0-0ubuntu1~cloud0                 all          OpenStack Key Management API client - Python 3.x
  
  2. Which hypervisor did you use?
  Libvirt:
  # dpkg -l | grep libvirt
  ii  libvirt-daemon                         6.0.0-0ubuntu8.16                                    amd64        Virtualization daemon
  ii  libvirt-daemon-driver-qemu             6.0.0-0ubuntu8.16                                    amd64        Virtualization daemon QEMU connection driver
  ii  libvirt-daemon-driver-storage-rbd      6.0.0-0ubuntu8.16                                    amd64        Virtualization daemon RBD storage driver
  ii  libvirt0:amd64                         6.0.0-0ubuntu8.16                                    amd64        library for interfacing with different virtualization systems
  ii  python3-libvirt                        6.1.0-1                                              amd64        libvirt Python 3 bindings
  
- 
  2. Which storage type did you use?
  iSCSI Huawei dorado
  
  3. Which networking type did you use?
  Neutron linuxbridge
  
  Logs & Configs
  ==============
  An error message from nova-compute log: https://paste.openstack.org/show/bNbPOHiQJOq8OsKZ5Gn2/

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1996622

Title:
  Cannot mount old encrypted volume to an instance with Invalid
  password, cannot unlock any keyslot

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  After an upgrade of barbican from ussuri to yoga version there is no possibility to attach encrypted volumes created before an upgrade to any instance, because of an error: "libvirt.libvirtError: internal error: unable to execute QEMU command 'blockdev-add': Invalid password, cannot unlock any keyslot". Encrypted volumes created after an upgrade are able to attach to instances, without such error.

  Steps to reproduce
  ==================
  1. Have already created encrypted volume
  2. Execute command:
  openstack server add volume my-new-instance my-old-encrypted-volume
  3. Check attachments details by:
  openstack server show my-new-instance

  Expected result
  ===============
  my-old-encrypted-volume visible in volumes_attached list. Inside VM OS newly attached drive should be visible

  Actual result
  =============
  my-old-encrypted-volume is not visible in volumes_attached list. During attachment I'm able to see such errors in nova-compute logs: https://paste.openstack.org/show/bNbPOHiQJOq8OsKZ5Gn2/
  Barbican logs or cinder logs are not saying anything wrong. What is more, I can correctly retrieve a payload of a key from barbican and secret, which is used for keeping passphrase for a my-old-encrypted-volume, by command:
  barbican secret get --payload_content_type application/octet-stream secret-id-and-href --file my_symmetric_key.key

  The same procedure, executed for a freshly created volume is working
  fine - new encrypted disk is visible inside instance OS.

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
  # dpkg -l | grep nova
  ii  nova-api                               2:21.2.4-0ubuntu1                                    all          OpenStack Compute - API frontend
  ii  nova-common                            2:21.2.4-0ubuntu1                                    all          OpenStack Compute - common files
  ii  nova-conductor                         2:21.2.4-0ubuntu1                                    all          OpenStack Compute - conductor service
  ii  nova-novncproxy                        2:21.2.4-0ubuntu1                                    all          OpenStack Compute - NoVNC proxy
  ii  nova-scheduler                         2:21.2.4-0ubuntu1                                    all          OpenStack Compute - virtual machine scheduler
  ii  python3-nova                           2:21.2.4-0ubuntu1                                    all          OpenStack Compute Python 3 libraries
  ii  python3-novaclient                     2:17.0.0-0ubuntu1                                    all          client library for OpenStack Compute API - 3.x

  # dpkg -l | grep barbican
  ii  barbican-api                          2:14.0.0-0ubuntu1~cloud0              all          OpenStack Key Management Service - API Server
  ii  barbican-common                       2:14.0.0-0ubuntu1~cloud0              all          OpenStack Key Management Service - common files
  ii  barbican-keystone-listener            2:14.0.0-0ubuntu1~cloud0              all          OpenStack Key Management Service - Keystone Listener
  ii  barbican-worker                       2:14.0.0-0ubuntu1~cloud0              all          OpenStack Key Management Service - Worker Node
  ii  python3-barbican                      2:14.0.0-0ubuntu1~cloud0              all          OpenStack Key Management Service - Python 3 files
  ii  python3-barbicanclient                5.2.0-0ubuntu1~cloud0                 all          OpenStack Key Management API client - Python 3.x

  2. Which hypervisor did you use?
  Libvirt:
  # dpkg -l | grep libvirt
  ii  libvirt-daemon                         6.0.0-0ubuntu8.16                                    amd64        Virtualization daemon
  ii  libvirt-daemon-driver-qemu             6.0.0-0ubuntu8.16                                    amd64        Virtualization daemon QEMU connection driver
  ii  libvirt-daemon-driver-storage-rbd      6.0.0-0ubuntu8.16                                    amd64        Virtualization daemon RBD storage driver
  ii  libvirt0:amd64                         6.0.0-0ubuntu8.16                                    amd64        library for interfacing with different virtualization systems
  ii  python3-libvirt                        6.1.0-1                                              amd64        libvirt Python 3 bindings

  2. Which storage type did you use?
  iSCSI Huawei dorado

  3. Which networking type did you use?
  Neutron linuxbridge

  Logs & Configs
  ==============
  An error message from nova-compute log: https://paste.openstack.org/show/bNbPOHiQJOq8OsKZ5Gn2/

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1996622/+subscriptions



Follow ups