yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1721522] Re: encrypted volumes: Cannot format device /dev/sdb which is still in use

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Launchpad Bug Tracker <1721522@xxxxxxxxxxxxxxxxxx>
Date: Wed, 27 Dec 2017 04:17:34 -0000
Reply-to: Bug 1721522 <1721522@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
[Expired for OpenStack Compute (nova) because there has been no activity
for 60 days.]

** Changed in: nova
       Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1721522

Title:
  encrypted volumes: Cannot format device /dev/sdb which is still in use

Status in OpenStack Compute (nova):
  Expired

Bug description:
  Hi,

  I followed the guide here -
  https://docs.openstack.org/cinder/pike/configuration/block-storage
  /volume-encryption.html

  I also use Barbican and for that I added [barbican] auth_endpoint =
  http://controller:5000 to cinder.conf and nova.conf

  Creation of LUKS disks is successful. I also created normal disks and
  could easily attach them to an instance.

  Cinder disks are on LVM

  Attaching LUKS disks fails with the following trace:

  017-10-05 11:44:57.445 1 INFO nova.compute.manager [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Attaching volume 5c03f92f-470a-4f15-aaca-49d9232512a8 to /dev/vdc
  2017-10-05 11:44:57.835 1 INFO oslo.privsep.daemon [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/nova/nova.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmpKwgHpn/privsep.sock']
  2017-10-05 11:44:58.598 1 INFO oslo.privsep.daemon [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] Spawned new privsep daemon via rootwrap
  2017-10-05 11:44:58.548 80 INFO oslo.privsep.daemon [-] privsep daemon starting
  2017-10-05 11:44:58.553 80 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
  2017-10-05 11:44:58.558 80 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none
  2017-10-05 11:44:58.558 80 INFO oslo.privsep.daemon [-] privsep daemon running as pid 80
  2017-10-05 11:45:01.468 1 INFO os_brick.initiator.connectors.iscsi [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] Trying to connect to iSCSI portal 10.10.245.211:3260
  2017-10-05 11:45:05.762 1 WARNING os_brick.encryptors [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] Use of the in tree encryptor class nova.volume.encryptors.luks.LuksEncryptor by directly referencing the implementation class will be blocked in the Queens release of os-brick.
  2017-10-05 11:45:07.431 1 WARNING os_brick.encryptors.luks [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] isLuks exited abnormally (status 1): Device /dev/sdb is not a valid LUKS device.
  Command failed with code 22: Invalid argument
  : ProcessExecutionError: Unexpected error while running command.
  Command: cryptsetup isLuks --verbose /dev/sdb
  Exit code: 1
  Stdout: u''
  Stderr: u'Device /dev/sdb is not a valid LUKS device.\nCommand failed with code 22: Invalid argument\n'
  2017-10-05 11:45:07.432 1 INFO os_brick.encryptors.luks [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] /dev/sdb is not a valid LUKS device; formatting device for first use
  2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Failed to attach volume at mountpoint: /dev/vdc: ProcessExecutionError: Unexpected error while running command.
  Command: cryptsetup --batch-mode luksFormat --key-file=- --cipher aes-xts-plain64 --key-size 256 /dev/sdb
  Exit code: 5
  Stdout: u''
  Stderr: u'Cannot format device /dev/sdb which is still in use.\n'
  2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Traceback (most recent call last):
  2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1250, in attach_volume
  2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d]     encryptor.attach_volume(context, **encryption)
  2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/encryptors/luks.py", line 160, in attach_volume
  2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d]     self._format_volume(passphrase, **kwargs)
  2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/encryptors/luks.py", line 87, in _format_volume
  2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d]     attempts=3)
  2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/executor.py", line 52, in _execute
  2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d]     result = self.__execute(*args, **kwargs)
  2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line 169, in execute
  2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d]     return execute_root(*cmd, **kwargs)
  2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 205, in _wrap
  2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d]     return self.channel.remote_call(name, args, kwargs)
  2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 202, in remote_call
  2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d]     raise exc_type(*result[2])
  2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] ProcessExecutionError: Unexpected error while running command.
  2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Command: cryptsetup --batch-mode luksFormat --key-file=- --cipher aes-xts-plain64 --key-size 256 /dev/sdb
  2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Exit code: 5
  2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Stdout: u''
  2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Stderr: u'Cannot format device /dev/sdb which is still in use.\n'
  2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d]
  2017-10-05 11:45:19.027 1 ERROR nova.virt.block_device [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Driver failed to attach volume 5c03f92f-470a-4f15-aaca-49d9232512a8 at /dev/vdc: ProcessExecutionError: Unexpected error while running command.
  Command: cryptsetup --batch-mode luksFormat --key-file=- --cipher aes-xts-plain64 --key-size 256 /dev/sdb
  Exit code: 5
  Stdout: u''
  Stderr: u'Cannot format device /dev/sdb which is still in use.\n'

  I have checked that /dev/sdb appears for short time (/dev/sda is
  unencrypted drive attached to an instance on the same host) and
  running cryptsetup luksFormat gives 'cannot format device...still in
  use'

  (nova-compute)[root@compute5 /]# pip freeze | grep nova
  nova==13.0.0.0rc4.dev8314
  python-novaclient==9.1.0

  (nova-compute)[root@compute5 /]# pip freeze | grep oslo
  oslo.cache==1.25.0
  oslo.concurrency==3.21.0
  oslo.config==4.11.0
  oslo.context==2.17.0
  oslo.db==4.25.0
  oslo.i18n==3.17.0
  oslo.log==3.30.0
  oslo.messaging==5.30.0
  oslo.middleware==3.30.0
  oslo.policy==1.25.1
  oslo.privsep==1.22.0
  oslo.reports==1.22.0
  oslo.rootwrap==5.9.0
  oslo.serialization==2.20.0
  oslo.service==1.25.0
  oslo.utils==3.28.0
  oslo.versionedobjects==1.26.0
  oslo.vmware==2.24.0

  (nova-compute)[root@compute5 /]# pip freeze | grep barbican
  python-barbicanclient==4.5.2

  (nova-compute)[root@compute5 /]# rpm -qa | grep crypt
  libgcrypt-devel-1.5.3-14.el7.x86_64
  libgcrypt-1.5.3-14.el7.x86_64
  cryptsetup-1.7.4-3.el7.x86_64
  cryptsetup-libs-1.7.4-3.el7.x86_64

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1721522/+subscriptions
References

[Bug 1721522] [NEW] encrypted volumes: Cannot format device /dev/sdb which is still in use
From: Vladislav Belogrudov, 2017-10-05